But in the context of superhuman systems, I think we need to be more concerned by the possibility that it’s performance-uncompetitive to restrict your system to only take actions that can be justified entirely with human-understandable reasoning.
Interestingly, this is already a well known phenomenon in the hedge fund world. In fact, quant funds discovered about 25 years ago that the most consistently profitable trading signals are reliably the ones that are the least human-interpretable. It makes intuitive sense: any signal that can be understood by a human is at risk of being copied by a human, so if you insist that your trading decisions have to be interpretable, you’ll pay for that insistence in alpha.
I’d imagine this kind of issue is already top-of-mind for folks who are working on the various transparency agendas, but it does imply that there’s a very strong optimization pressure directly against interpretability in many economically relevant contexts. In fact, it could hardly be stronger: your forcing function is literally “Want to be a billionaire? Then you’ll have to trade exclusively on the most incomprehensible signals you can find.”
(Of course this isn’t currently true of all hedge funds, only a few specialized ones.)
Interestingly, this is already a well known phenomenon in the hedge fund world. In fact, quant funds discovered about 25 years ago that the most consistently profitable trading signals are reliably the ones that are the least human-interpretable. It makes intuitive sense: any signal that can be understood by a human is at risk of being copied by a human, so if you insist that your trading decisions have to be interpretable, you’ll pay for that insistence in alpha.
I’d imagine this kind of issue is already top-of-mind for folks who are working on the various transparency agendas, but it does imply that there’s a very strong optimization pressure directly against interpretability in many economically relevant contexts. In fact, it could hardly be stronger: your forcing function is literally “Want to be a billionaire? Then you’ll have to trade exclusively on the most incomprehensible signals you can find.”
(Of course this isn’t currently true of all hedge funds, only a few specialized ones.)