The point is that “the AI beats the stock market, so it’s intelligent” rules out a completely unintelligent AI, but doesn’t rule out the AI having flaws which lead it to produce stupid results when it comes to humans having tails. This is still true if you add the extra step of “maybe I’m deluded about...”. Perhaps the AI is actually intelligent when it comes to the stock market, but you’re deluded into thinking the AI makes intelligent decisions in more subject areas than it really does.
(This is especially likely if you’ve examined the AI’s code and “proven” that the AI reasons perfectly. You may have missed something that doesn’t happen to affect conclusions about the stock market but does affect conclusions about tails.)
The point is that “the AI beats the stock market, so it’s intelligent” rules out a completely unintelligent AI, but doesn’t rule out the AI having flaws which lead it to produce stupid results when it comes to humans having tails. This is still true if you add the extra step of “maybe I’m deluded about...”. Perhaps the AI is actually intelligent when it comes to the stock market, but you’re deluded into thinking the AI makes intelligent decisions in more subject areas than it really does.
(This is especially likely if you’ve examined the AI’s code and “proven” that the AI reasons perfectly. You may have missed something that doesn’t happen to affect conclusions about the stock market but does affect conclusions about tails.)
I continue to fail to disagree with you...