All I did to get the supposed 99% success rate was to basically feed in the answers to the test problems along with the questions.
Presumably the AI was tested with questions whose answers were not known in advance to guard against the problem of self-deception (or more likely, to ensure that you are capable of convincing others that you are not self-deceiving about the AI’s accuracy).
Indeed, and I might believe such testing was carried out and was as effective as it was supposed to be. But my point is, it is much more likely that I am wrong in that belief, than that I am wrong in the belief that we don’t have tails. This remains true no matter how thorough the testing. It also remains true if you substitute aliens, gods etc. for the AI; the conclusion doesn’t depend on the specifics of the information source.
I think you go wrong when you say that it remains true no matter how thorough the testing. Suppose the AI is beating the stock market over the course of months based on massive online information collection; in the meantime, you’re reading webcomics and watching the graph of the AI’s money fund plot a trajectory ever upwards. According to you, upon being told by the AI that all humanity is hallucinating something utterly wacky, you should believe that it was actually you beating the stock market all the while, even though as far as you can recall, you are sane and you have had no direct input into the process for months.
I think there are some tests for which success and simultaneous self-deception of the human AI programmer is as unlikely as whatever the AI comes up with about humanity in general.
There are intermediates between “the AI isn’t intelligent at all and the statement about tails is just a random output that it produces when there’s no preprogrammed chatbot response” and “everything the AI says is done with intelligence”. The AI could be intelligent but with a flaw which leads it to make unintelligent statements some of the time without everything it says being unintelligent.
What were we talking about again? Replies to comments I made five years ago always catch me off guard...
Anyway, I don’t disagree with you. I’m not sure if you mean to be disagreeing with me, but if so, I would note that my comment doesn’t imply or rely on a hard dichotomization of the class of AIs into chatbots and infallible AGIs.
The point is that “the AI beats the stock market, so it’s intelligent” rules out a completely unintelligent AI, but doesn’t rule out the AI having flaws which lead it to produce stupid results when it comes to humans having tails. This is still true if you add the extra step of “maybe I’m deluded about...”. Perhaps the AI is actually intelligent when it comes to the stock market, but you’re deluded into thinking the AI makes intelligent decisions in more subject areas than it really does.
(This is especially likely if you’ve examined the AI’s code and “proven” that the AI reasons perfectly. You may have missed something that doesn’t happen to affect conclusions about the stock market but does affect conclusions about tails.)
Presumably the AI was tested with questions whose answers were not known in advance to guard against the problem of self-deception (or more likely, to ensure that you are capable of convincing others that you are not self-deceiving about the AI’s accuracy).
Indeed, and I might believe such testing was carried out and was as effective as it was supposed to be. But my point is, it is much more likely that I am wrong in that belief, than that I am wrong in the belief that we don’t have tails. This remains true no matter how thorough the testing. It also remains true if you substitute aliens, gods etc. for the AI; the conclusion doesn’t depend on the specifics of the information source.
Ah, I see. That’s a good argument.
I think you go wrong when you say that it remains true no matter how thorough the testing. Suppose the AI is beating the stock market over the course of months based on massive online information collection; in the meantime, you’re reading webcomics and watching the graph of the AI’s money fund plot a trajectory ever upwards. According to you, upon being told by the AI that all humanity is hallucinating something utterly wacky, you should believe that it was actually you beating the stock market all the while, even though as far as you can recall, you are sane and you have had no direct input into the process for months.
I think there are some tests for which success and simultaneous self-deception of the human AI programmer is as unlikely as whatever the AI comes up with about humanity in general.
There are intermediates between “the AI isn’t intelligent at all and the statement about tails is just a random output that it produces when there’s no preprogrammed chatbot response” and “everything the AI says is done with intelligence”. The AI could be intelligent but with a flaw which leads it to make unintelligent statements some of the time without everything it says being unintelligent.
What were we talking about again? Replies to comments I made five years ago always catch me off guard...
Anyway, I don’t disagree with you. I’m not sure if you mean to be disagreeing with me, but if so, I would note that my comment doesn’t imply or rely on a hard dichotomization of the class of AIs into chatbots and infallible AGIs.
The point is that “the AI beats the stock market, so it’s intelligent” rules out a completely unintelligent AI, but doesn’t rule out the AI having flaws which lead it to produce stupid results when it comes to humans having tails. This is still true if you add the extra step of “maybe I’m deluded about...”. Perhaps the AI is actually intelligent when it comes to the stock market, but you’re deluded into thinking the AI makes intelligent decisions in more subject areas than it really does.
(This is especially likely if you’ve examined the AI’s code and “proven” that the AI reasons perfectly. You may have missed something that doesn’t happen to affect conclusions about the stock market but does affect conclusions about tails.)
I continue to fail to disagree with you...