I’d like to use this post to ask a question I’ve had for a while about the incompleteness theorem and the halting problem but never had a good chance to ask.
When I first heard that that any sufficiently rich formal system, together with an interpretation, has strings which are true but unprovable or that it is not possible to determine whether a program halts, this felt like an amazing and important discovery. But when I think more about the standard proof of the halting problem, or the hear this explanation of incompleteness theorem or the English “this sentence is a lie” example the importance seems to diminish.
Sure, I cannot in general determine whether a program halts but the only counter example is this weird recursive statement that I do not actually care about. Maybe I can determine whether all programs that I do care about halt and ignore this one weirdness. Similarly if the only true unprovable statements are of this form then maybe I can still have formal systems that are complete for all other statements (those that I care about).
At this point my confusion becomes more fuzzy and I’m not sure how exactly to put it into words. It feels important that for the halting problem many systems turn out to be unexpectedly turing complete. Is one answer that it is impossible to determine whether a program is equivalent to the one used in the halting problem proof?
I agree and I am putting my money where my mouth is.
I will play this game under the rules linked in the OP with me as the gatekeeper and anyone as the AI. I will bet at odds 10:1 (favorable to you) that I will not let the AI out. The minimum bet is my 10 k USD against your 1 k USD and the maximum bet my 100 k USD against your 10 k USD. We will use a mutually accepted LW community member as referee.
If you believe you have at least a 10% chance of making me let the AI out, you should take this bet. I predict no one will take me up on it.
I speculate that the reason that gatekeepers let the AI out is that they do not take the experiment seriously enough. They don’t really care about not letting the AI out. Maybe they want the outcome to be that the AI has been let out so that people take AI safety more seriously. I’m not saying an actual superintelligence couldn’t do it, but no currently available intelligence can (with me as the gatekeeper).