I’m still a bit vague on this Löb business. Is it a good thing or a bad thing (from a creating AI perspective) that “Löb’s theorem fails” for the probabilistic version?
edit: The old post linked suggests that it’s good that it doesn’t apply here.
Er I don’t think this is right. Lob’s theorem says that an agent cannot trust future copies of itself, unless those future copies use strictly weaker axioms in their reasoning system.
I’m still a bit vague on this Löb business. Is it a good thing or a bad thing (from a creating AI perspective) that “Löb’s theorem fails” for the probabilistic version?
edit: The old post linked suggests that it’s good that it doesn’t apply here.
It’s a good thing. Löb’s theorem is an obstacle (the “Löbstacle,” if you will).
Löb’s theorem’s means an agent cannot trust future copies of itself, or simply identical copies of itself, to only prove true statements.
Er I don’t think this is right. Lob’s theorem says that an agent cannot trust future copies of itself, unless those future copies use strictly weaker axioms in their reasoning system.
The “can” has now been changed into “cannot”. D’oh!
Good for creating AGI, maybe bad for surviving it. Hopefully the knowledge will also help us predict the actions of strong self-modifying AI.
It does seem promising to this layman, since it removes the best reason I could imagine for considering that last goal impossible.