I like this story, it touches on certainty bounds relative to superintelligence, which I think is an underexplored concept. That is: if you’re an AGI that knows every bit of information relevant to a certain plan, and has the “cognitive” power to make use of this info, what would you estimate your plan’s chance of success to be? Can it ever reach 100%?
To be fair, I don’t think answers to this question are very tractable right now, afaict we don’t have detailed enough physics to usefully estimate our world’s level of determinism. But this feature-of-universe seems especially relevant to how likely an AGI is to attempt complex plans (both aligned and unaligned) which carry some probability of it being turned off forever.
If anyone knows of existing works which discuss this in more depth I’d love some recommendations!
I like this story, it touches on certainty bounds relative to superintelligence, which I think is an underexplored concept. That is: if you’re an AGI that knows every bit of information relevant to a certain plan, and has the “cognitive” power to make use of this info, what would you estimate your plan’s chance of success to be? Can it ever reach 100%?
To be fair, I don’t think answers to this question are very tractable right now, afaict we don’t have detailed enough physics to usefully estimate our world’s level of determinism. But this feature-of-universe seems especially relevant to how likely an AGI is to attempt complex plans (both aligned and unaligned) which carry some probability of it being turned off forever.
If anyone knows of existing works which discuss this in more depth I’d love some recommendations!