Here’s a “physicalist” scenario that I think is closely analogous if not identical, that I was thinking about, let me know if it makes sense or not.
Some humans are trying to figure out how the Tegmark level 4 multiverse works. If those humans were smarter—like how an AGI will be smarter—then maybe they’ll succeed. And if they succeed then they’d have a very sound prior for anthropic reasoning. And maybe they would do such reasoning and see that the overwhelming majority of observers-like-us are being simulated by other agents, with a googolplex lives on the line who will be happy or tortured depending on whether we do some particular action X.
Some reasons this scenario does not concern me is:
I don’t think there is enough information to disambiguate different equally-plausible schemes for weights / priors about the different universes in the multiverse (this is analogous to uncertainty about which universal Turing machine to use for your Solomonoff prior),
I don’t think that we could ever figure out that the basement-universe-people-who-are-simulating-us want us to do X, even if they did in fact want us to do X, and I don’t think a superintelligent AGI could figure that out either (this is related to us not having a Solomonoff oracle, as you mention)
Hopefully the AGI will adopt a policy of categorically not giving into threats, even if there are in fact a googolplex lives on the line, and therefore won’t try to figure out whether it’s being simulated or not. Also, by weird decision theory logic, if you make an assumption that you’re not being simulated, and never bother to check it, then it can be kinda self-fulfilling, because the basement-universe-people might be insightful enough to anticipate that and not bother simulating you a bunch of times in the first place.
Again, I haven’t thought about this very much, I may be confused.
I basically agree. There’s both the practical issue and the theoretical one.
In our universe, we can’t simulate these universes/mathematical objects that are simulating us, and reasoning about properties of computational things without simulating them is often hard to impossible.
Supposing we did have big compute and could simulate this thing that simulates us—can we choose actions such that they’re not incentivized to try to manipulate us? Should we even assign measure to the mathematical multiverse such that we care? I think the answers are “yes” and “maybe not,” but am unsure.
Here’s a “physicalist” scenario that I think is closely analogous if not identical, that I was thinking about, let me know if it makes sense or not.
Some humans are trying to figure out how the Tegmark level 4 multiverse works. If those humans were smarter—like how an AGI will be smarter—then maybe they’ll succeed. And if they succeed then they’d have a very sound prior for anthropic reasoning. And maybe they would do such reasoning and see that the overwhelming majority of observers-like-us are being simulated by other agents, with a googolplex lives on the line who will be happy or tortured depending on whether we do some particular action X.
Some reasons this scenario does not concern me is:
I don’t think there is enough information to disambiguate different equally-plausible schemes for weights / priors about the different universes in the multiverse (this is analogous to uncertainty about which universal Turing machine to use for your Solomonoff prior),
I don’t think that we could ever figure out that the basement-universe-people-who-are-simulating-us want us to do X, even if they did in fact want us to do X, and I don’t think a superintelligent AGI could figure that out either (this is related to us not having a Solomonoff oracle, as you mention)
Hopefully the AGI will adopt a policy of categorically not giving into threats, even if there are in fact a googolplex lives on the line, and therefore won’t try to figure out whether it’s being simulated or not. Also, by weird decision theory logic, if you make an assumption that you’re not being simulated, and never bother to check it, then it can be kinda self-fulfilling, because the basement-universe-people might be insightful enough to anticipate that and not bother simulating you a bunch of times in the first place.
Again, I haven’t thought about this very much, I may be confused.
I basically agree. There’s both the practical issue and the theoretical one.
In our universe, we can’t simulate these universes/mathematical objects that are simulating us, and reasoning about properties of computational things without simulating them is often hard to impossible.
Supposing we did have big compute and could simulate this thing that simulates us—can we choose actions such that they’re not incentivized to try to manipulate us? Should we even assign measure to the mathematical multiverse such that we care? I think the answers are “yes” and “maybe not,” but am unsure.