It also bears remembering that any simulation we build of reality is designed to fit a specific set of recorded observations, where agentic selection effects may skew data quite significantly in various places.
Yup. I’ve been idly considering some sort of generator of synthetic data designed to produce training sets which we could mix into real data to provably obscure such signals.[1] It is maybe sort of doable for math, but probably not for physics/biology. (I commend your paranoia here, by the way.)
Overall, though, getting into this sort of fight with potential misaligned superintelligent agents isn’t a great idea; their possibility should be crushed somewhere upstream of that point.
The existence of the simulation must have an influence in this world, since it would otherwise be pointless, and they can’t be drawing their insights from a simulation of their own since otherwise you lose interpretability in infinite recursion-wells, so the simulation must necessarily be disanalogous to here in at least one key way.
Mm-hm. My go-to heuristic here is to ask: how do human world-models handle this type of failure mode? Suppose we’re trying to model someone who gets access to a compute-unbounded oracle, asks it about the future, then takes some actions that depend on the answer, thereby creating a stable time loop. Suppose we care about accuracy, but we don’t have the unbounded compute to actually run this. We have to approximate.
Is modeling it as a sequence of nested simulations which terminates at some ground-floor simulation that doesn’t contain an oracle actually an accurate, faithful way to model this? Clearly no: the process we’re modeling explicitly doesn’t have any layers like this. The type signature of our “approximation” doesn’t match the original even approximately; it’s some completely unrelated thing.
Instead, we attempt to abstract over the whole process, and skip directly to the prediction of what happens in the limit of infinite nesting, without simulating anything. Same way we compute f(x)=x−x33!+x55!−x77!+... not by actually calculating all those terms, but by collapsing the infinite power series to sin(x).
One thing to note here is that the kind of world-model I’m aiming for is not a “brute-force” simulator. It’s explicitly using abstractions as basic units, and it should be able to “skip to the limit” in the above manner. So, for a large class of situations, the potential simulacrum escapees would not be actually simulated at the level of gears; just abstractly approximated.
(Somewhat relevant: this discussion about brute-force-y “gormless” simulators that get scammed by acausal terrorists, vs. more sophisticated simulators. I’m aiming for the more sophisticated kind. This should also take care of said acausal attacks.)
It’s a tree falling in a forest and is probably not a concern, but it’s also unlikely that anyone would build a system they definitionally cannot use for anything.
Alright, suppose the agent inside the box is tasked with producing some proofs of work. They are automatically verified, type-checked, then shown on a small, low-tech digital display on the box. The software is mathematically verified to be correct/unhackable, the hardware is robustly designed not to allow any sequence of computation to cause a malfunction that would move the software into theoretically impossible states, and the display can only show letters/numbers.
The only thing the agent can do is providing or not providing proofs of work; it can’t hack out or show any message which is not a valid proof of work.
This way, you can extract useful work from the agent, while still not letting the type signature of the box become “an agent”. It is just a PoW-generator.
Similarly, a simulation whose security trumps the capabilities of the agents it simulates can remain an “inert” simulation which just happens to house agents, rather than becoming an agent of its own.
Though the actual motivation there was figuring out if we can obscure information about our universe’s actual physics, biology, and human cognition, so that if something in the synthesis process goes wrong and it spawns misaligned agents, they’re less likely to be able to construct reliable Basilisk hacks. (Because if that failure mode is allowed, we can’t actually use the interpretability property to verify the synthesized world-model’s safety prior to running it.)
Yup. I’ve been idly considering some sort of generator of synthetic data designed to produce training sets which we could mix into real data to provably obscure such signals.[1] It is maybe sort of doable for math, but probably not for physics/biology. (I commend your paranoia here, by the way.)
Overall, though, getting into this sort of fight with potential misaligned superintelligent agents isn’t a great idea; their possibility should be crushed somewhere upstream of that point.
Mm-hm. My go-to heuristic here is to ask: how do human world-models handle this type of failure mode? Suppose we’re trying to model someone who gets access to a compute-unbounded oracle, asks it about the future, then takes some actions that depend on the answer, thereby creating a stable time loop. Suppose we care about accuracy, but we don’t have the unbounded compute to actually run this. We have to approximate.
Is modeling it as a sequence of nested simulations which terminates at some ground-floor simulation that doesn’t contain an oracle actually an accurate, faithful way to model this? Clearly no: the process we’re modeling explicitly doesn’t have any layers like this. The type signature of our “approximation” doesn’t match the original even approximately; it’s some completely unrelated thing.
Instead, we attempt to abstract over the whole process, and skip directly to the prediction of what happens in the limit of infinite nesting, without simulating anything. Same way we compute f(x)=x−x33!+x55!−x77!+... not by actually calculating all those terms, but by collapsing the infinite power series to sin(x).
One thing to note here is that the kind of world-model I’m aiming for is not a “brute-force” simulator. It’s explicitly using abstractions as basic units, and it should be able to “skip to the limit” in the above manner. So, for a large class of situations, the potential simulacrum escapees would not be actually simulated at the level of gears; just abstractly approximated.
(Somewhat relevant: this discussion about brute-force-y “gormless” simulators that get scammed by acausal terrorists, vs. more sophisticated simulators. I’m aiming for the more sophisticated kind. This should also take care of said acausal attacks.)
Alright, suppose the agent inside the box is tasked with producing some proofs of work. They are automatically verified, type-checked, then shown on a small, low-tech digital display on the box. The software is mathematically verified to be correct/unhackable, the hardware is robustly designed not to allow any sequence of computation to cause a malfunction that would move the software into theoretically impossible states, and the display can only show letters/numbers.
The only thing the agent can do is providing or not providing proofs of work; it can’t hack out or show any message which is not a valid proof of work.
This way, you can extract useful work from the agent, while still not letting the type signature of the box become “an agent”. It is just a PoW-generator.
Similarly, a simulation whose security trumps the capabilities of the agents it simulates can remain an “inert” simulation which just happens to house agents, rather than becoming an agent of its own.
Though the actual motivation there was figuring out if we can obscure information about our universe’s actual physics, biology, and human cognition, so that if something in the synthesis process goes wrong and it spawns misaligned agents, they’re less likely to be able to construct reliable Basilisk hacks. (Because if that failure mode is allowed, we can’t actually use the interpretability property to verify the synthesized world-model’s safety prior to running it.)