right, for sure, but a good solution to the high bandwidth case that doesn’t scale down to gridworld correctly is useless, it can’t be trusted to actually be a good solution to the larger problem if it definitely breaks when shrunk. it should work just as well in gridworld, or conway’s life*, or smoothlife*, or mujoco, or a fluid sim, or real life. my point is that a good solution should at an absolute bare minimum scale down. if your base rl algorithm can’t solve gridworld, it definitely just doesn’t work at all. similarly for rl safety.
* although I’m not sure if conway’s or smoothlife have an energy conservation law, and that might be pretty important. also, maybe continuous state spaces are really important somehow, I definitely see some possibilities where that’s the case. I don’t know, though. it seems to me like we want to be able to go all the way down to the smallest units and still have a working solution.
I think I have a lot more uncertainty here. Like, yes it is always nice when solutions scale down all the way into the simplest setting, but there is no guarantee that that’s how things work in this domain. Reality is allowed to say “the minimum requirements for building a FAI are X”, where X entails a high-bandwidth or otherwise highly-specific interface between us and the agent.
right, for sure, but a good solution to the high bandwidth case that doesn’t scale down to gridworld correctly is useless, it can’t be trusted to actually be a good solution to the larger problem if it definitely breaks when shrunk. it should work just as well in gridworld, or conway’s life*, or smoothlife*, or mujoco, or a fluid sim, or real life. my point is that a good solution should at an absolute bare minimum scale down. if your base rl algorithm can’t solve gridworld, it definitely just doesn’t work at all. similarly for rl safety.
* although I’m not sure if conway’s or smoothlife have an energy conservation law, and that might be pretty important. also, maybe continuous state spaces are really important somehow, I definitely see some possibilities where that’s the case. I don’t know, though. it seems to me like we want to be able to go all the way down to the smallest units and still have a working solution.
I think I have a lot more uncertainty here. Like, yes it is always nice when solutions scale down all the way into the simplest setting, but there is no guarantee that that’s how things work in this domain. Reality is allowed to say “the minimum requirements for building a FAI are X”, where X entails a high-bandwidth or otherwise highly-specific interface between us and the agent.