Sure. “If it’s smart, it won’t make simple mistakes.” But I’m also interested in the question of whether, given the first few in this sequence of approximate agents, one could do a good job at predicting the next one.
It seems like you could—like there is a simple rule governing these systems (“check whether there’s a human in the greenhouse”) that might involve difficult interaction with the world in practice but is much more straightforward when considered from the omniscient third-person view of imagination. And given that this rule is (arguendo) simple within a fairly natural (though not by any means unique) model of the world, and that it helps predict the sequence, one might be able to guess that this rule was likely just from looking at the sequence of systems.
(This also relies on the distinction between just trying to find likely or good-enough answers, and the AI doing search to find weird corner cases. The inferred next step in the sequence might be expected to give similar likely answers, with no similar guarantee for corner-case answers.)
Sure. “If it’s smart, it won’t make simple mistakes.” But I’m also interested in the question of whether, given the first few in this sequence of approximate agents, one could do a good job at predicting the next one.
It seems like you could—like there is a simple rule governing these systems (“check whether there’s a human in the greenhouse”) that might involve difficult interaction with the world in practice but is much more straightforward when considered from the omniscient third-person view of imagination. And given that this rule is (arguendo) simple within a fairly natural (though not by any means unique) model of the world, and that it helps predict the sequence, one might be able to guess that this rule was likely just from looking at the sequence of systems.
(This also relies on the distinction between just trying to find likely or good-enough answers, and the AI doing search to find weird corner cases. The inferred next step in the sequence might be expected to give similar likely answers, with no similar guarantee for corner-case answers.)