“Now look at that bulleted list above, and think about how it would apply here. For example: “You understand exactly what the thing is supposed to be doing, in every situation that you care about.””
I don’t understand what Jeff Bezos AGI is supposed to be doing in every situation I care about, but I know what it isn’t supposed to be doing in many situations I care about. In fact we already have a fairly detailed set of instructions for that: it’s the law of the land! Laws are a description of what non-AGI Jeff Bezos can and cannot do.
I understand laws are woefully underspecified for many human situations—which is why we have courts and trials and all that jazz—let alone AGI situations. But I think they’re a good starting point for what an entity, be it human or AGI, can and cannot do out in the wild.
I don’t understand what Jeff Bezos AGI is supposed to be doing in every situation I care about, but I know what it isn’t supposed to be doing in many situations I care about. In fact we already have a fairly detailed set of instructions for that: it’s the law of the land! Laws are a description of what non-AGI Jeff Bezos can and cannot do.
I understand laws are woefully underspecified for many human situations—which is why we have courts and trials and all that jazz—let alone AGI situations. But I think they’re a good starting point for what an entity, be it human or AGI, can and cannot do out in the wild.