I basically think this distinction is real and we are talking about problem 1 instead of problem 2. That said, I don’t feel like it’s quite right to frame it as “states” that the human does or doesn’t understand. Instead we’re thinking about properties of the world as being ambiguous or not in a given state.
As a silly example, you could imagine having two rooms where one room is normal and the other is crazy. Then questions about the first room are easy and questions about the second are hard. But in reality the degrees of freedom will be much more mixed up than that.
To give some more detail on my thoughts on state:
Obviously the human never knows the “real” state, which has a totally different type signature than their beliefs.
So it’s natural to talk about knowing states based on correctly predicting what will happen in the future starting from that state. But it’s ~never the case that the human’s predictions about what will happen next are nearly as good as the predictor’s.
We could try to say “you can make good predictions about what happens next for typical actions” or something, but even for typical actions the human predictions are bad relative to the predictor, and it’s not clear in what sense they are “good” other than some kind of calibration condition.
If we imagine an intuitive translation between two models of reality, most “weird” states aren’t outside of the domain of the translation, it’s just that there are predictively important parts of the state that are obscured by the translation (effectively turning into noise, perhaps very surprising noise).
Despite all of that, it seems like it really is sometimes unambiguous to say “You know that thing out there in the world that you would usually refer to by saying ‘the diamond is sitting there and nothing weird happened to it’? That thing which would lead you to predict that the camera will show a still frame of a diamond? That thing definitely happened, and is why the camera is showing a still frame of a diamond, it’s not for some other reason.”
Echoing Mark and Ajeya:
I basically think this distinction is real and we are talking about problem 1 instead of problem 2. That said, I don’t feel like it’s quite right to frame it as “states” that the human does or doesn’t understand. Instead we’re thinking about properties of the world as being ambiguous or not in a given state.
As a silly example, you could imagine having two rooms where one room is normal and the other is crazy. Then questions about the first room are easy and questions about the second are hard. But in reality the degrees of freedom will be much more mixed up than that.
To give some more detail on my thoughts on state:
Obviously the human never knows the “real” state, which has a totally different type signature than their beliefs.
So it’s natural to talk about knowing states based on correctly predicting what will happen in the future starting from that state. But it’s ~never the case that the human’s predictions about what will happen next are nearly as good as the predictor’s.
We could try to say “you can make good predictions about what happens next for typical actions” or something, but even for typical actions the human predictions are bad relative to the predictor, and it’s not clear in what sense they are “good” other than some kind of calibration condition.
If we imagine an intuitive translation between two models of reality, most “weird” states aren’t outside of the domain of the translation, it’s just that there are predictively important parts of the state that are obscured by the translation (effectively turning into noise, perhaps very surprising noise).
Despite all of that, it seems like it really is sometimes unambiguous to say “You know that thing out there in the world that you would usually refer to by saying ‘the diamond is sitting there and nothing weird happened to it’? That thing which would lead you to predict that the camera will show a still frame of a diamond? That thing definitely happened, and is why the camera is showing a still frame of a diamond, it’s not for some other reason.”