Non-wireheading-ness or non-Goodhart’s-law-prone-ness normally goes by the name of intentionality (SEP, Scholarpedia) or symbol grounding, both of which have an extensive associated literature.
Humans are pretty bad at symbol grounding, you can see a whole lot of ‘its map, not territory’ posts by people who happen to have a better map, and may well be saying ‘its your map, but not my more detailed one’.
That’s a good point and would warrant a discussion post if you could come up with good examples. The sin is especially egregious when there’s a large enough difference in the intelligence of the two people such that a comparison of maps isn’t directly meaningful. E.g. it’s only sometimes normatively appropriate to point out the alleged difference between “the sun goes around the Earth” and “the Earth goes around the sun”.
Non-wireheading-ness or non-Goodhart’s-law-prone-ness normally goes by the name of intentionality (SEP, Scholarpedia) or symbol grounding, both of which have an extensive associated literature.
Humans are pretty bad at symbol grounding, you can see a whole lot of ‘its map, not territory’ posts by people who happen to have a better map, and may well be saying ‘its your map, but not my more detailed one’.
That’s a good point and would warrant a discussion post if you could come up with good examples. The sin is especially egregious when there’s a large enough difference in the intelligence of the two people such that a comparison of maps isn’t directly meaningful. E.g. it’s only sometimes normatively appropriate to point out the alleged difference between “the sun goes around the Earth” and “the Earth goes around the sun”.