Why isn’t this an example of the mind projection fallacy?
It is. I think Eliezer’s merely trying to drive home the point that Quantum Mechanics is the closest thing we have to the territory. More accurately, it’s the most accurate map. But it’s still a map. Classical mechanics might be like a Beck map, and this simple, high-detail geographical map might be virtually indistinguishable from the territory by comparison, but Quantum Mechanics fails to describe the world accurately in some respects. (Think General Relativity.) It’s a sad truth, but not one ignored lightly.
And, to be pedantic, even if we one day make a model that reflects reality exactly, our equations will still be describing the model first, and only reality incidentally.
2^^^2 is 4, so I’d choose that in a heartbeat. 2^^^3 is the kind of number you were probably thinking about. Though, if we’re choosing fair-sounding situations, I’d like to cut one of my fingernails too short to generate a MJ/K of negentropy.
I’ve got one way of thinking this problem through that seems to fit with what you’re saying – though of course, it has its own flaws: represent each person’s utility (is that the right word in this case) such that 0 is the maximum possible utility they can have, then map each individual’s utility with x ⟼ -(e^(-x)), so that lots of harm to one person is weighted higher than tiny harms to many people. This is almost certainly a case of forcing the model to say what we want it to say.