The intro shows me how I disagree with the veil of ignorance. I think it’s one of those thought experiments that only seems neutral because using your own perspective is so natural as to slide beneath notice.
Humans don’t have utility functions. They’re physical systems, and assigning utility functions to them is an act of modeling. What models are best depends (at least) on a choice of universal Turing machine, and different people can and do come to impasses about what humans “really want,” or what physical systems are “really human.”
“Behind the veil of ignorance” isn’t really a place that exists, because that’s like saying “find the best model without choosing a UTM.” It sounds neutral but it’s actually nonsense. In practice, of course, people will just imagine this happening by using the modeling assumptions that are natural to them.
Even more off-topic, this might lead real-world attempts to implement a veil of ignorance to run into steganography-like problems. An agent might be motivated to smuggle self-recognition behind the veil, and it can do so by making itself the only fixed point of its modeling assumptions—i.e. the agent that would endorse its own specific modeling assumptions (when interpreted with its own specific modeling assumptions).
The intro shows me how I disagree with the veil of ignorance. I think it’s one of those thought experiments that only seems neutral because using your own perspective is so natural as to slide beneath notice.
Humans don’t have utility functions. They’re physical systems, and assigning utility functions to them is an act of modeling. What models are best depends (at least) on a choice of universal Turing machine, and different people can and do come to impasses about what humans “really want,” or what physical systems are “really human.”
“Behind the veil of ignorance” isn’t really a place that exists, because that’s like saying “find the best model without choosing a UTM.” It sounds neutral but it’s actually nonsense. In practice, of course, people will just imagine this happening by using the modeling assumptions that are natural to them.
Even more off-topic, this might lead real-world attempts to implement a veil of ignorance to run into steganography-like problems. An agent might be motivated to smuggle self-recognition behind the veil, and it can do so by making itself the only fixed point of its modeling assumptions—i.e. the agent that would endorse its own specific modeling assumptions (when interpreted with its own specific modeling assumptions).