Fascinating questions! Do you think the answers matter pragmatically right now? I assume you do but in my world model they basically don’t. So I’m curious about your model of how AGI and alignment unfold.
I do enjoy moral philosophy, but I haven’t indulged much since starting to work on alignment. That’s because I’m not sure I’d act any differently based on different answers to the questions you raise.
It seems like most of those answers would be served by the same efforts: solve alignment and the “societal alignment” of who commands an intent-aligned AGI. If they’re (a) basically well-intentioned person(s), we’ll get something along the lines of Ord’s Long Reflection and get our answers then.
Or do you think if we rush to get provisional answers now, it will sharply change our actions? Like lobbying to get intent aligned AGI into different hands?
I’d love to think through the complexities of what “best future” means because I’m in what I take to be your boat: moral realism is sort of false but sort of true in that there’s a true answer to “what do humans want” if we mean what do current humans want now or what would they enjoy in the future, and it’s roughly that all sentient beings get to do what they want. But that doesn’t produce straightforward answers if we broaden the scope of the question outside of a sum of current humanity’s current desires...
I will just say that many of the resource arguments you note to be thorny probably aren’t. It takes so much computronium to simulate a sentient mind, more for more sapience (very roughly on average) and actually relatively little to simulate that mind’s environment since of course we’re not going to waste resources simulating irrelevant molecules when we could fudge it and the minds would be just as satisfied...
So a universe chock full of computronium simulating sapient minds each in their own fantasy realm (with maybe some overlap with other minds for efficiency and/or fun) seems like it’s what minds-like-ours mean by “best”. That’s not agreed upon now but it seems quite plausible that we just haven’t thought things through carefully as a species. Of course the guy in charge of the first ASI could easily say “fine but I myself prefer chaos and suffering in which I get to bully anyone I want so that’s the best eutopia IMO, which you can argue with but you shouldn’t if you don’t like extra suffering....”
Thanks! I think these questions do matter, though it’s certainly exploratory work, and at the moment tractability is still unclear. I talk about some of the practical upshots in the final essay.
Fascinating questions! Do you think the answers matter pragmatically right now? I assume you do but in my world model they basically don’t. So I’m curious about your model of how AGI and alignment unfold.
I do enjoy moral philosophy, but I haven’t indulged much since starting to work on alignment. That’s because I’m not sure I’d act any differently based on different answers to the questions you raise.
It seems like most of those answers would be served by the same efforts: solve alignment and the “societal alignment” of who commands an intent-aligned AGI. If they’re (a) basically well-intentioned person(s), we’ll get something along the lines of Ord’s Long Reflection and get our answers then.
Or do you think if we rush to get provisional answers now, it will sharply change our actions? Like lobbying to get intent aligned AGI into different hands?
I’d love to think through the complexities of what “best future” means because I’m in what I take to be your boat: moral realism is sort of false but sort of true in that there’s a true answer to “what do humans want” if we mean what do current humans want now or what would they enjoy in the future, and it’s roughly that all sentient beings get to do what they want. But that doesn’t produce straightforward answers if we broaden the scope of the question outside of a sum of current humanity’s current desires...
I will just say that many of the resource arguments you note to be thorny probably aren’t. It takes so much computronium to simulate a sentient mind, more for more sapience (very roughly on average) and actually relatively little to simulate that mind’s environment since of course we’re not going to waste resources simulating irrelevant molecules when we could fudge it and the minds would be just as satisfied...
So a universe chock full of computronium simulating sapient minds each in their own fantasy realm (with maybe some overlap with other minds for efficiency and/or fun) seems like it’s what minds-like-ours mean by “best”. That’s not agreed upon now but it seems quite plausible that we just haven’t thought things through carefully as a species. Of course the guy in charge of the first ASI could easily say “fine but I myself prefer chaos and suffering in which I get to bully anyone I want so that’s the best eutopia IMO, which you can argue with but you shouldn’t if you don’t like extra suffering....”
Oh dammit there I’m indulging. Back to work. :)
Thanks! I think these questions do matter, though it’s certainly exploratory work, and at the moment tractability is still unclear. I talk about some of the practical upshots in the final essay.