Kaj_Sotala provided a good answer, but I want to give an intuitive example:
If you could decide whether: A: a single person lives on Earth, supported by aligned AGI, its knowledge and all resources of the planet in service of nothing but his welfare, living in abundance not even the greatest emperors ever dreamed of. B: a civilization of tens of billions living on Earth, supported by aligned AGI, thanks to which all of them have at least the living standard of a current upper-middle class American.
I believe most people would choose option B. Of course, this is not independent of living conditions (greatly influenced by anchoring), but for me covers the general “feeling” of the idea. I would formulate it along the lines of “due to diminishing returns, spending resources on increasing living standards above a certain level is wasteful, more goodness/utility is created if other humans are included”.
I would like to also suggest for reading a sci-fi short story by one of the LessWrongers, which deals a lot with this question (among other things that are also memeworthy), especially in chapter 3: https://timunderwoodscifi.wordpress.com/index/
Kaj_Sotala provided a good answer, but I want to give an intuitive example:
If you could decide whether:
A: a single person lives on Earth, supported by aligned AGI, its knowledge and all resources of the planet in service of nothing but his welfare, living in abundance not even the greatest emperors ever dreamed of.
B: a civilization of tens of billions living on Earth, supported by aligned AGI, thanks to which all of them have at least the living standard of a current upper-middle class American.
I believe most people would choose option B. Of course, this is not independent of living conditions (greatly influenced by anchoring), but for me covers the general “feeling” of the idea. I would formulate it along the lines of “due to diminishing returns, spending resources on increasing living standards above a certain level is wasteful, more goodness/utility is created if other humans are included”.
I would like to also suggest for reading a sci-fi short story by one of the LessWrongers, which deals a lot with this question (among other things that are also memeworthy), especially in chapter 3:
https://timunderwoodscifi.wordpress.com/index/