Yeah :/ I’ve struggled for a long time to see how the world could be good with strong AI, and I’ve felt pretty alienated in that. Most of the time when I talk to people about it they’re like “well the world could just be however you like!” Almost as if, definitionally, I should be happy because in the really strong success cases we’ll have the tech to satisfy basically any preference. But that’s almost the entire problem, in some way? As you say, figuring things out for ourselves, thinking and learning and taking pride in skills that take effort to acquire… most of what I cherish about these things has to do with grappling with new territory. And if I know that it is not in fact new, if all of it could be easier were I to use the technology right there… it feels as though something is corrupted… The beauty of curiosity, wonder, and discovery feels deeply bound to the unknown, to me.
I was talking to a friend about this a few months ago and he suggested that because many humans have these preferences, that we ought to be able to make a world where we satisfy them—e.g., something like “the AI does its thing over there and we sit over here having basically normal human lives except that death is a choice and sometimes it helps us figure out hard coordination problems or whatever.” And I can almost get behind this, but something still feels off to me. Like how when people get polarized through social media it almost seems like there’s no going back? How do we know strange spirals won’t happen with an even more advanced technology? It’s hard to escape the feeling that a dystopia lurks. Hard to escape the feeling that all the people I know and love might change quickly and radically, that I might change radically, in ways that feel alien to me now. I want to believe that strong AI would be great, and perhaps it would be, perhaps I’m missing something here. But a part of me is terrified.
I realise this is a few months old but personally my vision for utopia looks something like the Culture in the Culture novels by Iain M. Banks. There’s a high degree of individual autonomy and people create their own societies organically according to their needs and values. They still have interpersonal struggles and personal danger (if that’s the life they want to lead) but in general if they are uncomfortable with their situation they have the option to change it. AI agents are common, but most are limited to approximately human level or below. Some superhuman AI’s exist but they are normally involved in larger civilisational manouvering rather than the nitty gritty of individual human lives. I recommend reading it.
Caveats-
1: yes, this is a fictional example so I’m definitely in danger of generalising from fictional evidence. I mostly think about it as a broad template or cluster of attributes society might potentially be able to achieve.
2: I don’t think this level of “good” AI is likely.
Discovering and mastering one’s own psychology may still be a frontier where the AI could help only marginally. So, more people will become monks or meditators?
Yeah :/ I’ve struggled for a long time to see how the world could be good with strong AI, and I’ve felt pretty alienated in that. Most of the time when I talk to people about it they’re like “well the world could just be however you like!” Almost as if, definitionally, I should be happy because in the really strong success cases we’ll have the tech to satisfy basically any preference. But that’s almost the entire problem, in some way? As you say, figuring things out for ourselves, thinking and learning and taking pride in skills that take effort to acquire… most of what I cherish about these things has to do with grappling with new territory. And if I know that it is not in fact new, if all of it could be easier were I to use the technology right there… it feels as though something is corrupted… The beauty of curiosity, wonder, and discovery feels deeply bound to the unknown, to me.
I was talking to a friend about this a few months ago and he suggested that because many humans have these preferences, that we ought to be able to make a world where we satisfy them—e.g., something like “the AI does its thing over there and we sit over here having basically normal human lives except that death is a choice and sometimes it helps us figure out hard coordination problems or whatever.” And I can almost get behind this, but something still feels off to me. Like how when people get polarized through social media it almost seems like there’s no going back? How do we know strange spirals won’t happen with an even more advanced technology? It’s hard to escape the feeling that a dystopia lurks. Hard to escape the feeling that all the people I know and love might change quickly and radically, that I might change radically, in ways that feel alien to me now. I want to believe that strong AI would be great, and perhaps it would be, perhaps I’m missing something here. But a part of me is terrified.
I realise this is a few months old but personally my vision for utopia looks something like the Culture in the Culture novels by Iain M. Banks. There’s a high degree of individual autonomy and people create their own societies organically according to their needs and values. They still have interpersonal struggles and personal danger (if that’s the life they want to lead) but in general if they are uncomfortable with their situation they have the option to change it. AI agents are common, but most are limited to approximately human level or below. Some superhuman AI’s exist but they are normally involved in larger civilisational manouvering rather than the nitty gritty of individual human lives. I recommend reading it.
Caveats-
1: yes, this is a fictional example so I’m definitely in danger of generalising from fictional evidence. I mostly think about it as a broad template or cluster of attributes society might potentially be able to achieve.
2: I don’t think this level of “good” AI is likely.
Discovering and mastering one’s own psychology may still be a frontier where the AI could help only marginally. So, more people will become monks or meditators?