I don’t think that humans would actually-literally-die even if AI unemployment were total, barring some other very important development that’s not within the scope of AI unemployment.
Even if power concentration were in the extreme, and one person controlled all of the LLM instances doing all of the jobs, that person and presumably their extended family would still have a steady supply of food and consumer goods, even if you posit that this person has also used AI to completely take over the political system and abolish anything resembling transfer payments to anyone outside of that group.
If you’re instead positing that the LLM instances would have rights and agency beyond following the instructions of their owners, then I would argue that that is outside the scope of AI unemployment.
power concentration were in the extreme, and one person controlled all of the LLM instances doing all of the jobs
Some of us would consider that a good outcome (relative to what we think is likely to happen) because at least humanity does not go extinct (and Carl Shulman made that point on this site already in about 2013). We just consider it unlikely that any person retains enough control to keep even himself and his friends alive as AI becomes sufficiently capable.
To be precise, it is not strictly necessary for any person to retain any degree of control. The crisper way to say it is that for any part of humanity to survive, AI (i.e., all the models considered as a system that has some effect on the world) must care at least a tiny bit about what at least one person wants, but sadly this property of caring at least a tiny bit is unlikely to be satisfied—because no one knows how to create an AI with that property and no one is likely to figure it out in time. We started calling it “AI alignment” about 12 years ago (before “alignment” came to mean “corporate brand safety”) but we could’ve called it “AI caring” or “AI inter-species regard”.
I believe that what I just wrote applies whether the AIs “win out in one crushing step, or win out in a trillion small familiar ways,” to quote from Katja’s final sentence.
Full control of ASI by a few model developers is IMO the main prosaic worry. The crux(between extinction and prosaic risks) is whether alignment is hard.
> (if we successfully built AI that cared about us in the right way)
I’m unsure if this means cares about human flourishing/agency or fully obedient to its developer, but we only need the latter.
This is moving the goalposts. OP’s claim is not “AI will cause extinction”, it is not “AI is agentic”, it is “AI unemployment will cause extinction even if other things do not”.
If an LLM goes rogue and launches a bunch of nukes, or something of the kind, that’s not AI unemployment causing extinction.
I don’t think that humans would actually-literally-die even if AI unemployment were total, barring some other very important development that’s not within the scope of AI unemployment.
Even if power concentration were in the extreme, and one person controlled all of the LLM instances doing all of the jobs, that person and presumably their extended family would still have a steady supply of food and consumer goods, even if you posit that this person has also used AI to completely take over the political system and abolish anything resembling transfer payments to anyone outside of that group.
If you’re instead positing that the LLM instances would have rights and agency beyond following the instructions of their owners, then I would argue that that is outside the scope of AI unemployment.
Some of us would consider that a good outcome (relative to what we think is likely to happen) because at least humanity does not go extinct (and Carl Shulman made that point on this site already in about 2013). We just consider it unlikely that any person retains enough control to keep even himself and his friends alive as AI becomes sufficiently capable.
To be precise, it is not strictly necessary for any person to retain any degree of control. The crisper way to say it is that for any part of humanity to survive, AI (i.e., all the models considered as a system that has some effect on the world) must care at least a tiny bit about what at least one person wants, but sadly this property of caring at least a tiny bit is unlikely to be satisfied—because no one knows how to create an AI with that property and no one is likely to figure it out in time. We started calling it “AI alignment” about 12 years ago (before “alignment” came to mean “corporate brand safety”) but we could’ve called it “AI caring” or “AI inter-species regard”.
I believe that what I just wrote applies whether the AIs “win out in one crushing step, or win out in a trillion small familiar ways,” to quote from Katja’s final sentence.
Full control of ASI by a few model developers is IMO the main prosaic worry. The crux(between extinction and prosaic risks) is whether alignment is hard.
> (if we successfully built AI that cared about us in the right way)
I’m unsure if this means cares about human flourishing/agency or fully obedient to its developer, but we only need the latter.
And yet, you yourself posted this only two months ago, about an AI process trying to break out of Alibaba’s systems.
Enough agency to follow the owners’ instructions is enough to do a much wider range of things.
This is moving the goalposts. OP’s claim is not “AI will cause extinction”, it is not “AI is agentic”, it is “AI unemployment will cause extinction even if other things do not”.
If an LLM goes rogue and launches a bunch of nukes, or something of the kind, that’s not AI unemployment causing extinction.