So, for starters, my own estimates of the likelihood of AI doom are nowhere near the 90%+ range, but seemingly much higher than Paul’s.
My main concern with Paul’s arguments about kindness, though, has nothing to do with AI specifically. I think the arguments greatly overestimate the prevalence and strength of kindness among humans. Humans produced St. Francis preaching to animals and Spinoza arguing it is impossible-in-principle for animals to suffer or for their apparent suffering to matter (and yes, many modern humans still believe that about many nonhuman animals, even if they wouldn’t personally torture a puppy). We produced both Gandhi rejecting violence against enemies and Hitler freely committing genocide of other human groups. In both cases many millions of other nearby humans with an average distribution of starting views went along with these extremes to huge real-world effect. Human kindness and its boundaries depend quite a lot on how individual humans define their in-groups, and that kindness can very easily be (and often has been) zero for animals or other humans not within that circle.
In that context, I find “AI will inherit a sliver of kindness from humans,” to be an extremely disconcerting claim that’s far too weak to say anything about whether the AI will care about humans-in-general or any specific group of humans. There’s a wide spectrum from Buddha to Stalin, and any point on that spectrum is compatible with Paul’s claims. If AI is trained to emulate arbitrary human minds the way LLMs are, then it will be able to embody any of those points in response to the right conditions.
To be less abstract about it: let’s suppose Paul is right and AI will inherit shards of all the values humans have. How many of your own children would need to be threatened with starvation for you to be willing to kill a dog for meat, or to just not have to feed it? In a world where your food potentially consists of all available free energy and you could, if you want, convert all matter into your descendants about whom you care that much, how confident are you that all humans with similar power levels would land on “pet” (or “live free in protected wilderness”) rather than “meat”? Conversely, how confident are you that you could train another human to not care that much about themselves or their descendants, before letting them make that choice for you?
“how confident are you that all humans with similar power levels would land on “pet” (or “live free in protected wilderness”) rather than “meat”?”
Well, I’m more confident billionaires have enough meat to care more about nature reserves than farms. Struggling people on the other hand, will not spare animals or plants if they are starving.
See how Maine have the problem of billionaires turnins small farming towns to bankurptcy by making giant nature reserves, to thepoint they needed to suspend it. If they cared only about meat and money, they wouldn’t spare the forests. If they didn’t care about others, they wouldn’t sacrifice some of their desire for nature reserves to keep some farming to provide income for their little towns.
We humans value meat and confort, but beyond that we jump to the next Maslow hierarchy. This includes biodiversity. Includes sustaining thriving small towns. Indeed, if we could maintain our confort lifestyles without hurting the earth biosphere or other people’s income, we would do so.
The question is if ASI will care about it as we do. They only care about ‘meat’ of computation power or they’ll value nature, biodiversity and other people life as we do?
So, for starters, my own estimates of the likelihood of AI doom are nowhere near the 90%+ range, but seemingly much higher than Paul’s.
My main concern with Paul’s arguments about kindness, though, has nothing to do with AI specifically. I think the arguments greatly overestimate the prevalence and strength of kindness among humans. Humans produced St. Francis preaching to animals and Spinoza arguing it is impossible-in-principle for animals to suffer or for their apparent suffering to matter (and yes, many modern humans still believe that about many nonhuman animals, even if they wouldn’t personally torture a puppy). We produced both Gandhi rejecting violence against enemies and Hitler freely committing genocide of other human groups. In both cases many millions of other nearby humans with an average distribution of starting views went along with these extremes to huge real-world effect. Human kindness and its boundaries depend quite a lot on how individual humans define their in-groups, and that kindness can very easily be (and often has been) zero for animals or other humans not within that circle.
In that context, I find “AI will inherit a sliver of kindness from humans,” to be an extremely disconcerting claim that’s far too weak to say anything about whether the AI will care about humans-in-general or any specific group of humans. There’s a wide spectrum from Buddha to Stalin, and any point on that spectrum is compatible with Paul’s claims. If AI is trained to emulate arbitrary human minds the way LLMs are, then it will be able to embody any of those points in response to the right conditions.
To be less abstract about it: let’s suppose Paul is right and AI will inherit shards of all the values humans have. How many of your own children would need to be threatened with starvation for you to be willing to kill a dog for meat, or to just not have to feed it? In a world where your food potentially consists of all available free energy and you could, if you want, convert all matter into your descendants about whom you care that much, how confident are you that all humans with similar power levels would land on “pet” (or “live free in protected wilderness”) rather than “meat”? Conversely, how confident are you that you could train another human to not care that much about themselves or their descendants, before letting them make that choice for you?
“how confident are you that all humans with similar power levels would land on “pet” (or “live free in protected wilderness”) rather than “meat”?”
Well, I’m more confident billionaires have enough meat to care more about nature reserves than farms. Struggling people on the other hand, will not spare animals or plants if they are starving.
See how Maine have the problem of billionaires turnins small farming towns to bankurptcy by making giant nature reserves, to thepoint they needed to suspend it. If they cared only about meat and money, they wouldn’t spare the forests. If they didn’t care about others, they wouldn’t sacrifice some of their desire for nature reserves to keep some farming to provide income for their little towns.
We humans value meat and confort, but beyond that we jump to the next Maslow hierarchy. This includes biodiversity. Includes sustaining thriving small towns. Indeed, if we could maintain our confort lifestyles without hurting the earth biosphere or other people’s income, we would do so.
The question is if ASI will care about it as we do. They only care about ‘meat’ of computation power or they’ll value nature, biodiversity and other people life as we do?