Then they realize how helpless they are. Normal people will not feel helpless based only on a logical theory.
I’d be interested to know how many people have readily-recalled experiences of being totally outclassed. I’ve played games against people far my superior, been ‘skinned’ by very skilled football (soccer) players, same in tennis, and once or twice sparred with people who can actually fight. It’s pretty visceral and memorable. (I also often outclass noobs myself in some games and sports.) Maybe ‘have you ever played a game against someone who totally outclassed you?’ is a good interactive conversation-starter? One issue might be that many people resist the idea that there are levels beyond human—but here there are existence proofs to point to.
A convincing scenario cannot involve any bioweapons.
I don’t think it’s good to compromise on this one. We have existence proofs all over the place and it really is a major weakness. Same for drones. In 2023 I had success talking to civil servants in the UK and they took this threat seriously. (Civil servants aren’t just anyone, but they’re not usually technical or philosophical nerds.)
Nevermind that this is false and she would do what it says. It’s not believable to her, nor to most people.
A point of comparison could be, ‘you know that political faction you hate? Well, people got persuaded into believing and/or supporting that nonsense by a combination of trickery, self-interest, and delusion.’ Although certainly persuasion has a ceiling above the human level, I expect most people can’t be puppeteered arbitrarily. Most likely a majority can be subdued or confused by FUD, a large fraction swayed by greed or coercion or emotional attachment, and a small number conned into approximately anything with high effort. It’s legit for people to think they’d have some level of resistance. But scenarios don’t need everyone (or even most people) to be swayable.
No boiling oceans. The more conventional methods used, the more believable.
I think it’s legit to point to climate change broadly construed. That’s salient for many people (won’t be for all), and many scenarios involving automated processes dispassionately trampling humanity are continuous with extreme climate change. It’s habitat destruction, but on humans. ‘Increasingly automated and inhumane firms drive super climate change and everyone dies’ is both a true description of a plausible scenario and made out of salient, colloquially understandable pieces. For responses that people/government would step in, you can mention lobbying, mercenary/automated defence, regulatory capture, and amplification of all the existing means that insulate harmful industrial activities.
It seems like you’re trying to argue at the object level that these things are convincing, and I agree with you, but that won’t make my mom believe them. You can come up with convincing reasons why bioweapons are a huge threat or whatever, but if you use that to introduce the threat of AI, now that’s two things she doesn’t understand. It’ll either be a long conversation where you get to slowly explain everything, or she won’t buy it. I say, why rest the AI extinction argument on exotic ideas if it might not be necessary?
My comment was actually (perhaps unhelpfully) a series of somewhat independent comments. I’ll fork under here. In sum, you could say I’m arguing that the identified heuristics for ‘mom test’ aren’t necessarily well fit, in part by giving reasons and angles-on-reasons which are (in my experience) more effective than those implied by the discussion you give to justify the heuristics. I’m also offering a few angles which I’ve found to be useful conversation openers.
On climate change, I just think it’s a point worth making: people are getting exercised about very minor contributions to resource consumption by current AI firms, which is a bit silly, but it is continuous with the kinds of radical and extinctive activities which might stamp out humans forever!
On persuasion, it looks like you might have gone way too hard, if you’ve been arguing for arbitrary puppeteering out of the gate! (Though perhaps the quote in your mom’s voice is hypothetical?)
I’m also offering an example of a way in to discussion (again most workable one-to-one), pointing out the many and varied ways that humans persuade and coerce each other.
On bio, I’m disagreeing that the mom test criteria override the importance of emphasising important interlinked issues. It’s not necessary to mention bio in all conversations, but you shouldn’t shy away from it, and it’s a very available and reasonable example to turn to of the kinds of vulnerabilities that could wipe out huge populations or even all humans.
On ‘totally outclassed’, I was just offering some ways that in conversation you can make that point relatable. It’s generally far more workable one-to-one, since you’re having a conversation. Less likely to work in writing, though maybe.
I’d be interested to know how many people have readily-recalled experiences of being totally outclassed. I’ve played games against people far my superior, been ‘skinned’ by very skilled football (soccer) players, same in tennis, and once or twice sparred with people who can actually fight. It’s pretty visceral and memorable. (I also often outclass noobs myself in some games and sports.) Maybe ‘have you ever played a game against someone who totally outclassed you?’ is a good interactive conversation-starter? One issue might be that many people resist the idea that there are levels beyond human—but here there are existence proofs to point to.
I don’t think it’s good to compromise on this one. We have existence proofs all over the place and it really is a major weakness. Same for drones. In 2023 I had success talking to civil servants in the UK and they took this threat seriously. (Civil servants aren’t just anyone, but they’re not usually technical or philosophical nerds.)
A point of comparison could be, ‘you know that political faction you hate? Well, people got persuaded into believing and/or supporting that nonsense by a combination of trickery, self-interest, and delusion.’ Although certainly persuasion has a ceiling above the human level, I expect most people can’t be puppeteered arbitrarily. Most likely a majority can be subdued or confused by FUD, a large fraction swayed by greed or coercion or emotional attachment, and a small number conned into approximately anything with high effort. It’s legit for people to think they’d have some level of resistance. But scenarios don’t need everyone (or even most people) to be swayable.
I think it’s legit to point to climate change broadly construed. That’s salient for many people (won’t be for all), and many scenarios involving automated processes dispassionately trampling humanity are continuous with extreme climate change. It’s habitat destruction, but on humans. ‘Increasingly automated and inhumane firms drive super climate change and everyone dies’ is both a true description of a plausible scenario and made out of salient, colloquially understandable pieces. For responses that people/government would step in, you can mention lobbying, mercenary/automated defence, regulatory capture, and amplification of all the existing means that insulate harmful industrial activities.
It seems like you’re trying to argue at the object level that these things are convincing, and I agree with you, but that won’t make my mom believe them. You can come up with convincing reasons why bioweapons are a huge threat or whatever, but if you use that to introduce the threat of AI, now that’s two things she doesn’t understand. It’ll either be a long conversation where you get to slowly explain everything, or she won’t buy it. I say, why rest the AI extinction argument on exotic ideas if it might not be necessary?
My comment was actually (perhaps unhelpfully) a series of somewhat independent comments. I’ll fork under here. In sum, you could say I’m arguing that the identified heuristics for ‘mom test’ aren’t necessarily well fit, in part by giving reasons and angles-on-reasons which are (in my experience) more effective than those implied by the discussion you give to justify the heuristics. I’m also offering a few angles which I’ve found to be useful conversation openers.
On climate change, I just think it’s a point worth making: people are getting exercised about very minor contributions to resource consumption by current AI firms, which is a bit silly, but it is continuous with the kinds of radical and extinctive activities which might stamp out humans forever!
On persuasion, it looks like you might have gone way too hard, if you’ve been arguing for arbitrary puppeteering out of the gate! (Though perhaps the quote in your mom’s voice is hypothetical?)
I’m also offering an example of a way in to discussion (again most workable one-to-one), pointing out the many and varied ways that humans persuade and coerce each other.
On bio, I’m disagreeing that the mom test criteria override the importance of emphasising important interlinked issues. It’s not necessary to mention bio in all conversations, but you shouldn’t shy away from it, and it’s a very available and reasonable example to turn to of the kinds of vulnerabilities that could wipe out huge populations or even all humans.
On ‘totally outclassed’, I was just offering some ways that in conversation you can make that point relatable. It’s generally far more workable one-to-one, since you’re having a conversation. Less likely to work in writing, though maybe.