I feel like I understand this topic reasonably well for a casual reader, and I’m trying to convince my friends that they should take the threat seriously and think about it. I haven’t moved the needle on any of them, which actually surprises me. This isn’t really so much a question as just putting out there This is usually where I get stuck when talking to bright people who haven’t considered AGI before:
Them: OK but what is it going to do?
Me: Well I’m not totally sure, but if it’s much more intelligent than us, whatever it will come up with could kill us. Who knows, maybe a virus or something?
Them: OK, but it has to physically DO something. How will it do that?
Me: Again, I’m not sure, but surely it could easily convince someone on the planet to combine chemicals or whatever it needs? Just an example. Again, I’m not sure
Them: Right, but it has to kill us all at once. It can’t physically do anything. The second we see something potentially cataclysmic happening, we will shut it down.
Me: Right, but it KNOWS that. It won’t strike until it knows that our shut-down counterattack won’t work.
Them: OK, but how is it going to get to that state without DOING anything? At some point, it needs to physically manipulate the world, and we are back to our last point.
Me: Surely something so intelligent will be able to come up with an effective plan.
Them: No way. Have you ever tried to implement a plan in reality? It never works, and now you’re telling me that a superintelligence will be able to wait in the weeds until the exact right time when it burst out of hiding and kills all of humanity all at once? You’re crazy
Cool post :)
Twist: It’s actually an AGI who made this post to lull me into one second spent on this god-forsaken website not gripped with fear and anti-AI sentiment.
Just kidding more juneberry content plz
“Knowing that a medium-strength system of inscrutable matrices is planning to kill us, does not thereby let us build a high-strength system of inscrutable matrices that isn’t planning to kill us.”
Maybe if people become convinced of the first clause, people will start destroying GPUs or a war will start or something?
Yeah—for me the difference with an AI is that maybe they could make you live forever. I think it’s trivially obvious that no scenario that ends in death, no matter how gruesome and inhumane, would be sufficient to make us consider suicide just to avoid its possibility. It’s pretty dumb to consider killing yourself to avoid death 🙂
Living forever though might in theory change the calculation.
I 100% agree with you on the EV calculation (I’m still alive after all); it just struck me that I might rather be dead than deal with a semi-malevalent AI.
Point taken though that if we can get an AI to even give a shit about us we can probably also prevent it from keeping us alive for all eternity in the deepest hell imaginable lol
Not the place for this comment, but I’m just fully discovering this topic and thinking about it.
Just to say, I’m an extremely joyful and happy person with a baby on the way so I hope nobody takes this the wrong way—I’m not serious about this, but I think it’s interesting.
Doesn’t the precautionary principle in some way indicate that we should kill ourselves? Everyone seems to agree that AGI is on the way. Everyone also seems to agree that its effects are unpredictable. Imagine an AI who calculates that the best way to keep humans happy is to ensure they follow the AI’s (good!) commands. The AI calculates that the best way to ensure compliance is to take one human being, keep him alive for eternity (!) in a metal box with an opening just big enough for his body, and broadcast that image to anyone who is considering disobeying the AI’s commands.
If something like this is even possible, shouldn’t we just kill ourselves now to avoid the chance that we are that person in the box?
Obviously I have no idea what I’m talking about and this example is preposterous, but surely there is some similarly horrible outcome that we can’t even imagine.
Holy crap love this
Long-term meditator here (~4400 total hours).
I actually think you may have it backwards here: “In the mental realm, the opposite may be true: the average person may be experiencing a pretty thorough mental workout just from day-to-day life”
In my view, mental “exercise” actually requires an absence of stimulation. This is increasingly difficult to find in the modern world, due to email, text, twitter etc.
Also in my view this may be why so many people are complaining of burnout. Boredom I believe may have benefits for mental health, and boredom is declining in our world
Just my two cents—great piece :)
One quick thing is to consider animals—I bet my dog is conscious, but I’m not sure she has “thoughts” as we conceive of them.
I bet you can have thoughts without consciousness though. I’m imagining consciousness as something like a computer program. The program is written such that various sub-modules probabilistically pitch “ideas” based on inputs from the environment, etc. (“Pay more attention to that corner of the room!” “Start running!”) Another module sort of probabilistically “evaluates” these ideas and either initiates behavior or not. With enough complexity and inter-connectedness, consciousness emerges.
In theory, you could write a program where there are only a couple of commands that are being “evaluated.” These are thoughts, but the system isn’t complex enough to be conscious.
Obviously this is all massive bull-shitty conjecture and there a million caveats but that’s my immediate reaction.
Thinking more about the animal example, I might say that you can’t have consciousness without thoughts, but it depends how you define “thoughts”. So I bet that an iguana is conscious, but I bet its thoughts are mostly “ah fuck I hate how this feels” or “oooh nice I like how this feels”. Obviously not in words. I would basically consider these to be thoughts, and as long as you define thoughts like that it seems impossible to have consciousness without thoughts. My guess is that ppl who have taken drug trips would say consciousness can exist without thoughts, but I bet it’s a semantic thing—those trips might create a state of consciousness totally alien to me for instance but I bet it’s still a “thought” given how widely I would define what a thought is.