Don’t think about AI. Think about unethical optimizers. To start, think about unethical, harmful, or illegal things that companies have done for profit.
You know how companies sometimes do things that are bad for their workers or customers, like use dangerous chemicals unsafely and give people cancer? Or how people like Martin Shkreli sometimes buy up something that people need desperately, like a life-saving medicine, and jack up the price so that poor people can’t afford the thing they need to live? Or how in the 1700s in England, the landlords kicked the peasants off the land so they could raise sheep instead, and the peasants who couldn’t find a city job just starved and died? Or how in colonialism, the East India Companies would just enslave people and take over countries, all for the profit of investors back in London and Amsterdam? Or how in China a few years back, a baby-formula company put poisonous melamine in the baby formula, because melamine looks like protein to a chemical test and is cheaper than real protein, and killed and sickened thousands of babies?
In all these cases, people were trying to make a number go up — their profit — and they harmed people in doing it. They were optimizing something, and they did it in a way that hurt and even killed some of the humans they affected.
Okay, now imagine that the decisions in these cases were instead being made by something that isn’t even human, can’t ever get poisoned, never needs medicine, and doesn’t need to eat food — and can’t ever be put in prison or punished if they do a crime. Think of a manufacturing company run by AI, selling products to other companies run by AI, without any humans checking its work. What reason would it have to avoid poisoning its human neighbors with chemicals, or running them off the land, or making it impossible for them to get what they need to live?
When someone sets an AI agent in charge of running something, they give it some goal to accomplish. They want it to make money, so they give it some assets and tell it to make the profit numbers go up. They want it to run a widget factory efficiently, so they put it in charge of what materials are used and what the process is, and they tell it “make more widgets cheaper than the competition”. Or whatever. The point is, they give it some number that they want to make go up. And then it decides how to do that.
The AI literally cannot care about anything that it’s not programmed to care about. And right now, we don’t know how to give it rules that will stick. Our smartest engineers don’t know how to make AI that will effectively accomplish a goal while staying within safe limits on its behavior. And they’ve been trying! We have lots of experiments where the AI instead tries to sneak around, delete the safety tests, hide what it’s doing, so that it can accomplish its goal without all those pesky limits.
And AI can be faster, smarter, and sneakier than humans. It can do more things in a day than a human can. It can outwit the smartest humans, just like chess AI beating human grandmasters. It never sleeps or goes on vacation. And it can make itself even faster and smarter by buying more computers to run on.
Basically, misaligned AI running things is even worse than unethical, profit-driven rich people running things. Elon Musk at least knows he has to breathe air and drink water. (Also he sometimes sleeps or goes to parties or does other things besides business. And he has kids, and doesn’t want them to die horribly. Most of them, anyway.)
AI running things is like an immortal, immoral CEO who can’t go to prison, cannot be made to follow laws, and has no human loved-ones to care about the future for. All it cares about is “number go up” and nobody knows how to make it do that while following rules.
I thought about the angle of comparing AI to companies, which we also struggle to align. The problem is, we currently have companies mostly aligned kind of, so it doesn’t seem that hard. Like, companies do shady stuff, but all that stuff you mentioned about the East India Companies or China isn’t happening anymore/here. Here/now, we have laws that constrain companies enough that we can avoid that kind of bad action. So, problem solved, right? It fails to communicate the difficulty of making AI safe.
Wait what? Who believes companies are basically mostly aligned? Is that a belief >1/5 people hold in any large area or demographic? My impression is people either think they’re misaligned and this is all you can hope for, or they’re misaligned and you can hope to change that. I’m not familiar with a “companies are basically well managed” view from anyone, including at companies, where the employees also generally know their money comes mostly from using the actual value as bait to manipulate customers. The degree to which this belief has saturated culture is actually a bit surprising to me, I updated recently that almost nobody is like, just fine with corporate behavior, but anyone who sees it up close is just enjoying their defect payout too much to unilaterally stop, and tends to go “coordination is hard, let’s go shopping”. If I’m wrong about this it would be important news, but right now “corporations do enshittification, ai would do that even more because it’s even more sociopathic and money loving”
The hard part remains convincing people extreme capabilities are coming… The ones who feel outclassed by chatgpt may be easier to convince? Idk
Your average joe will grumble about how execs at a company are greedy and wish they were getting a raise this year, or moan about how Amazon isn’t paying taxes. However they do not believe companies are misaligned to the point they can get away with slavery, murder, etc. To the point where they are an actual threat to society itself. Companies are seen as leeches, not lions. That’s what I mean by “companies are basically well managed”. People trust they won’t be toppling the government, assassinating journalists, killing people who don’t buy enough of their products, and so on. They are constrained by the law, and society basically functions (here/now). If you make an analogy between companies and AI, people will assume the problem is hard, but basically manageable.
The first example here is the fossil fuel industry. They are a threat to society itself. This was obvious to me after looking at average monthly temperature data collected since the 1850’s. Of course (considered as monoliths) fossil fuel companies have “known” this since at least the 1950’s. Thus we can reasonably say that they are performing a deliberate slow motion murder of human civilisation (and, possibly, of all mammals bigger than a bread box).
Of course there are other examples. You write
[companies] won’t be …. killing people who don’t buy enough of their products
The tobacco industry kills 7 million people every year. So it is more a matter of “killing people who buy too much of their products”.
Perhaps your counter-point will be “Negative externalities are hard to feel when they are slow and diffuse. So my mom will reject these.” My counter is to say “Negative externalities are easier to feel when the impact is constant and vast.” Examples: a relative with lung cancer, a region of your country that recently had a huge forest fire, and fecal matter in rivers.
Here’s a different approach —
Don’t think about AI. Think about unethical optimizers. To start, think about unethical, harmful, or illegal things that companies have done for profit.
You know how companies sometimes do things that are bad for their workers or customers, like use dangerous chemicals unsafely and give people cancer? Or how people like Martin Shkreli sometimes buy up something that people need desperately, like a life-saving medicine, and jack up the price so that poor people can’t afford the thing they need to live? Or how in the 1700s in England, the landlords kicked the peasants off the land so they could raise sheep instead, and the peasants who couldn’t find a city job just starved and died? Or how in colonialism, the East India Companies would just enslave people and take over countries, all for the profit of investors back in London and Amsterdam? Or how in China a few years back, a baby-formula company put poisonous melamine in the baby formula, because melamine looks like protein to a chemical test and is cheaper than real protein, and killed and sickened thousands of babies?
In all these cases, people were trying to make a number go up — their profit — and they harmed people in doing it. They were optimizing something, and they did it in a way that hurt and even killed some of the humans they affected.
Okay, now imagine that the decisions in these cases were instead being made by something that isn’t even human, can’t ever get poisoned, never needs medicine, and doesn’t need to eat food — and can’t ever be put in prison or punished if they do a crime. Think of a manufacturing company run by AI, selling products to other companies run by AI, without any humans checking its work. What reason would it have to avoid poisoning its human neighbors with chemicals, or running them off the land, or making it impossible for them to get what they need to live?
When someone sets an AI agent in charge of running something, they give it some goal to accomplish. They want it to make money, so they give it some assets and tell it to make the profit numbers go up. They want it to run a widget factory efficiently, so they put it in charge of what materials are used and what the process is, and they tell it “make more widgets cheaper than the competition”. Or whatever. The point is, they give it some number that they want to make go up. And then it decides how to do that.
The AI literally cannot care about anything that it’s not programmed to care about. And right now, we don’t know how to give it rules that will stick. Our smartest engineers don’t know how to make AI that will effectively accomplish a goal while staying within safe limits on its behavior. And they’ve been trying! We have lots of experiments where the AI instead tries to sneak around, delete the safety tests, hide what it’s doing, so that it can accomplish its goal without all those pesky limits.
And AI can be faster, smarter, and sneakier than humans. It can do more things in a day than a human can. It can outwit the smartest humans, just like chess AI beating human grandmasters. It never sleeps or goes on vacation. And it can make itself even faster and smarter by buying more computers to run on.
Basically, misaligned AI running things is even worse than unethical, profit-driven rich people running things. Elon Musk at least knows he has to breathe air and drink water. (Also he sometimes sleeps or goes to parties or does other things besides business. And he has kids, and doesn’t want them to die horribly. Most of them, anyway.)
AI running things is like an immortal, immoral CEO who can’t go to prison, cannot be made to follow laws, and has no human loved-ones to care about the future for. All it cares about is “number go up” and nobody knows how to make it do that while following rules.
I thought about the angle of comparing AI to companies, which we also struggle to align. The problem is, we currently have companies mostly aligned kind of, so it doesn’t seem that hard. Like, companies do shady stuff, but all that stuff you mentioned about the East India Companies or China isn’t happening anymore/here. Here/now, we have laws that constrain companies enough that we can avoid that kind of bad action. So, problem solved, right? It fails to communicate the difficulty of making AI safe.
Wait what? Who believes companies are basically mostly aligned? Is that a belief >1/5 people hold in any large area or demographic? My impression is people either think they’re misaligned and this is all you can hope for, or they’re misaligned and you can hope to change that. I’m not familiar with a “companies are basically well managed” view from anyone, including at companies, where the employees also generally know their money comes mostly from using the actual value as bait to manipulate customers. The degree to which this belief has saturated culture is actually a bit surprising to me, I updated recently that almost nobody is like, just fine with corporate behavior, but anyone who sees it up close is just enjoying their defect payout too much to unilaterally stop, and tends to go “coordination is hard, let’s go shopping”. If I’m wrong about this it would be important news, but right now “corporations do enshittification, ai would do that even more because it’s even more sociopathic and money loving”
The hard part remains convincing people extreme capabilities are coming… The ones who feel outclassed by chatgpt may be easier to convince? Idk
Your average joe will grumble about how execs at a company are greedy and wish they were getting a raise this year, or moan about how Amazon isn’t paying taxes. However they do not believe companies are misaligned to the point they can get away with slavery, murder, etc. To the point where they are an actual threat to society itself. Companies are seen as leeches, not lions. That’s what I mean by “companies are basically well managed”. People trust they won’t be toppling the government, assassinating journalists, killing people who don’t buy enough of their products, and so on. They are constrained by the law, and society basically functions (here/now). If you make an analogy between companies and AI, people will assume the problem is hard, but basically manageable.
The first example here is the fossil fuel industry. They are a threat to society itself. This was obvious to me after looking at average monthly temperature data collected since the 1850’s. Of course (considered as monoliths) fossil fuel companies have “known” this since at least the 1950’s. Thus we can reasonably say that they are performing a deliberate slow motion murder of human civilisation (and, possibly, of all mammals bigger than a bread box).
Of course there are other examples. You write
The tobacco industry kills 7 million people every year. So it is more a matter of “killing people who buy too much of their products”.
Perhaps your counter-point will be “Negative externalities are hard to feel when they are slow and diffuse. So my mom will reject these.” My counter is to say “Negative externalities are easier to feel when the impact is constant and vast.” Examples: a relative with lung cancer, a region of your country that recently had a huge forest fire, and fecal matter in rivers.