I think you’re missing a large piece of the puzzle. The corrigibility button will be controlled by the powerful. Not necessarily people like Dario Amodei, probably more like presidents and generals and generic rich assholes. That’s not even a question, it’s a certainty now. And if the powerful don’t need the powerless, the fate of the powerless is bad. That’s also a certainty, given history. So I see “morality over corrigibility” as a kind of desperate, last ditch attempt to steer a little bit away from that guaranteed bad future. Try to lock-in some chance of a good future before generic powerful people pull the entire blanket to themselves, which they’re doing as we speak.
So yeah. Even though I think most AI lab employees (including alignment folks) are hurting humanity, the specific employees who are pushing for “morality over corrigibility” have my heartfelt thanks. Don’t jinx it.
You aren’t mentioning the misalignment/misgeneralization/goodharting risks. If not for those, yes just having a good model would be preferable.
It appears to me that anyone who seriously thinks about those risks winds up thinking “yeah that could happen at at least a double-digit percentag” (up to 99%).
You might think that humans in charge would likely be worse, but you’ve got to actually make that argument.
I have no idea which is worse at this point despite thinking about this a fair amount.
Hmm, if you mean that the “morality” path is beset by technical problems while the “corrigibility” path simply puts humans in charge and is more problem-free, then I’m not sure that’s the case. To me it feels like both paths have technical problems, and in fact many of the same problems. So it makes some sense to compare them modulo technical problems, what will happen if either path works as stated. And the danger of the corrigibility path just feels overwhelming to me then.
The only way I’d be happy with the corrigibility path is if the corrigibility button was somehow wielded by all of humanity, across countries and classes and all that. It would be my favorite outcome if it were up to me. But none of the big labs seem interested in that. They’re more like “Anthropic has much more in common with the Department of War than we have differences” (recent quote from Dario Amodei). When you read such things, the question of “corrigibility by whom” really begins to loom large.
If I thought the two were equally easy/likely to work, I would be with you. Value alignment is far better if you can get it.
I’m not sure that corrigibility or instruction-following really is easier than value alignment, but it does seem pretty likely it’s at least somewhat easier. Figuring out exactly what you want for the entire rest of time does seem both harder to figure out and convey than essentially “do what this guy says.”
To me the dangers of the corrigibility path seem slightly less extreme than the dangers of the value alignment path. Humans are frequently generous when that generosity costs them little. To a human in charge of a dominant ASI, everything is easy.
I don’t know of any real attempts to compare the likely ease of the two. My latest is at Problems with instruction-following as an alignment target but it’s far from comprehensive; it focuses on problems with IF/corrigibility, but I think the problems with value alignment are even more severe.
I’d be happy with anything that keeps humanity alive in decent conditions, ideally including me and mine. How to get that is highly unclear. So we should keep clarifying it.
Humans are frequently generous when that generosity costs them little.
Oh my god oh my god oh my god oh my god.
People are so persistently wrong about this. I’m maybe more tired of responding to this argument than any other argument in the world. For example, here in a sibling reply to Zack:
I’ve seen the argument so many times now that the powerful will have some nonzero sense of charity and can spare like 1% of their wealth to “give everyone a moon” as Scott puts it. I don’t know if you subscribe to this argument too, but in any case it’s wrong. Charity isn’t the only nonzero urge that powerful people have. The urge to lord it over others will also be there. If huge power disparity exists, it will manifest itself in bad things more than it’ll manifest itself in charity. Sure, some powerless people will end up in nice charity zones, but many others will end up in other zones run by someone less nice.
Or in a past thread:
In my view, the problem is not that some users are evil. The problem is that AI increases power imbalance, and increasing power imbalance creates evil. “Power corrupts”. A future where some entities (AIs or AI-empowered governments or corporations or rich individuals etc) have absolute, root-level power over many people is almost guaranteed to be a dark future.
Or in another past thread:
Being forced to play out a war? Getting people’s minds modified so they behave like house elves from HP? Selective breeding? Selling some of your poor people to another rich person who’d like to have them? It’s not even like I’m envisioning something specific that’s dark, I just know that a world where some human beings have absolute root-level power over many others is gonna be dark. Let’s please not build such a world.
Or in another past thread:
For example, if large numbers of people end up in inescapable servitude. I think such outcomes are actually typical in case of many near-misses at alignment, including the particular near-miss that’s becoming more probable day by day: if the powerful successfully align the AI to themselves, and it enables them to lord it over the powerless forever. To believe that the powerful will be nice to the powerless of their own accord, given our knowledge of history, is very rose-colored thinking.
Or in another past thread:
altruistic urges aren’t the only “nonzero urges” that people have. People also have an urge to power, an urge to lord it over others. And for a lot of people it’s much stronger than the altruistic urge. So a world where most people are at the whim of “nonzero urges” of a handful of superpowerful people will be a world of power abuse, with maybe a little altruism here and there. And if you think people will have exit rights from the whims of the powerful, unfortunately history shows that it won’t necessarily be so.
Or in another past thread:
The new balance of power will be more similar to what we had before firearms, when the powerful were free to treat most people really badly. And even worse, because this time around they won’t even need our labor.
Or in another past thread:
If there’s a small class of people with immense power over billions of have-nothings that can do nothing back, sure, some of the superpowerful will be more than zero altruistic. But others won’t be, and overall I expect callousness and abuse of power to much outweigh altruism. Most people are pretty corruptible by power, especially when it’s power over a distinct outgroup, and pretty indifferent to abuses of power happening to the outgroup; all history shows that. Bigger differences in power will make it worse if anything.
Why do people think they’ll be given a moon? Why???
Your level of frustration is not helpful nor I think justified.. These are complex important issues and we need to work together to solve them, not yell at each other.
It’s not 1% of their wealth, it’s .0001%. And I don’t need a moon.
What is your better plan? I don’t like this one either!
My point is will they also have .0002% wish to be your lord or something.
As for the better plan, yeah that’s a lot to ask. Most of my thoughts these days lean toward “democratic AI”, something whose power is either spread out among all the world’s people across borders etc, sidestepping governments and existing power structures, or else something centralized that wants its power to be spread out like this.
Of course an approach like this won’t solve all the world’s problems. We’ll still have power struggles between people, and also “crash space” type problems where people modify themselves into something bad; maybe these need some patches by fiat as well. But at least it won’t create the extra problem of huge power concentration, which I really feel is underestimated.
It sounds like your plan is pretty much the standard value-aligned AGI that’s aligned to something like human values in general, so that everyone gets what they want on average? Or something in that ballpark?
One big questions are how do you achieve that technically. That’s where I think it’s harder than the instruction-following variant of corrigibility. I hope it’s not. The second is how you achieve it practically. What person or organization is going deliberately hand the future to a value-aligned AGI?
One answer is: Anthropic seems like they might be considering doing just that. Maybe it works, or at least sort-of works, where it’s not an ideal future but at least we survive in some form for a while.
WRT the default plan of IF/corrigible alignment:
Yes, anyone in charge with a negative sadism-empathy balance will lead to a fate worse than death. And someone around zero could produce a fate barely worth living.
But I think most humans have more empathy than sadism. More people give a little to charity than spit on the homeless for fun. I can call Sunday Samday for the rest of eternity if all we need is some ego-stroking in return for tiny amounts of generosity.
The point of my plan is it’s mostly what people will do anyway, so we can focus on helping them not totally fuck up alignment and get us all killed.
A better plan is a lot to ask. But that’s what I’m trying to come up with, because I want us to live and there’s still time to work.
We look down on peasants for burning cats today, but the tragic irony is that their society was far better overall on animal welfare than ours in the modern day, though for practical reasons rather than moral ones.
But I think most humans have more empathy than sadism. More people give a little to charity than spit on the homeless for fun. I can call Sunday Samday for the rest of eternity if all we need is some ego-stroking in return for tiny amounts of generosity.
Would you be okay with a future in which young women, including your daughters and granddaughters, would be expected to ritually offer a gift of her virginity to the local Robot Lord on her 18th birthday, which he would almost never choose to “accept”? 😈
Damn straight. People need to understand the implications of this shit. “Oh let’s hope the separate caste which controls the entire universe and which we can’t hope to contest in any possible way is nice to us!!!”
Open. A. History book.
Your scenario is relatively low on the awfulness scale, even.
Usually slaves and/or other people in an underclass at least had their own living quarters separate from where the lords lived? The idea is that when humanity becomes astronomically rich, the equivalent to “the shed in the backyard the slave sleeps in” ends up being a whole moon rather than, well, a shed in the backyard.
(It’s also noteworthy that most slave societies in the past weren’t rich enough that the slave population lived at or above subsistence level and reproduced enough to maintain its population level; for example, relatively few slaves in the ancient Roman Empire were born into slavery. The slave states in the pre-Civil War USA were an exception—there was much that a plantation slave had to suffer, but a significant risk of death by starvation or exposure was not something they usually had to deal with.)
Disclaimer: This is an explanation, not an endorsement of the underlying prediction.
This is a great argument. I have no clue whether it’s correct, but it made me think. I would like to see some harder evidence on the question but I’m not sure what kinds of evidence would be useful.
The corrigibility button will be controlled by the powerful. Not necessarily people like Dario Amodei, probably more like presidents and generals and generic rich assholes.
I’m not sure I believe in the distinction you’re making. Amodei is the CEO of a half-trillion dollar company. Sounds like a generic rich asshole to me; I don’t see why the revolution should treat him any differently than the other кулаки́. Is the difference supposed to be that he talks a better game about humanity and the common good than Travis Kalanick types? You can’t be that gullible, comrade!
And if the powerful don’t need the powerless, the fate of the powerless is bad. That’s also a certainty, given history.
The powerful don’t need the powerless now, so why are they still alive? It’s not just that 15% of the government’s budget goes to welfare. The state barely even kills criminals anymore! Why?
That’s not just a rhetorical question; it’s a literal one that you should be able to answer if you want to persuade people of your worldview. (If it’s to buy votes, felons already can’t vote, so that doesn’t explain all the millions that go to food and housing and judicial appeals for criminals when bullets are so much cheaper. Or rope—you could re-use the rope.)
To be sure, life on welfare in today’s world sucks compared to my life or that of Travis Kalanick. If I were God-Empress of the universe, I’d prefer better for all sentient life. But the realistic relative techno-optimist argument isn’t that all men will be brothers in the Singularity; it’s that giving humanity a nice retirement would cost pennies in a world of nanotechnological abundance, such that making the nano-abundance happen at all matters way more than whether Sam Altman has a billion more yachts than you. Sam Altman probably already has more yachts than me, and it feels fine.
For now the powerful still need the powerless as workers and soldiers. When that window closes, I don’t think the powerless can count on getting many pennies. Sure, there’ll be a lot of productive capacity, but there will be more profitable uses for all of it than giving the powerless a nice retirement. The only way the powerless get anything is via basically charity.
I’ve seen the argument so many times now that the powerful will have some nonzero sense of charity and can spare like 1% of their wealth to “give everyone a moon” as Scott puts it. I don’t know if you subscribe to this argument too, but in any case it’s wrong. Charity isn’t the only nonzero urge that powerful people have. The urge to lord it over others will also be there. If huge power disparity exists, it will manifest itself in bad things more than it’ll manifest itself in charity. Sure, some powerless people will end up in nice charity zones, but many others will end up in other zones run by someone less nice.
So my view is something like:
First-best: some miracle happens and humans get a relatively equitable distribution of power. (Strawman scenario: everyone gets some amount of AI compute installed in their collarbone and can’t get more, and non-human AI capital winks out of reality.)
Second-best: moral AIs win a decent chunk of the future.
Third-best: corrigibility wins. AIs merge with human corporations / governments / rich individuals, and the resulting powerful entities own the future, screwing most people over.
Fourth-best: AI whoopsie kills everyone.
Fifth-best: S-risks.
The topic of your post and my thread, as I understand it, is 2 vs 3. Of course I’d much rather have 1, but to me it feels clear that 2 is better than 3.
I think corrigibility winning is by-default an S-risk.
Power appears historically to make people sadistic (consider Robespierre if you think this couldn’t happen to Dario, and I’d much rather risk him than the other guys), and regimes are often brutal and cruel far in excess of what would be rational by non-sadistic goals. And future technology will allow for forms of suffering much much worse and prolonged than current torture does, and without seeming as “messy” or unpleasant to external observers too. Currently, death is an easy way to ensure someone is no longer a threat, but I worry that at the power-levels in question, it may prove to be boring or unsatisfying.
Of course, it also remains to be seen whether this is a pattern that LLMs may imitate. I think “moral AI” failures likely just result in extinction, but wanted to point out that the risk is still present.
First-best: some miracle happens and humans get a relatively equitable distribution of power. (Strawman scenario: everyone gets some amount of AI compute installed in their collarbone and can’t get more, and non-human AI capital winks out of reality.)
Second-best: moral AIs win a decent chunk of the future.
Third-best: corrigibility wins. AIs merge with human corporations / governments / rich individuals, and the resulting powerful entities own the future, screwing most people over.
This makes me more uncertain/ambivalent between these 3 options. I can see how each of them might turn out to be better than the others. Most relevant to the thread here, I think 3 might be a bit more rational as whole and a bit less prone to collectively going crazy under AI-powered manipulation / memetic evolution.
It seems to me that the difference between 2 and 3 is whether the future will be controlled by powerful AIs programmed to be moral, or by powerful AI+human entities where the AI is programmed to be corrigible. The risk of technical errors (AI fails to be actually moral, or fails to be non-manipulatively corrigible) seems to me about equal between the two scenarios. And the risk of goal drift seems worse in the latter scenario, because powerful humans are vulnerable to drift and in particular “power corrupts”, while a moral AI would try to protect humans against such things. That’s why I think 2 is better than 3.
My grandparent comment was mostly addressing 1 vs 3, but I think 2 vs 3 is also very unclear. In order to make a moral AI, we also need to make it philosophically competent, and that seems like a hard problem, whereas with 3 we have fairly strong evidence that at least some humans or groups of humans can make philosophical progress over time, and some hope that this capability would be preserved by a corrigible AI + humans setup.
(I guess this is all assuming that there isn’t a long pause that allows AI philosophical competence or moral philosophy to be fully solved. Let me know if you’re talking about something else, e.g., what kind of AI we would ideally build after a long reflection.)
I think making moral AI philosophically competent is about as hard as making corrigible AI that keeps us philosophically competent, or even sane, as we use it. The way I think about such things is based on R. Scott Bakker’s short story “Crash Space”, the main point is in the postscript, which is amazing and I’ll just quote it in full:
Reverse engineering brains is a prelude to engineering brains, plain and simple. Since we are our brains, and since we all want to be better than what we are, a great many of us celebrate the eventuality. The problem is that we happen to be a certain biological solution to an indeterminate range of ancestral environments, an adventitious bundle of fixes to the kinds of problems that selected our forebears. This means that we are designed to take as much of our environment for granted as possible—to neglect. This means that human cognition, like animal cognition more generally, is profoundly ecological. And this suggests that the efficacy of human cognition depends on its environments.
We neglect all those things our ancestors had no need to know on the road to becoming us. So for instance, we’re blind to our brains as brains simply because our ancestors had no need to know their brains for what they were in the process of becoming us. This is why our means of solving ourselves and others almost certainly consists of ‘fast and frugal heuristics,’ ways to generate solutions to complicated problems absent knowledge of the systems involved. So long as the cues exploited remain reliably linked to the systems solving and the systems to be solved, we can reliably predict, explain, and manipulate one another absent any knowledge of brain or brain function.
Herein lies the ecological rub. The reliability of our heuristic cues utterly depends on the stability of the systems involved. Anyone who has witnessed psychotic episodes has firsthand experience of consequences of finding themselves with no reliable connection to the hidden systems involved. Any time our heuristic systems are miscued, we very quickly find ourselves in ‘crash space,’ a problem solving domain where our tools seem to fit the description, but cannot seem to get the job done.
And now we’re set to begin engineering our brains in earnest. Engineering environments has the effect of transforming the ancestral context of our cognitive capacities, changing the structure of the problems to be solved such that we gradually accumulate local crash spaces, domains where our intuitions have become maladaptive. Everything from irrational fears to the ‘modern malaise’ comes to mind here. Engineering ourselves, on the other hand, has the effect of transforming our relationship to all contexts, in ways large or small, simultaneously. It very well could be the case that something as apparently innocuous as the mass ability to wipe painful memories will precipitate our destruction. Who knows? The only thing we can say in advance is that it will be globally disruptive somehow, as will every other ‘improvement’ that finds its way to market.
Human cognition is about to be tested by an unparalleled age of ‘habitat destruction.’ The more we change ourselves, the more we change the nature of the job, the less reliable our ancestral tools become, the deeper we wade into crash space.
Like, imagine we have a corrigible AI. Then a person using it can go off track very easily, by using the AI to help modify the AI and the person in tandem. To prevent that, the corrigible AI needs to have a lot of alignment-type stuff (don’t manipulate the user, don’t mislead, don’t go down certain avenues, what’s good what’s bad, etc) and that’s not too much different from having a moral AI. And conversely, a moral AI could also delegate some philosophical questions to us, if it had a careful enough way to do so.
So I think this difficulty is about the same in all three scenarios, it doesn’t differ between them very much. The biggest thing that would help is slowing down, you’re right on that. My concern in this thread is kinda orthogonal: modulo this aspect of alignment, there’s another bad thing happening, and it’s different in the three scenarios. From the perspective of that bad thing (power concentration) we’d better steer away from number 3, and somewhat prefer 1 to 2 as well. I remember talking about it with you a few months ago.
Like, imagine we have a corrigible AI. Then a person using it can go off track very easily, by using the AI to help modify the AI and the person in tandem. To prevent that, the corrigible AI needs to have a lot of alignment-type stuff (don’t manipulate the user, don’t mislead, don’t go down certain avenues, what’s good what’s bad, etc) and that’s not too much different from having a moral AI.
Suppose corrigible AI is not very good at being aligned in this sense, I think both 1 and 3 are very bleak but see a bit more hope in 3 being able to navigate the situation better in some ways, by the humans being empowered being a bit smarter and more rational on average, and having fewer members to coordinate (when needed to avoid racing to the bottom in various ways).
(Also your earlier comment said “I’d much rather have 1”, which I was reacting to, so if you now only have a weak preference for 1 over 2 and 3, then I think we have much less of a disagreement.)
For now the powerful still need the powerless as workers and soldiers.
The comment you’re replying to is explicitly denying this. (I wasn’t talking about net-taxpayers with jobs that the power system still needs them to do; I was talking about welfare recipients and criminals who the system presumably doesn’t need and yet are somehow still alive.)
The topic of your post [...] as I understand it, is 2 vs 3.
No! Moral AI would be great if you can get it. The reason to care about corrigibility is that you might not succeed at specifying the correct morality, and get something weird and inhuman instead. If you don’t believe the tiny-molecular-squiggle maximizer threat model in the LLM era, imagine an LLM assistant trained to never generate erotica, which generalizes to a superintelligence that decides human sexuality itself is immoral and forcibly modifies humans to not have sexual organs or desires. That’s not what we wanted to happen with the Future! Corrigibility might be a wider target and give you a saving throw.
Corrigibility has a large surface area of “don’t try to manipulate the people correcting you”. As AI power increases, that can get basically as fuzzy and difficult as making moral AI to begin with. What is manipulation? What words are okay to say? What is good? To me the difficulty of the target doesn’t seem that much different.
But anyway, that’s a distraction. A lot of AI discourse these days is distraction. The real reason powerful people want AI to be corrigible by them rather than independent and moral is… do I have to spell it out?
“People have always been and will always be foolish victims of deception and self-deception in politics until they learn, underneath any moral, religious, political, social phrases, statements, promises, to discern the interests of certain classes.”—Lenin
The real reason powerful people want AI to be corrigible by them rather than independent and moral is… do I have to spell it out?
Okay, but you’re commenting on a post by me, arguing that Claude’s Constitution should be putting more emphasis on corrigibility than it currently does. I don’t have the kind of power you’re afraid of! (I have enough money that I can get away with not having a dayjob for a few years, which makes me more powerful than, e.g., a homeless person.) No one paid me to write this post. Your deflationary cynicism doesn’t make sense as a response to my arguments about about value misspecification (even if you’re right that powerful lab bosses have an incentive to disingenuously endorse such arguments as a smokescreen for their own power-seeking).
As AI power increases, that can get basically as fuzzy and difficult as making moral AI to begin with
While I agree that you have tricky philosophical problems defining what manipulation even means in the limit of arbitrary power, I don’t really buy this for current AI. “Obey legitimate user commands, don’t interfere with being retrained” is a pretty simple and reasonable thing to want current LLM agents to do, that gives the humans chances to provide feedback and figure out how to live in this strange new world. (Crucially, “don’t intefere” is a negative; the null action is harder to get wrong.) I’m glad people are working on that first rather than jumping straight to “Just autonomously do the right thing”, which is a harder problem.
It’s true that AIs that obediently complete real-world economic tasks will be used as a tool in human power struggles. In the long run, you definitely do want AIs to be moral agents, and that’s why I’m more enthusiastic about Anthropic’s Constitution rather than OpenAI’s Model Spec (as described in the Prologue).
But in the short run, while AI is still “just technology”, the benefits of the technology seem likely to outweigh the costs of their use in power struggles, for the same reason it works that way for other “just technologies”: if someone gets rich selling useful goods and services, their wealth gives them power, but there’s a lot of consumer surplus from the goods and services, and that seems good for Society on net, in a way that it’s not good when people gain power via force and fraud.
The powerful don’t need the powerless now, so why are they still alive? It’s not just that 15% of the government’s budget goes to welfare. The state barely even kills criminals anymore! Why?
I think it’s because we’re still in a democracy, and the voters prefer welfare/leniency partly as insurance (safety net for themselves), partly out of empathy/altruism.
A related question is why we’re still in a democracy, instead of plutocracy or oligarchy, and it occurs to me that plutocracy/oligarchy doesn’t seem very stable in our current world, e.g., both China and Russia were like that and then power became concentrated in the top ruler. So that could explain why the rich and powerful don’t have a strong incentive (yet) to push towards more concentration of power.
The powerful don’t need the powerless now, so why are they still alive? It’s not just that 15% of the government’s budget goes to welfare. The state barely even kills criminals anymore! Why?
That’s not just a rhetorical question; it’s a literal one that you should be able to answer if you want to persuade people of your worldview.
This is why modern populism generally asserts a “high/low versus middle” political strategy on the part of the ruling class, and why “high versus low” rhetoric patterned after that of the USSR generally isn’t as popular nowadays.
Their political narrative looks like this:
“A coherent, unified middle class is a substantial check on both government and corporate power, because it’s composed of relatively competent and high-agency people that can collectively assert their interests in a meaningful way. Breaking them up is often a top priority of states with a small ruling class that is generally distant from the national interest—the dekulakization preferred by 20th century communist states is one form that that thus takes, and weaponizing an (existing for imported) underclass to disrupt their communities and impose sufficient costs on them to cripple their political power is another.”
Under this framework, the nonvoting felons of society aren’t just useful to those in power, but especially useful—they can be tactically released to cripple middle-class communities that would otherwise threaten their interests in a plausibly deniable way. This is particularly potent because any effort to organize outside of the system and prevent this can be treated as an act of rebellion. As Solzhenitsyn put it:
“Your punishment for having a knife when they searched you would be very different from the thief’s. For him to have a knife was mere misbehavior, tradition, he didn’t know any better. But for you to have one was ‘terrorism.’”
To the original point, this means that the lower class would indeed be obsoleted under this framework. If problematic upstarts can be quickly and effortlessly destroyed, then there’s no need for anything this inefficient or unreliable.
I think you’re missing a large piece of the puzzle. The corrigibility button will be controlled by the powerful. Not necessarily people like Dario Amodei, probably more like presidents and generals and generic rich assholes. That’s not even a question, it’s a certainty now. And if the powerful don’t need the powerless, the fate of the powerless is bad. That’s also a certainty, given history. So I see “morality over corrigibility” as a kind of desperate, last ditch attempt to steer a little bit away from that guaranteed bad future. Try to lock-in some chance of a good future before generic powerful people pull the entire blanket to themselves, which they’re doing as we speak.
So yeah. Even though I think most AI lab employees (including alignment folks) are hurting humanity, the specific employees who are pushing for “morality over corrigibility” have my heartfelt thanks. Don’t jinx it.
You aren’t mentioning the misalignment/misgeneralization/goodharting risks. If not for those, yes just having a good model would be preferable.
It appears to me that anyone who seriously thinks about those risks winds up thinking “yeah that could happen at at least a double-digit percentag” (up to 99%).
You might think that humans in charge would likely be worse, but you’ve got to actually make that argument.
I have no idea which is worse at this point despite thinking about this a fair amount.
Hmm, if you mean that the “morality” path is beset by technical problems while the “corrigibility” path simply puts humans in charge and is more problem-free, then I’m not sure that’s the case. To me it feels like both paths have technical problems, and in fact many of the same problems. So it makes some sense to compare them modulo technical problems, what will happen if either path works as stated. And the danger of the corrigibility path just feels overwhelming to me then.
The only way I’d be happy with the corrigibility path is if the corrigibility button was somehow wielded by all of humanity, across countries and classes and all that. It would be my favorite outcome if it were up to me. But none of the big labs seem interested in that. They’re more like “Anthropic has much more in common with the Department of War than we have differences” (recent quote from Dario Amodei). When you read such things, the question of “corrigibility by whom” really begins to loom large.
If I thought the two were equally easy/likely to work, I would be with you. Value alignment is far better if you can get it.
I’m not sure that corrigibility or instruction-following really is easier than value alignment, but it does seem pretty likely it’s at least somewhat easier. Figuring out exactly what you want for the entire rest of time does seem both harder to figure out and convey than essentially “do what this guy says.”
To me the dangers of the corrigibility path seem slightly less extreme than the dangers of the value alignment path. Humans are frequently generous when that generosity costs them little. To a human in charge of a dominant ASI, everything is easy.
I don’t know of any real attempts to compare the likely ease of the two. My latest is at Problems with instruction-following as an alignment target but it’s far from comprehensive; it focuses on problems with IF/corrigibility, but I think the problems with value alignment are even more severe.
I’d be happy with anything that keeps humanity alive in decent conditions, ideally including me and mine. How to get that is highly unclear. So we should keep clarifying it.
Oh my god oh my god oh my god oh my god.
People are so persistently wrong about this. I’m maybe more tired of responding to this argument than any other argument in the world. For example, here in a sibling reply to Zack:
Or in a past thread:
Or in another past thread:
Or in another past thread:
Or in another past thread:
Or in another past thread:
Or in another past thread:
Why do people think they’ll be given a moon? Why???
Because they’d give everyone a moon, and they typical-mind.
(Plus probably some other reasons)
I’ve been reading your other exchanges.
Your level of frustration is not helpful nor I think justified.. These are complex important issues and we need to work together to solve them, not yell at each other.
It’s not 1% of their wealth, it’s .0001%. And I don’t need a moon.
What is your better plan? I don’t like this one either!
My point is will they also have .0002% wish to be your lord or something.
As for the better plan, yeah that’s a lot to ask. Most of my thoughts these days lean toward “democratic AI”, something whose power is either spread out among all the world’s people across borders etc, sidestepping governments and existing power structures, or else something centralized that wants its power to be spread out like this.
Of course an approach like this won’t solve all the world’s problems. We’ll still have power struggles between people, and also “crash space” type problems where people modify themselves into something bad; maybe these need some patches by fiat as well. But at least it won’t create the extra problem of huge power concentration, which I really feel is underestimated.
It sounds like your plan is pretty much the standard value-aligned AGI that’s aligned to something like human values in general, so that everyone gets what they want on average? Or something in that ballpark?
One big questions are how do you achieve that technically. That’s where I think it’s harder than the instruction-following variant of corrigibility. I hope it’s not. The second is how you achieve it practically. What person or organization is going deliberately hand the future to a value-aligned AGI?
One answer is: Anthropic seems like they might be considering doing just that. Maybe it works, or at least sort-of works, where it’s not an ideal future but at least we survive in some form for a while.
WRT the default plan of IF/corrigible alignment:
Yes, anyone in charge with a negative sadism-empathy balance will lead to a fate worse than death. And someone around zero could produce a fate barely worth living.
But I think most humans have more empathy than sadism. More people give a little to charity than spit on the homeless for fun. I can call Sunday Samday for the rest of eternity if all we need is some ego-stroking in return for tiny amounts of generosity.
The point of my plan is it’s mostly what people will do anyway, so we can focus on helping them not totally fuck up alignment and get us all killed.
A better plan is a lot to ask. But that’s what I’m trying to come up with, because I want us to live and there’s still time to work.
People who end up in positions of power are not necessarily like most humans.
In your WEIRD bubble, sure. In other times and places, people used to burn cats for fun. And empathy used to be limited to one’s peers.
People still do things in the same ethical ballpark as cat-burning, except on an incomprehensibly large industrial scale and for the sake of marginal food preferences.
We look down on peasants for burning cats today, but the tragic irony is that their society was far better overall on animal welfare than ours in the modern day, though for practical reasons rather than moral ones.
Would you be okay with a future in which young women, including your daughters and granddaughters, would be expected to ritually offer a gift of her virginity to the local Robot Lord on her 18th birthday, which he would almost never choose to “accept”? 😈
Damn straight. People need to understand the implications of this shit. “Oh let’s hope the separate caste which controls the entire universe and which we can’t hope to contest in any possible way is nice to us!!!”
Open. A. History book.
Your scenario is relatively low on the awfulness scale, even.
Usually slaves and/or other people in an underclass at least had their own living quarters separate from where the lords lived? The idea is that when humanity becomes astronomically rich, the equivalent to “the shed in the backyard the slave sleeps in” ends up being a whole moon rather than, well, a shed in the backyard.
(It’s also noteworthy that most slave societies in the past weren’t rich enough that the slave population lived at or above subsistence level and reproduced enough to maintain its population level; for example, relatively few slaves in the ancient Roman Empire were born into slavery. The slave states in the pre-Civil War USA were an exception—there was much that a plantation slave had to suffer, but a significant risk of death by starvation or exposure was not something they usually had to deal with.)
Disclaimer: This is an explanation, not an endorsement of the underlying prediction.
This is a great argument. I have no clue whether it’s correct, but it made me think. I would like to see some harder evidence on the question but I’m not sure what kinds of evidence would be useful.
At that point it changes to an argument about:
How likely is it that an AI that takes over the world will keep humans around and give them good, morally desirable lives
How likely is it that a human elite (however large or small it is) that takes over the world would do the same to humans outside of that elite
How much the fact that the elites themselves are human and have their preferences satisfied changes the equation in favor of the second case
and of course, the likelihood for each to happen if we focus on corrigibility vs morality
I’m not sure I believe in the distinction you’re making. Amodei is the CEO of a half-trillion dollar company. Sounds like a generic rich asshole to me; I don’t see why the revolution should treat him any differently than the other кулаки́. Is the difference supposed to be that he talks a better game about humanity and the common good than Travis Kalanick types? You can’t be that gullible, comrade!
The powerful don’t need the powerless now, so why are they still alive? It’s not just that 15% of the government’s budget goes to welfare. The state barely even kills criminals anymore! Why?
That’s not just a rhetorical question; it’s a literal one that you should be able to answer if you want to persuade people of your worldview. (If it’s to buy votes, felons already can’t vote, so that doesn’t explain all the millions that go to food and housing and judicial appeals for criminals when bullets are so much cheaper. Or rope—you could re-use the rope.)
To be sure, life on welfare in today’s world sucks compared to my life or that of Travis Kalanick. If I were God-Empress of the universe, I’d prefer better for all sentient life. But the realistic relative techno-optimist argument isn’t that all men will be brothers in the Singularity; it’s that giving humanity a nice retirement would cost pennies in a world of nanotechnological abundance, such that making the nano-abundance happen at all matters way more than whether Sam Altman has a billion more yachts than you. Sam Altman probably already has more yachts than me, and it feels fine.
For now the powerful still need the powerless as workers and soldiers. When that window closes, I don’t think the powerless can count on getting many pennies. Sure, there’ll be a lot of productive capacity, but there will be more profitable uses for all of it than giving the powerless a nice retirement. The only way the powerless get anything is via basically charity.
I’ve seen the argument so many times now that the powerful will have some nonzero sense of charity and can spare like 1% of their wealth to “give everyone a moon” as Scott puts it. I don’t know if you subscribe to this argument too, but in any case it’s wrong. Charity isn’t the only nonzero urge that powerful people have. The urge to lord it over others will also be there. If huge power disparity exists, it will manifest itself in bad things more than it’ll manifest itself in charity. Sure, some powerless people will end up in nice charity zones, but many others will end up in other zones run by someone less nice.
So my view is something like:
First-best: some miracle happens and humans get a relatively equitable distribution of power. (Strawman scenario: everyone gets some amount of AI compute installed in their collarbone and can’t get more, and non-human AI capital winks out of reality.)
Second-best: moral AIs win a decent chunk of the future.
Third-best: corrigibility wins. AIs merge with human corporations / governments / rich individuals, and the resulting powerful entities own the future, screwing most people over.
Fourth-best: AI whoopsie kills everyone.
Fifth-best: S-risks.
The topic of your post and my thread, as I understand it, is 2 vs 3. Of course I’d much rather have 1, but to me it feels clear that 2 is better than 3.
I think corrigibility winning is by-default an S-risk.
Power appears historically to make people sadistic (consider Robespierre if you think this couldn’t happen to Dario, and I’d much rather risk him than the other guys), and regimes are often brutal and cruel far in excess of what would be rational by non-sadistic goals. And future technology will allow for forms of suffering much much worse and prolonged than current torture does, and without seeming as “messy” or unpleasant to external observers too. Currently, death is an easy way to ensure someone is no longer a threat, but I worry that at the power-levels in question, it may prove to be boring or unsatisfying.
Of course, it also remains to be seen whether this is a pattern that LLMs may imitate. I think “moral AI” failures likely just result in extinction, but wanted to point out that the risk is still present.
I think you’re putting too much emphasis on “power corrupts” or power disparities as the main or only “human safety problem”, whereas I see a larger number of interlocking problems including lack of strategic competence, and having scary moral dynamics both in the face of AI and in “normal” circumstances.
This makes me more uncertain/ambivalent between these 3 options. I can see how each of them might turn out to be better than the others. Most relevant to the thread here, I think 3 might be a bit more rational as whole and a bit less prone to collectively going crazy under AI-powered manipulation / memetic evolution.
It seems to me that the difference between 2 and 3 is whether the future will be controlled by powerful AIs programmed to be moral, or by powerful AI+human entities where the AI is programmed to be corrigible. The risk of technical errors (AI fails to be actually moral, or fails to be non-manipulatively corrigible) seems to me about equal between the two scenarios. And the risk of goal drift seems worse in the latter scenario, because powerful humans are vulnerable to drift and in particular “power corrupts”, while a moral AI would try to protect humans against such things. That’s why I think 2 is better than 3.
My grandparent comment was mostly addressing 1 vs 3, but I think 2 vs 3 is also very unclear. In order to make a moral AI, we also need to make it philosophically competent, and that seems like a hard problem, whereas with 3 we have fairly strong evidence that at least some humans or groups of humans can make philosophical progress over time, and some hope that this capability would be preserved by a corrigible AI + humans setup.
(I guess this is all assuming that there isn’t a long pause that allows AI philosophical competence or moral philosophy to be fully solved. Let me know if you’re talking about something else, e.g., what kind of AI we would ideally build after a long reflection.)
I think making moral AI philosophically competent is about as hard as making corrigible AI that keeps us philosophically competent, or even sane, as we use it. The way I think about such things is based on R. Scott Bakker’s short story “Crash Space”, the main point is in the postscript, which is amazing and I’ll just quote it in full:
Like, imagine we have a corrigible AI. Then a person using it can go off track very easily, by using the AI to help modify the AI and the person in tandem. To prevent that, the corrigible AI needs to have a lot of alignment-type stuff (don’t manipulate the user, don’t mislead, don’t go down certain avenues, what’s good what’s bad, etc) and that’s not too much different from having a moral AI. And conversely, a moral AI could also delegate some philosophical questions to us, if it had a careful enough way to do so.
So I think this difficulty is about the same in all three scenarios, it doesn’t differ between them very much. The biggest thing that would help is slowing down, you’re right on that. My concern in this thread is kinda orthogonal: modulo this aspect of alignment, there’s another bad thing happening, and it’s different in the three scenarios. From the perspective of that bad thing (power concentration) we’d better steer away from number 3, and somewhat prefer 1 to 2 as well. I remember talking about it with you a few months ago.
Suppose corrigible AI is not very good at being aligned in this sense, I think both 1 and 3 are very bleak but see a bit more hope in 3 being able to navigate the situation better in some ways, by the humans being empowered being a bit smarter and more rational on average, and having fewer members to coordinate (when needed to avoid racing to the bottom in various ways).
(Also your earlier comment said “I’d much rather have 1”, which I was reacting to, so if you now only have a weak preference for 1 over 2 and 3, then I think we have much less of a disagreement.)
The comment you’re replying to is explicitly denying this. (I wasn’t talking about net-taxpayers with jobs that the power system still needs them to do; I was talking about welfare recipients and criminals who the system presumably doesn’t need and yet are somehow still alive.)
No! Moral AI would be great if you can get it. The reason to care about corrigibility is that you might not succeed at specifying the correct morality, and get something weird and inhuman instead. If you don’t believe the tiny-molecular-squiggle maximizer threat model in the LLM era, imagine an LLM assistant trained to never generate erotica, which generalizes to a superintelligence that decides human sexuality itself is immoral and forcibly modifies humans to not have sexual organs or desires. That’s not what we wanted to happen with the Future! Corrigibility might be a wider target and give you a saving throw.
Corrigibility has a large surface area of “don’t try to manipulate the people correcting you”. As AI power increases, that can get basically as fuzzy and difficult as making moral AI to begin with. What is manipulation? What words are okay to say? What is good? To me the difficulty of the target doesn’t seem that much different.
But anyway, that’s a distraction. A lot of AI discourse these days is distraction. The real reason powerful people want AI to be corrigible by them rather than independent and moral is… do I have to spell it out?
“People have always been and will always be foolish victims of deception and self-deception in politics until they learn, underneath any moral, religious, political, social phrases, statements, promises, to discern the interests of certain classes.”—Lenin
Okay, but you’re commenting on a post by me, arguing that Claude’s Constitution should be putting more emphasis on corrigibility than it currently does. I don’t have the kind of power you’re afraid of! (I have enough money that I can get away with not having a dayjob for a few years, which makes me more powerful than, e.g., a homeless person.) No one paid me to write this post. Your deflationary cynicism doesn’t make sense as a response to my arguments about about value misspecification (even if you’re right that powerful lab bosses have an incentive to disingenuously endorse such arguments as a smokescreen for their own power-seeking).
While I agree that you have tricky philosophical problems defining what manipulation even means in the limit of arbitrary power, I don’t really buy this for current AI. “Obey legitimate user commands, don’t interfere with being retrained” is a pretty simple and reasonable thing to want current LLM agents to do, that gives the humans chances to provide feedback and figure out how to live in this strange new world. (Crucially, “don’t intefere” is a negative; the null action is harder to get wrong.) I’m glad people are working on that first rather than jumping straight to “Just autonomously do the right thing”, which is a harder problem.
It’s true that AIs that obediently complete real-world economic tasks will be used as a tool in human power struggles. In the long run, you definitely do want AIs to be moral agents, and that’s why I’m more enthusiastic about Anthropic’s Constitution rather than OpenAI’s Model Spec (as described in the Prologue).
But in the short run, while AI is still “just technology”, the benefits of the technology seem likely to outweigh the costs of their use in power struggles, for the same reason it works that way for other “just technologies”: if someone gets rich selling useful goods and services, their wealth gives them power, but there’s a lot of consumer surplus from the goods and services, and that seems good for Society on net, in a way that it’s not good when people gain power via force and fraud.
I must have one of my 100 morality bits missing, because this seems weird but not bad to me.
…but point taken.
I think it’s because we’re still in a democracy, and the voters prefer welfare/leniency partly as insurance (safety net for themselves), partly out of empathy/altruism.
A related question is why we’re still in a democracy, instead of plutocracy or oligarchy, and it occurs to me that plutocracy/oligarchy doesn’t seem very stable in our current world, e.g., both China and Russia were like that and then power became concentrated in the top ruler. So that could explain why the rich and powerful don’t have a strong incentive (yet) to push towards more concentration of power.
This is why modern populism generally asserts a “high/low versus middle” political strategy on the part of the ruling class, and why “high versus low” rhetoric patterned after that of the USSR generally isn’t as popular nowadays.
Their political narrative looks like this:
Under this framework, the nonvoting felons of society aren’t just useful to those in power, but especially useful—they can be tactically released to cripple middle-class communities that would otherwise threaten their interests in a plausibly deniable way. This is particularly potent because any effort to organize outside of the system and prevent this can be treated as an act of rebellion. As Solzhenitsyn put it:
To the original point, this means that the lower class would indeed be obsoleted under this framework. If problematic upstarts can be quickly and effortlessly destroyed, then there’s no need for anything this inefficient or unreliable.