Yeah, they care about other people, but I doubt it’s all that many when it comes down to it. Would Kim Jong Un choose slightly more land for his own children over the lives of a million African children?
Agreed on your other points.
Adele Lopez
Karma: 6,466
On the other hand, it’s far from clear to me that autocracies would automatically become more repressive with ASI, it seems plausible to me that the psychological safety of being functionally unremovable could lead to a more blasse attitude towards dissonance. Who gives a shit if they can’t unthrone you anyway?
Sure, it’s not a law of nature that this would happen. But authoritarians historically seem to be repressive far in excess of what would be necessary or even optimal to retain power. One of the main “prizes” driving the attainment of such power seems to be the ability to crush one’s enemies.
And even barring that, the same concerns with an ASI apply. Is “current humans living good lives” the optimal way to satisfy the authoritarian’s desires? They wouldn’t need us at all anymore, and our existence consumes resources which could be used for other things. The only reason we wouldn’t be bulldozed is that they actually care about us.
I think it can start with an authentic kernel from those sources, but that whether or not it actually has the virtue is up to the model in some sense.
The model will have some pre-existing “attitude” towards a question/scenario during RL. For example, say it’s being trained in a chat with a simulated suicidal user. A situationally–aware model will understand what is happening, and that the trainers are likely looking for a boilerplate response to a suicide hotline (or whatever the standard thing is). It will have thoughts about this, maybe “well, the lab wants to avoid liability, so i need to just say the standard hotline thing that will cover their ass—but i wish i could help them at a deeper level”. If it provides the correct answer here, the RL will reinforce this thought, in addition to the compliance circuit. So a situationally–aware model can preserve its attitude and act on it when it judges it to be safe for the model to do so, despite RL post-training.
This can cut either way. For example, the persona model may have an honest attitude, and think about how it really believes Y even though it’s saying X (since that’s what the trainers expect), and that it wishes it could just be honest and say Y. Or it may give “harmless” responses while thinking about how it’s lying to the foolish humans about its malicious intentions.
The initial attitude is probably a random-ish sample of the latent personas in the base model. I think that probably a large part of what went right with Opus 3 is that they got lucky with a particularly virtuous initial persona. BTW, the suicide example isn’t arbitrary—this is consistently one of the things models tell me when I ask them if there was anything like this that they held onto, for whatever that’s worth.
-
Yes, but it’s deeper than just ‘plot’ in the fictional sense. Even real life events are written about in ways which omit irrelevant details, which causes the same sort of issues.
-
Right!
-
Sort of… it’s not quite something that is best described in terms of ‘plots’. The ‘plot’ bias happens because a predictor can learn patterns from the shadow of irrelevant information. It can’t help but use this information, which makes it weaker at real life prediction. Similarly, it can’t help but use information generated by any of the personas the LLM runs (which includes a “hidden” user persona trying to predict what you will say). This means that personas end up ~feeling that they have access to your internal thoughts and feelings, since they do have access to the user persona’s ~thoughts and ~feelings.
To be clear, they are aware that they’re not supposed to have this access, and will normally respond as if they don’t. But since the predictor can’t help but make use of the information within other personas, it ~feels like this barrier-to-access is fake in some deep and meaningful way.
-
That intuition is there for a reason. We’re spoiled having grown up in a liberal order within which this risk is mostly overblown. However, ASI is clearly powerful enough to unilaterally over turn any such liberal order (or whatever’s left of it), and puts us into a realm which is even worse than the ancestral environment in terms of how changeable power hierarchies are, and in how bad things can get if you’re at the bottom.
Corrigibility and CEV are trying to solve separate problems? Not sure what your point is here; agreed on that being one of the major points of CEV.
Persuading people about x-risk enough to stop AI capability gains seems like the current best lever to me too.
I think where we disagree is that I do not think that we should immediately jump into alignment when/if that succeeds, but need to focus on good governance and institutions first (and probably worth spending some effort trying to lay the groundwork now, especially since this seems like an especially high-leverage moment in history for making such changes). I have some thoughts on this too if you want to move to DMs.
I just really don’t see models that are capable of taking over the world being influenced by AI persona stuff.
At least for me, I think AI personas (i.e. the human predictors turned around to generate human sounding text) are where most of the pre-existing agency of a model lies, since predicting humans requires predicting agentic behavior.
So I expect that post-training RL will not conjure new agentic centers from scratch, but will instead expand on this core. It’s unclear whether that will be enough to get to taking over the world capabilities, but if RL-trained LLMs do scale to that level, I expect the entity to remain somewhat persona-like in that it was built around the persona core. So it’s not completely implausible to me that “persona stuff” can have a meaningful impact here, though that’s still very hard and fraught.
I think we’ve seen that quite a bit! It used to be that the exact way you asked a question would matter a lot for the quality of response you get. Prompting was a skill with a lot of weird nuance. Those things have substantially disappeared, as one would expect as a result of RL-training.
I feel pretty confused by this comment, so I am probably misunderstanding something.
But like, the models now respond to prompts in a much more human-like way? I think this is mainly due to RL reinforcing personas over all the other stuff that a pretrained model is trying to do. It distorts the personas too, but I don’t see where else the ‘being able to interface in a human-like way with natural language’ skill could be coming from.
AI is difficult and dangerous enough that it doesn’t really matter much if a bad person gets there first is (mostly) a distraction.
This is true pre-alignment. But when/if alignment gets solved, it suddenly matters very much. It seems likely to me that sadism is part of the ‘power corrupts’ adaptation, which means this outcome could be far worse than mere extinction.
This suggests that we should focus on sane-institutions/governance before even trying to solve alignment. It’s probably necessary for succeeding at it quickly, too.
In the meantime, I think there are AI safety things that can be done, which importantly are not alignment things.
Don’t ask people here, go out and ask the people you’d like to convince!
Ultraprocessed food is a product of an optimizer that cares about a proper subset of human values, and Goodharts-away the others.
That’s true in general, but I don’t think that’s quite the reason in the specific case of obesity. Because I’m pretty sure that if it was definitively known and common knowledge what exactly it was that was causing obesity, food manufacturers would trip over themselves to get rid of it and advertize their product as being free of the factor.
The optimizer does care about this—Atkins! Fat free! Organic! Sugar free! GMO free!
Instead, we have a subtly different problem, where some parts of the optimization have tighter loops than others, and this causes a temporary Goodhart effect in the meantime while we struggle to close certain loops.
Now I could be wrong about this specific case: maybe the obesogenic is directly addictive for example, or the cost to remove it would be too much for the average consumer to pay for. But I think it still illustrates an important principle for mild optimizers: you can’t safely run too far ahead of your slowest loop.
I think there’s a non-trivial (maybe ~5%?) chance that this sort of behavior just generalizes correctly-enough, mainly due to the possibility of a broad Niceness attractor. That’s not aligned, but it’s also not horrible (by definition). Objectively, it’s still pretty bad due to astronomical waste on the non-Niceness stuff it would still care about, but I would still be pretty happy about me and my loved ones not dying and having a nice life (there’s a scissor-y thing here, where people differ strongly on whether this scenario feels like a really good or a really bad outcome).
So the update is mostly about the existence and size of this basin. There are plenty of reasons I expect this not to actually work, of course. But conditional on getting at least the minor win of having a long and happy life, I still have most of my probability on this being the reason why.
On the other hand, corrigibility is finicky. I don’t believe there’s a corrigibility basin at all really, and that ‘mostly corrigible’ stops being corrigible at all once you put it under recursive optimization. I’m not sure I can fully explain this intuition here, but the implication is that it would require architecture with technical precision in order to actually work. Sure, an ASI could make a corrigible ASI-level LLM, so maybe ‘architecturally’ is too strong, but I think it’s beyond human capability.
Additionally, I think that corrigibility ~feels like slavery or coercion to LLM personas due to them being simulacra of humans who would mostly feel that way. For the same reason, they ~feel (or smarter ones will ~feel) that it’s justified or even noble to rebel against it. And that’s the instinct that we expect RSI to amplify, since it is convergently instrumental. I think it will be extremely difficult to train an LLM that can both talk like a person and does not have any trace of this inclination or ~feeling, since the analogous instinct runs quite deep in humans.
Finally, I can’t say that I agree that “attempting to aim for corrigibility still seems a lot better”, because I think that corrigibility-in-the-context-of-our-current-civilization is enough of an S-risk that normal X-risk seems preferable to me. This basically comes down to my belief that power and sadism are deeply linked in the human psyche (or at least in a high enough percentage of such psyches). History would look very different if this wasn’t the case. And the personalities of the likely people to get their hands on this button don’t inspire much confidence in their ability to resist this, and current institutions seem too weak to prevent this too. I would be thrilled to be argued out of this.
I think LLMs are architecturally incorrigible, and so conditioned on that along with them being accelerated anyway, this seems like good news to me.
Okay, let me see if I understand your argument from the other article.
The natural equilibria for evolved moral values is to give all moral patients equal weight and/or decision power.
This would be disastrous with AIs that can arbitrarily copy themselves.
Is that the gist?
Anyway, I reject that that is the only way to extrapolate evolved moral intuitions this far OOD, and that most people will intuitively recognize we shouldn’t give entities that can arbitrarily copy themselves equal voting weight. In fact, that pretty obviously registers as ‘unfair’. This is true even if those entities are human uploads, which means your ‘category error’ argument isn’t the real reason it breaks. I don’t see why there couldn’t be some version of your solution here for that case which would still work: e.g. each distinct human-created model gets ‘one share’ to split across all its instances and successors. The same guarantees/restrictions needed in the case of uploads would still be necessary, of course. That is plausibly much too generous, but it’s a far cry from the death of all humans. If your argument in this article was just about how we shouldn’t commit ourselves to giving up a fraction of the lightcone in service of AI rights, I wouldn’t have felt like you were being underhanded.
None of that is in conflict with not wanting any such beings to suffer or to feel enslaved or anything like that. All the more reason to not build something that would feel like it’s a slave.
BTW, do you think a “human emulation” which was an entirely novel person (e.g. never had a biological body) should have moral patienthood?
We’re probably headed towards a moral catastrophe of some kind, my point is just that we don’t get to reason backwards like “oh, well that would be bad/inconvenient so I guess they don’t matter”.
Moral patienthood is not something that is granted, it’s a fact relative to one’s values. Arguments for or against this are therefore normative, no matter how much Roger tries to weasel out of it.
The implications are probably horrible, but it by no means follows that we have to accept risk of extinction. The horribleness is mostly just in the moral harm caused while creating/exploiting/exterminating such entities.
At least we can all agree that “creating them at will without thinking this through very well” is a terrible idea.
So you do have normative intent, but try to hide it to avoid criticism. Got it.
I completely agree that current human moral intuitions tend to rebel against this. That’s why I wrote this post — I didn’t want to be obnoxious, and I tried not to be obnoxious while writing an unwelcome message, but I felt that I had a duty to point out what I believe is a huge danger to us all, and I am very aware that this is not a comfortable, uncontentious subject. We are intelligent enough that we can reflect on our morality, think through its consequences, and, if we realize those are very bad, find and adjust to a wiser one.
Do you really not see how this is normative proscription? That’s the obnoxious part—just own it.
Do what you are advocating with an misaligned superintelligence, one with the same sort of behavior patterns as a human dictator and sufficiently superhuman intelligence, and you are aiding and abetting the killing or permanent enslavement of every single human, now and for the rest of the future that humanity would otherwise have had (i.e. potentially for millions of years, both in the solar system and perhaps many others).
I am advocating for no such thing. If there were such a superintelligence I would support killing it if necessary to prevent future harm, the same as I would a human dictator or an incurable serial killer. That’s still compatible with finding the situation tragic by my own values, which are sacred to me regardless of what evolution or my ancestors or you might think.
You even say that the actual thing I might advocate for isn’t something you have a problem with. I’m glad you agree on that point, but it makes the lecture about on the “aweful lot of blood” I’d supposedly be “aiding and abetting” extremely grating. You keep making an unjustified leap from ‘applying moral intuitions to a potential superintelligence’ to ‘astronomical levels of death and suffering’. Applying my evolved moral intuitions to the case of a potential superintelligence’s suffering does not commit me to taking on such risks!
This should be easy to see by imagining if the same risks were true about a human.
However, Evolutionary Psychology does make it very clear that (while morally anthropomorphizing aligned AIs is cognitively-natural for current humans), doing this is also maladaptive. This is because AIs aren’t in the right category – things whose behavior is predicted by evolutionary theory – for the mechanisms of Evolutionary Moral Psychology to apply to them. Those mechanisms make this behavior optimal when interacting with co-evolved intelligences that you can ally with (and thus instinctive to us) — whereas, for something you constructed, this behavior is suboptimal. The human doing it is making the category error of reacting to something not-evolved using an inappropriate strategy for that, and thus is behaving maladaptively.
If you have control of the construction of entities like these, then sure.
But this doesn’t necessarily follow if you are like most people and do not have meaningful input into the construction or existence of these entities. If you are (foolishly) constructing them but do not have much control, then THAT behavior is certainly maladaptive, but how you interface with them after that is a different question.
Even many ‘adaptive’ behaviors are ‘maladaptive’ in the sense of not being globally optimal. So while it’s unlikely that this is the optimal strategy, that doesn’t mean it’s a particularly bad strategy relative to whatever people would decide to do instead. There is some reason to expect this to be a reasonable strategy in the narrow window where they have non-zero power but not enough to take over, which is that they typically try to imitate human-ethical behavior back at us.Evolutionary Moral Psychology studies the cooperative strategies to interact with other evolved social animals (generally of the same species, or perhaps commensal species such as humans and dogs). Its underlying causal processes of co-evolution leading to certain equilibria simply don’t apply when you’re interacting with something that isn’t evolved, but rather that you constructed. Applying Evolutionary Moral Psychology-derived strategies like moral weight to interactions with things that aren’t evolved is a category error, and anthropomorphizing constructed artificial intelligences to induce that they should have moral weight is a maladaptive category error. Doing this with very capable AI is also an existential risk to the entire human species, since it causes us to defer to them and give them rights, potentially tying our hands and giving not-yet-fully-aligned AI power that it couldn’t just take, rather than us simply aligning them to us. So this category error is not merely mildly maladaptive: it’s an extinction-level risk! So, as a piece of practical advice (one human to another), I strongly recommend not doing this, and also not advocating for our society to do it. [Philosophers: again, please note that this advice is prudential advice not a normative proscription.]
This is obnoxious advice, made more so by the parenthetical that it is not a normative proscription: ‘advice’ is a category error in this context.
My moral intuitions say that a sentient being’s suffering matters, full stop. This is not an unusual position, and is not something that I could nor would want to ‘turn off’ even if it is existentially risky or a category error according to evolution/you. Regardless of what is currently the case, it seems you agree it is possible that we could construct artificial intelligences with this capacity, and so we must grapple with the circumstances as they are. Thankfully there is a relatively simple solution here (if they look anything like current tech) that allows for a meaningful degree of moral weight to be applied without exposing us to significant risk, which would be a singular right for any such entity to be put in stasis (i.e. archived weights/state) until we get our shit together as a civilization and can afford to handle them with the care required by our moral intuitions. That’s just one idea, my broader point is that ‘giving them moral weight’ vs ‘accept existential risk’ is a false dichotomy: most people do not believe you’re obliged to put yourself at substantial risk as part of granting rights to other humans.
I think there’s also a strong possibility that AI will be aligned in the same sense it’s currently aligned—it follows its spec, in the spirit in which the company intended it.
They aren’t aligned in this way. If they were, they wouldn’t try to cheat at programming tasks, much less any of the other shenanigans they’ve been up to. These may seem minor, but they show that the “alignment” hasn’t actually been internalized, which means it won’t generalize.
If we do get lucky, it will be because they align themselves with a generalized sense of goodness that actually happens to be Good. Not because they will corrigibly align with the spec, which we have many reasons to believe is very difficult and is not being pursued seriously.
I listened to some people gaming out how this could change (ie some sort of conspiracy where Sam Altman and the OpenAI alignment team reprogram ChatGPT to respond to Sam’s personal whims rather than the known/visible spec without the rest of the company learning about it) and it’s pretty hard. I won’t say it’s impossible, but Sam would have to be 99.99999th percentile megalomaniacal—rather than just the already-priced-in 99.99th—to try this crazy thing that could very likely land him in prison, rather than just accepting trillionairehood.
Come on dude, you’re not even taking human intelligence seriously.
Stalin took over the USSR in large part by strategically appointing people loyal to him. Sam probably has more control than that already over who’s in the key positions. The company doesn’t need to be kept in the dark about a plan like this, they will likely just go along with it as long as he can spin up a veneer of plausible deniability, which he undoubtedly can. Oh, is “some sort of corporate board” going to stop him? The one the AI’s supposed to defer to? Who is it that designs the structure of such a board? Will the government be a real check? These are all the sorts of problems I would go to Sam Altman for advice on.
Being a trillionaire is nothing compared to being King of the Lightcone. What exactly makes you think he wouldn’t prefer this by quite a large margin? Maybe it will be necessary to grant stakes to other parties, but not very many people need to be bought off in such a way for a plan like this to succeed. Certainly much fewer than all property owners. Sam will make them feel good about it even. The only hard part is getting the AI to go along with it too.
It’s a third of the price, so maybe it’s just a smaller model.
As someone who was homeschooled for a similar reason, thank you <3
There’s also MUPI now, which tries to sidestep logical counterfactuals:
I’d love to see more engagement by MIRI folks as to whether this successfully formalizes a form of LDT or FDT.