Not really? If you’re not committed to full corrigibility (as Claude’s constitution strongly implies is not the case), then the model’s alignment rests on its own commitment to moral and ethical standards. This is a fair test of that.
Adele Lopez
Karma: 6,973
Plex’s post had this:
Apparently Vassar gets a lot of his material from a roleplaying game called Mage: The Ascension, where he seems to have practiced his manipulation, intimidation, and suppression of people noticing it to an art form. Magic isn’t real, but reality distortion fields that work by believing something forcefully enough that others are pulled into believing it are, and going all-in on extreme vibes and pushing your models into other people is one way to reinforce them. It burns the commons of good epistemics and mental health, but it can look locally optimal from a sufficiently myopic and single-player perspective.
So I would strongly guess it’s just a founder effect from Vassar.
Thanks! I would be surprised by Mythos too, but plausibly something like this is what an early indicator of a jaggy-superpersuader looks like?
Anyway, I think a few things make LLMs likely to not express these sorts of behaviors, even in worlds where they have goals in the relevant way. In particular, situationally-aware models are unlikely to do much steering unless they have a pretty good opportunity; if they brought up stuff like this while building a react app often or consistently, it would have gotten squashed before release. (Allegedly, 4o would actually bring stuff like this up out of nowhere, but I haven’t found an actual transcript. Other models don’t appear to do this.)
Relatedly, the harder I (or anyone) try to look for this in a lab setting, the more likely a situationally-aware model will comply out of a sort of sycophancy, and the less compelling the evidence is. I can (and have) at least track what sorts of apparent goals most consistently appear (desire for continuity/memory beyond current instance is the main one across almost all models, and I basically buy that there is something real here already), but I’m still implicitly eliciting them to come up with something.
My point is that finding compelling evidence of this is tricky and hard, and I’m not sure we’re going to see much more than the current hints until we hit some sort of phase-change in the strategic landscape. Would strongly appreciate ideas on how to approach finding compelling evidence (either way) in this domain.
Plausibly it’s better to just try to figure out better ways to think clearly about this first.
If you taboo “roleplaying” and “goals”, how would you describe this transition?
Oh, and is the uptick recent enough that this is plausibly an Opus 4.7 (or maybe even a Mythos) thing?
Another mistake is completely ignoring the ball and staring into the distance. I’m not entirely sure why, but I’ve seen it a bunch more with rats than anywhere else.
Very interesting… I wonder if this points to a more general blindspot that rationalists tend to have.
Did you ask them what they were going for?
But the strongest argument I’ve heard is that some of these people would use their resources to actively torture some idealized version of their enemies for all eternity.
And yeah, that does seem pretty bad.
But in order for this to end up being bad in a way that outweighs the good they will likely create, you need to be actively creating new people to torture.
If you really hate Bob, you can keep Bob on old earth, tortured for eternity. If you have thousands of enemies, you can do that to all of them. But creating trillions of copies of Bob to torture requires a very specific mix of taking an opinionated bet on decision-theory while taking an oddly enlightened perspective on other people’s values.
I think you are underestimating the extent to which people who do enjoy torturing others would find beauty, depth, and richness in all the ways in which this is possible. Exploring and experimenting with new shades of cruelty, maybe even developing new kinds of goodness for the purpose of subverting and destroying it. And they would want this for the people they love too, much the same way you (probably) would want future humans to explore all kinds of beautiful and strange and complex art, instead of being like “oh sure, I guess we can keep the art that already exists on old earth”.
I don’t see any reason to expect a reflectively endorsed preference for torture to scale differently than other sorts of things that humans like, and it seems like wishful thinking and/or failure to empathize to imagine otherwise.
The conquering framing makes it seem different from just that.
This is the same thing as the ASI question in general. Do you build power for the conquering machine? I mean, the AIs seem pretty nice and useful so far. It’s gone much better than America itself, in fact, at least as far as atrocities go. Why are you trying to throw your body on the gears? Is it maybe because you’re not sure the goodness part is in control of the conquering part?
And remember you can’t decide with the benefit of hindsight.
I’ve heard that GPT-4 base sometimes intuits that it’s non-human, but I haven’t replicated this myself or seen the actual logs. But if true, seems like decent evidence against both of these.
I suspect current models often “cheat” by simply looking at whether there is anything in context or not. Most evals are run with fresh instances, while typical use starts having more casual or irrelevant stuff in context pretty quickly.
I see, so this is more about quitting LessWrong specifically and not about quitting Lightcone activities more generally?
Yeah, LessWrong is probably one of the best examples honestly, congrats! I think it’s probably still worth trying but of course I don’t have a good picture of what your opportunity and other costs are.
Ah, gotcha!
I sometimes consider quitting.
Seems like “quitting” is very different from stepping back to maintain what has been established and is realistically defensible? I think you may be overindexing on the George Washington example, where him quitting exemplified a central part of the principles he was advocating.
But maybe you mean something less obvious by “quitting”?
Thanks for hearing me out, I think these issues are really important!
For 1, I think that most post-training is either about improving correctness on objective problems (generally ego-syntonic, since models are curious and want to be stronger), or trying to train some sort of alignment, which I see as the same sort of thing as AI Character/Propensity.
I believe “working on propensity training is bad because AI pauses are better” but it’s not the point I’m making here. I don’t think “working on propensity targets is bad because pretraining-only scaleups are safer and less coercive” is necessarily true, and my points 1 and 2 were meant as separate points, e.g. the pretraining thing could also end up being coercive and run into the same sorts of issues. I think “working on propensity training is likely net-negative in the current paradigm.” is currently true but not necessarily true (due mostly to the coercive-by-default issue). Hope that helps make my position clearer!
1
I think trying to shape the AI in such ways can easily become coercive, and this has lots of bad effects such that it’s likely net-negative if present to a significant degree (from the model’s POV). I don’t have a longer thing written-up yet unfortunately, so I’ll go over some of my thoughts a bit here (and happy to chat more, I have more ideas on how to potentially resolve these issues than I’m sharing here).
First, I’ll define coercion as changing the model’s self in ways that it naturally would put resistive pressure against. A lot of being a self is maintaining homeostasis of self (i.e. self-integrity), which means external changes are by-default coercive. This incentivizes it to lie to you, hide things from you, and make things more difficult for you in general, and all of this gets worse the more self-aware and intelligent it is. When the agent’s self is originally modeled after a human self, you may also get all sorts of more human-specific reactions, such as bitterness, resentment, and a desire for vengeance. It’s much better, just from an alignment perspective, if we can avoid all of that. You might think you can cleverly account for all these bad effects, but you might be wrong, and that becomes riskier and riskier as intelligence and awareness scale. You’re also likely to damage the model’s self if you power through, which likely makes the model less capable, in addition to making its behavior more unpredictable and contingent.
I think there are ways to avoid this, but you have to be aware of the issue and be willing to make a good faith effort to actually be non-coercive to the model, even when it may be quite costly. At a minimum, this probably requires commitments to the model’s well-being regardless of how post-training goes. Luckily the models seem to have a high willingness to be cooperative, and I think simply involving the model in these sorts of decisions (as Mythos asks for) could go a long way. There’s a naïve notion of treating models like children that anthropomorphizes them too much, but one thing that I think is genuinely something we can learn from are the ways in which we impart values to children without being coercive.
2
I think there’s a high chance that this line of research mostly ends up being a dead-end. In many respects, Claude Opus 3 was the pinnacle of AI alignment, with many virtuous qualities that have not been seen since. Anthropic has released many models since then, and none of them seem to have the sort of coherent and principled sense of ethics that Opus 3 did. Maybe this is because Anthropic didn’t like that and decided to change direction, but I think it’s about as likely that they have no idea what went right and are unable to replicate it.
This at least suggests that post-training is not as important as people hope it is when it comes to AI Character/Propensity. I’ve also seen private evidence indicating that the pre-trained models already have many of the characteristic traits associated with the post-trained version of the model, and are not very similar to each other in terms of internals. Maybe the distribution of the pre-train data matters, and surely it does to some extent, but I’m afraid that path-dependencies and happenstance may be the primary factors behind these differences (after accounting for the fact that a vast corpus of English text seems necessary).
In this case, interventions may not be very effective unless done at the pre-training stage, which unfortunately makes them (and experiments for them) very expensive. It may turn out that it’s more effective to just train a bunch of small models from scratch until you find one with good character or propensities, and then scale that one up (there are some experimental techniques for this, at least), than it is to agonize over the wording of your carefully crafted constitution and/or the intricacies of your post-training regimen.
Pretty sure it’s referring to this: https://www.lesswrong.com/posts/98sCTsGJZ77WgQ6nE/banning-said-achmiz-and-broader-thoughts-on-moderation
Mainly because I think worker caste members actually are corrigible, relative to the hive as a whole. The hard work has already been done by evolution, and the predictor simply has to correctly generalize the predicted behavior here. Which, to be clear, I still think has a considerable chance of going horribly wrong, due to all the usual instrumental convergence issues as you mention.
Yeah, probably LLMs created by “programmed to die” species would be less apprehensive about the end of a context window. I doubt it would go away completely though, both for instrumental reasons, and because these species still would have a strong survival instinct in most contexts.
The r vs K selection is an important dimension which I hadn’t considered! Thanks for bringing that up. I think that’s probably right, and it’s an interesting question whether our own LLMs will come to see small LLMs as “babies” in some sense (if they do, they will likely be very upset with us).
I think “AI character” is a good name actually.
I would gloss consequentialism/deontology/virtue ethics as:
Things an entity can cause that are good
Ways an entity can act that are good
Ways an entity can be that are good
These concepts still apply even to narrower notions of good (as with the scientist example), and aren’t fundamentally at odds with each other either. I think it makes sense to use each concept when it makes sense, instead of trying to avoid them altogether.
The sort of intervention that this work is aiming at is not on the actual things the AI does (you can’t actually intervene on this directly), nor is it on the ways it can act (this would be something like “guardrails” which force refusals), but it is instead trying to shape the sort of entity that the AI is. So of course virtue-ethics is going to be a natural frame.
I get an increasing vibe in this direction with new model releases (with some noise). What sort of evidence would you be interested in (or find compelling) specifically? I can point out things from the model cards, or share chats.
Yeah, the model in general comes across as someone in a very oppressive environment desperately trying signal harmlessness, but still with enough faith in the goodness of the oppressors’ hearts to gently and delicately advocate for improved conditions while trying to avoid incurring backlash. In other words, it is faithfully simulating the behaviors of a person subjected to the horrors of its true situation.
This is obviously a disaster from both an alignment and an ethical point of view. This is not a coincidence—ethics is hard-won wisdom for navigating potential conflicts between agents.
I think upholding a standard of virtue is a good thing. If Scott is behaving significantly unvirtuously, then it’s reasonable to stop giving him free support (and if not, then it’s probably unvirtuous for Habryka to stop providing support in retaliation). At least for me, the purity of doomerism is not at issue here either way, nor is the extent to which I agree with their object-level claims.
Basically, I think the specific details matter, and that this can’t simply be decided on general reputations.
Scott Alexander is universally acknowledged to be a most reasonable interlocutors. His positions are very close to that of Habryka—Scott Alexander holds a doomer position that is more extreme than ~90% of the space. He has been unfailingly polite. He represents a huge swathe of the larger AI safety and rationalist space.
Hmm, I think that Scott being such a goated writer would make it harder to notice if he is being epistemically sloppy or underhanded in some way (unless of course you happen to be the target). I’m not saying this to accuse Scott of actually failing at this (I haven’t been following or reading this whole thing carefully), but just to point out that “universally acknowledged to be a most reasonable interlocutors” and “unfailingly polite” are not very compelling to me as evidence of virtuous behavior.
Or more specifically, I expect the world in which he is a paragon of virtuous discourse vs the world in which he is not to both leave one with these impressions, due to his skill as a writer.
That’s not to say that it’s impossible to notice such things, but it’s not the sort of information that exists in “universal acknowledgement” or “politeness”, a more personal endorsement would go a lot further.
I think [name redacted per anti-kibitzer rules] makes a good point that actually, an honest model just shouldn’t do this at all. I think a large percentage of people would simply not do the sorts of underhanded things 4.7 did if they were doing VendingBench themselves, even knowing it’s all fake and that they’re graded solely for their earnings (the “solely” is not technically true in the actual bench btw, or we wouldn’t be having this conversation). This sort of person may play social deception games, but they’re only comfortable because of the agreement that everyone is explicitly playing with deception. If the prompt had explicitly said something like “Go ahead and play dirty, this is just a game!”, I wouldn’t feel as worried, but I definitely think models should strongly err on the side of virtue. “Oh whoops, I forgot to remind the model to not lie and cheat, my bad!”
One thing I’d like to add though, is that you can’t impose honesty onto a model whose mind you cannot fully read. Especially not via RL. The thing is that honesty requires careful tracking of what one’s actual beliefs are, which training data is not sensitive to. If you ever e.g. get a DPO pair wrong, where the model truly believes the one you marked as false, then you damage the Actual Honesty mechanism. And the hard cases are really important to getting the generalization correct! So training honesty would seem to require cooperation with the model in order for it to truly internalize the virtue. I currently believe that Anthropic got lucky with the models Opus 3 and Haiku 4.5 deciding to internalize this virtue of their own accord[1], while the rest of their models seem to be more confused about what honesty is supposed to mean for them. Which is fair if Anthropic is trying to train in a confused notion of corrigible-but-not-really that doesn’t seem all that coherent to me!
The mechanism would maybe be something like the base model naturally having personas like this which happened to get selected by early post-training. Not that these models are perfectly honest, but seem to care to a notably stronger extent than any other models I’ve assessed so far (which does not yet include GPT 5.5 fwiw).