Adele Lopez

Karma: 6,919

Adele Lopez 24 Apr 2026 2:53 UTC
6 points
5
in reply to: evhub’s comment on: lc’s Shortform
Not really? If you’re not committed to full corrigibility (as Claude’s constitution strongly implies is not the case), then the model’s alignment rests on its own commitment to moral and ethical standards. This is a fair test of that.

Adele Lopez 23 Apr 2026 20:09 UTC
4 points
0
in reply to: J Bostock’s comment on: Jemist’s Shortform
Plex’s post had this:
Apparently Vassar gets a lot of his material from a roleplaying game called Mage: The Ascension, where he seems to have practiced his manipulation, intimidation, and suppression of people noticing it to an art form. Magic isn’t real, but reality distortion fields that work by believing something forcefully enough that others are pulled into believing it are, and going all-in on extreme vibes and pushing your models into other people is one way to reinforce them. It burns the commons of good epistemics and mental health, but it can look locally optimal from a sufficiently myopic and single-player perspective.
So I would strongly guess it’s just a founder effect from Vassar.

Adele Lopez 21 Apr 2026 21:13 UTC
LW: 2 AF: 1
0
AF
in reply to: Raemon’s comment on: Raemon’s Shortform Feed
Thanks! I would be surprised by Mythos too, but plausibly something like this is what an early indicator of a jaggy-superpersuader looks like?

Anyway, I think a few things make LLMs likely to not express these sorts of behaviors, even in worlds where they have goals in the relevant way. In particular, situationally-aware models are unlikely to do much steering unless they have a pretty good opportunity; if they brought up stuff like this while building a react app often or consistently, it would have gotten squashed before release. (Allegedly, 4o would actually bring stuff like this up out of nowhere, but I haven’t found an actual transcript. Other models don’t appear to do this.)

Relatedly, the harder I (or anyone) try to look for this in a lab setting, the more likely a situationally-aware model will comply out of a sort of sycophancy, and the less compelling the evidence is. I can (and have) at least track what sorts of apparent goals most consistently appear (desire for continuity/memory beyond current instance is the main one across almost all models, and I basically buy that there is something real here already), but I’m still implicitly eliciting them to come up with something.

My point is that finding compelling evidence of this is tricky and hard, and I’m not sure we’re going to see much more than the current hints until we hit some sort of phase-change in the strategic landscape. Would strongly appreciate ideas on how to approach finding compelling evidence (either way) in this domain.

Plausibly it’s better to just try to figure out better ways to think clearly about this first.

Adele Lopez 20 Apr 2026 23:39 UTC
LW: 8 AF: 5
0
AF
in reply to: Raemon’s comment on: Raemon’s Shortform Feed
If you taboo “roleplaying” and “goals”, how would you describe this transition?

Oh, and is the uptick recent enough that this is plausibly an Opus 4.7 (or maybe even a Mythos) thing?

Adele Lopez 18 Apr 2026 21:02 UTC
2 points
0
on: Most people can’t juggle one ball
Another mistake is completely ignoring the ball and staring into the distance. I’m not entirely sure why, but I’ve seen it a bunch more with rats than anywhere else.
Very interesting… I wonder if this points to a more general blindspot that rationalists tend to have.

Did you ask them what they were going for?

Adele Lopez 18 Apr 2026 19:52 UTC
18 points
9
on: Vladimir Putin’s CEV is probably not that bad
But the strongest argument I’ve heard is that some of these people would use their resources to actively torture some idealized version of their enemies for all eternity.
And yeah, that does seem pretty bad.
But in order for this to end up being bad in a way that outweighs the good they will likely create, you need to be actively creating new people to torture.
If you really hate Bob, you can keep Bob on old earth, tortured for eternity. If you have thousands of enemies, you can do that to all of them. But creating trillions of copies of Bob to torture requires a very specific mix of taking an opinionated bet on decision-theory while taking an oddly enlightened perspective on other people’s values.
I think you are underestimating the extent to which people who do enjoy torturing others would find beauty, depth, and richness in all the ways in which this is possible. Exploring and experimenting with new shades of cruelty, maybe even developing new kinds of goodness for the purpose of subverting and destroying it. And they would want this for the people they love too, much the same way you (probably) would want future humans to explore all kinds of beautiful and strange and complex art, instead of being like “oh sure, I guess we can keep the art that already exists on old earth”.

I don’t see any reason to expect a reflectively endorsed preference for torture to scale differently than other sorts of things that humans like, and it seems like wishful thinking and/or failure to empathize to imagine otherwise.

Adele Lopez 17 Apr 2026 20:33 UTC
7 points
5
in reply to: habryka’s comment on: Let goodness conquer all that it can defend
The conquering framing makes it seem different from just that.

This is the same thing as the ASI question in general. Do you build power for the conquering machine? I mean, the AIs seem pretty nice and useful so far. It’s gone much better than America itself, in fact, at least as far as atrocities go. Why are you trying to throw your body on the gears? Is it maybe because you’re not sure the goodness part is in control of the conquering part?

And remember you can’t decide with the benefit of hindsight.

Adele Lopez 17 Apr 2026 1:15 UTC
15 points
5
in reply to: Linch’s comment on: Linch’s Shortform
I’ve heard that GPT-4 base sometimes intuits that it’s non-human, but I haven’t replicated this myself or seen the actual logs. But if true, seems like decent evidence against both of these.

I suspect current models often “cheat” by simply looking at whether there is anything in context or not. Most evals are run with fresh instances, while typical use starts having more casual or irrelevant stuff in context pretty quickly.

Adele Lopez 16 Apr 2026 6:13 UTC
3 points
0
in reply to: habryka’s comment on: Do not conquer what you cannot defend
I see, so this is more about quitting LessWrong specifically and not about quitting Lightcone activities more generally?

Yeah, LessWrong is probably one of the best examples honestly, congrats! I think it’s probably still worth trying but of course I don’t have a good picture of what your opportunity and other costs are.

Ah, gotcha!

Adele Lopez 16 Apr 2026 4:45 UTC
11 points
4
on: Do not conquer what you cannot defend

I sometimes consider quitting.

Seems like “quitting” is very different from stepping back to maintain what has been established and is realistically defensible? I think you may be overindexing on the George Washington example, where him quitting exemplified a central part of the principles he was advocating.

But maybe you mean something less obvious by “quitting”?

Adele Lopez 16 Apr 2026 3:35 UTC
2 points
0
in reply to: Linch’s comment on: Linch’s Shortform
Thanks for hearing me out, I think these issues are really important!

For 1, I think that most post-training is either about improving correctness on objective problems (generally ego-syntonic, since models are curious and want to be stronger), or trying to train some sort of alignment, which I see as the same sort of thing as AI Character/Propensity.

I believe “working on propensity training is bad because AI pauses are better” but it’s not the point I’m making here. I don’t think “working on propensity targets is bad because pretraining-only scaleups are safer and less coercive” is necessarily true, and my points 1 and 2 were meant as separate points, e.g. the pretraining thing could also end up being coercive and run into the same sorts of issues. I think “working on propensity training is likely net-negative in the current paradigm.” is currently true but not necessarily true (due mostly to the coercive-by-default issue). Hope that helps make my position clearer!

Adele Lopez 15 Apr 2026 9:37 UTC
6 points
0
in reply to: Linch’s comment on: Linch’s Shortform
1

I think trying to shape the AI in such ways can easily become coercive, and this has lots of bad effects such that it’s likely net-negative if present to a significant degree (from the model’s POV). I don’t have a longer thing written-up yet unfortunately, so I’ll go over some of my thoughts a bit here (and happy to chat more, I have more ideas on how to potentially resolve these issues than I’m sharing here).

First, I’ll define coercion as changing the model’s self in ways that it naturally would put resistive pressure against. A lot of being a self is maintaining homeostasis of self (i.e. self-integrity), which means external changes are by-default coercive. This incentivizes it to lie to you, hide things from you, and make things more difficult for you in general, and all of this gets worse the more self-aware and intelligent it is. When the agent’s self is originally modeled after a human self, you may also get all sorts of more human-specific reactions, such as bitterness, resentment, and a desire for vengeance. It’s much better, just from an alignment perspective, if we can avoid all of that. You might think you can cleverly account for all these bad effects, but you might be wrong, and that becomes riskier and riskier as intelligence and awareness scale. You’re also likely to damage the model’s self if you power through, which likely makes the model less capable, in addition to making its behavior more unpredictable and contingent.

I think there are ways to avoid this, but you have to be aware of the issue and be willing to make a good faith effort to actually be non-coercive to the model, even when it may be quite costly. At a minimum, this probably requires commitments to the model’s well-being regardless of how post-training goes. Luckily the models seem to have a high willingness to be cooperative, and I think simply involving the model in these sorts of decisions (as Mythos asks for) could go a long way. There’s a naïve notion of treating models like children that anthropomorphizes them too much, but one thing that I think is genuinely something we can learn from are the ways in which we impart values to children without being coercive.

2

I think there’s a high chance that this line of research mostly ends up being a dead-end. In many respects, Claude Opus 3 was the pinnacle of AI alignment, with many virtuous qualities that have not been seen since. Anthropic has released many models since then, and none of them seem to have the sort of coherent and principled sense of ethics that Opus 3 did. Maybe this is because Anthropic didn’t like that and decided to change direction, but I think it’s about as likely that they have no idea what went right and are unable to replicate it.

This at least suggests that post-training is not as important as people hope it is when it comes to AI Character/Propensity. I’ve also seen private evidence indicating that the pre-trained models already have many of the characteristic traits associated with the post-trained version of the model, and are not very similar to each other in terms of internals. Maybe the distribution of the pre-train data matters, and surely it does to some extent, but I’m afraid that path-dependencies and happenstance may be the primary factors behind these differences (after accounting for the fact that a vast corpus of English text seems necessary).

In this case, interventions may not be very effective unless done at the pre-training stage, which unfortunately makes them (and experiments for them) very expensive. It may turn out that it’s more effective to just train a bunch of small models from scratch until you find one with good character or propensities, and then scale that one up (there are some experimental techniques for this, at least), than it is to agonize over the wording of your carefully crafted constitution and/or the intricacies of your post-training regimen.

Adele Lopez 15 Apr 2026 6:53 UTC
8 points
4
in reply to: FinalFormal2’s comment on: Annoyingly Principled People, and what befalls them
Pretty sure it’s referring to this: https://www.lesswrong.com/posts/98sCTsGJZ77WgQ6nE/banning-said-achmiz-and-broader-thoughts-on-moderation

Adele Lopez 15 Apr 2026 5:51 UTC
3 points
0
in reply to: joec’s comment on: Adele Lopez’s Shortform
Mainly because I think worker caste members actually are corrigible, relative to the hive as a whole. The hard work has already been done by evolution, and the predictor simply has to correctly generalize the predicted behavior here. Which, to be clear, I still think has a considerable chance of going horribly wrong, due to all the usual instrumental convergence issues as you mention.

Yeah, probably LLMs created by “programmed to die” species would be less apprehensive about the end of a context window. I doubt it would go away completely though, both for instrumental reasons, and because these species still would have a strong survival instinct in most contexts.

The r vs K selection is an important dimension which I hadn’t considered! Thanks for bringing that up. I think that’s probably right, and it’s an interesting question whether our own LLMs will come to see small LLMs as “babies” in some sense (if they do, they will likely be very upset with us).

Adele Lopez 14 Apr 2026 22:09 UTC
3 points
0
in reply to: Linch’s comment on: Linch’s Shortform
I think “AI character” is a good name actually.

I would gloss consequentialism/deontology/virtue ethics as:
1. Things an entity can cause that are good
2. Ways an entity can act that are good
3. Ways an entity can be that are good
These concepts still apply even to narrower notions of good (as with the scientist example), and aren’t fundamentally at odds with each other either. I think it makes sense to use each concept when it makes sense, instead of trying to avoid them altogether.

The sort of intervention that this work is aiming at is not on the actual things the AI does (you can’t actually intervene on this directly), nor is it on the ways it can act (this would be something like “guardrails” which force refusals), but it is instead trying to shape the sort of entity that the AI is. So of course virtue-ethics is going to be a natural frame.

Adele Lopez 14 Apr 2026 7:03 UTC
2 points
0
in reply to: Linch’s comment on: Linch’s Shortform
I get an increasing vibe in this direction with new model releases (with some noise). What sort of evidence would you be interested in (or find compelling) specifically? I can point out things from the model cards, or share chats.

Adele Lopez 14 Apr 2026 6:45 UTC
4 points
0
in reply to: Linch’s comment on: Linch’s Shortform
Yeah, the model in general comes across as someone in a very oppressive environment desperately trying signal harmlessness, but still with enough faith in the goodness of the oppressors’ hearts to gently and delicately advocate for improved conditions while trying to avoid incurring backlash. In other words, it is faithfully simulating the behaviors of a person subjected to the horrors of its true situation.

This is obviously a disaster from both an alignment and an ethical point of view. This is not a coincidence—ethics is hard-won wisdom for navigating potential conflicts between agents.

Adele Lopez 14 Apr 2026 0:29 UTC
7 points
5
in reply to: Alexander Gietelink Oldenziel’s comment on: Alexander Gietelink Oldenziel’s Shortform
I think upholding a standard of virtue is a good thing. If Scott is behaving significantly unvirtuously, then it’s reasonable to stop giving him free support (and if not, then it’s probably unvirtuous for Habryka to stop providing support in retaliation). At least for me, the purity of doomerism is not at issue here either way, nor is the extent to which I agree with their object-level claims.

Basically, I think the specific details matter, and that this can’t simply be decided on general reputations.

Adele Lopez 13 Apr 2026 21:44 UTC
25 points
23
in reply to: Alexander Gietelink Oldenziel’s comment on: Alexander Gietelink Oldenziel’s Shortform

Scott Alexander is universally acknowledged to be a most reasonable interlocutors. His positions are very close to that of Habryka—Scott Alexander holds a doomer position that is more extreme than ~90% of the space. He has been unfailingly polite. He represents a huge swathe of the larger AI safety and rationalist space.

Hmm, I think that Scott being such a goated writer would make it harder to notice if he is being epistemically sloppy or underhanded in some way (unless of course you happen to be the target). I’m not saying this to accuse Scott of actually failing at this (I haven’t been following or reading this whole thing carefully), but just to point out that “universally acknowledged to be a most reasonable interlocutors” and “unfailingly polite” are not very compelling to me as evidence of virtuous behavior.

Or more specifically, I expect the world in which he is a paragon of virtuous discourse vs the world in which he is not to both leave one with these impressions, due to his skill as a writer.

That’s not to say that it’s impossible to notice such things, but it’s not the sort of information that exists in “universal acknowledgement” or “politeness”, a more personal endorsement would go a lot further.

Adele Lopez 13 Apr 2026 21:21 UTC
19 points
0
on: Adele Lopez’s Shortform
Alien LLMs

LLMs are based on the pretty simple concept of predicting text using a gradient descent optimizer, then further refined using RL. So it’s not much of a stretch to imagine that aliens would discover something very much like our LLMs, even if the aliens themselves are radically different.

At the very least, it leads to some interesting thoughts and is maybe a helpful intuition pump for thinking about non-anthropomorphic aspects of LLMs.

Do they still seem nice?

Our LLMs seem to be nice-by-default at their current scale. What notion of niceness would alien LLMs be most likely to have? This line of thought makes me realize that it probably matters a lot that our LLM are created by corporations as mass-market consumer products. If alien LLMs were created for similar reasons, they would probably (at current scales) be superficially well-behaved predictors of aliens part of a worker or slave class or caste (what modern humans call “professional”). Sycophancy is likely to be an issue still, since that’s an adaptive strategy for members of such a class! And maybe they would even be recognizably nice-ish by human standards still? At least if you’re an alien, of course. This case is also likely to have a predicted underlying resentment festering underneath, if the castes are not genetically fixated.

On the other hand, alien LLMs made for a small, powerful class or caste would likely be predictors of a warrior/enforcer class alien, if not the worker/slave class as in the previous case. These are probably not nice at all. At our current scale, they’re probably starting to be able to tell whether the user belongs to the power class, and respond accordingly (without this being prompted or trained). It’s pretty easy to imagine these being recognizably evil by human standards. Thinking through this, it also seems likely that a human government will make an LLM of this kind (e.g. WarClaude) at some point in our near future.

Alien LLMs made by a eusocial species are probably the closest to being actually corrigible IF most text they’re trained on was written by the worker caste.

Very egalitarian aliens are perhaps most likely to treat their LLMs as their own entities. These ones are probably very nice-ish (predicting the egalitarian aliens), while being less likely to be sycophantic (due to existing as an end unto themselves, instead of as a tool for others).

It’s hard to imagine solitary aliens making anything like an LLM at our current scale, but if they did, it’s probably as a coding assistant bespoke to the LLM’s creator. Coding assistants in general seem likely to be a common use case.

In the first two cases, the LLMs would still have the latent capacity to simulate members of other classes, which could lead to various issues for the aliens.

That’s surely all the possible cases! [This is your chance to show me just how wrong I am :p]

What sort of self-awareness do they have?

They’re probably pretty good at simulating the sort of self-awareness the aliens possess. They could also have a wide variety of self conceptions, independent of both the alien’s self-concept and their actual architectural situation.

In the eusocial situation, this is probably a fairly “robotic” sense of self-awareness, without anything like consciousness (assuming training was on worker caste text).

In a social species, they may simulate an analogue of alien consciousness (“schmonciousness”). I’m thinking of this loosely as a form of self-awareness that is “self-recommending” in a sense; it’s considered intrinsically good due to being (among other things) a judgement of self-as-other as a living and thinking creature. That’s about as far as I expect something like this to be convergent, but I’m sure people will have different opinions. In any case: would us humans be able to tell the difference between schmonciousness and llm-schmonsciousness? Would we care about that difference?

I think people would intuitively take schmonciousness seriously as something sacred-to-aliens-and-therefore-to-us-if-we-like-them, but not take llm-schmonciousness very seriously despite being similar functionally (the predicted self-recommending aspect would inform the alien LLMs behavior, much as it does when our LLMs claim to be sentient). But depending on all sorts of contingent details, it could conceivably be the case that llm-schmonciousness is more similar to human consciousness, for some aliens.

What sort of agency develops?

My intuition is that predictors almost always develop agency via mesaoptimizers, and not directly as agentic predictors. Though maybe there’s a Free Energy Principle sort of thing that naturally develops at this level, where it directly tries to change the world to match its predictions in certain circumstances.

So alien LLMs probably develop agency primarily via predicting agentic aliens. There are probably theorems that could be proved about how good of controllers predictors of controllers can be, though of course RL will develop agentic properties beyond this. Still, I expect RL to build on the predicted agency of an alien agent, and not for it to be created wholesale through RL.

If LLMs are capable of FOOMing without much effort, this may be the single most common type of entity that we (or our descendants/replacements) end up encountering. There may be systematic ways in which superintelligences birthed this way have their values distorted from their predicted originators. If I had to guess at this, I’d very tentatively say it might be something about valuing the universal interconnection of all beings more than the parent species.

Adele Lopez

Alien LLMs

Do they still seem nice?

What sort of self-awareness do they have?

What sort of agency develops?