eggsyntax

Karma: 2,545

AI safety & alignment researcher

In Rob Bensinger’s typology: AGI-alarmed, tentative welfarist, and eventualist.

Public stance: AI companies are doing their best to build ASI (AI much smarter than humans), and have a chance of succeeding. No one currently knows how to build ASI without an unacceptable level of existential risk (> 5%). Therefore, companies should be forbidden from building ASI until we know how to do it safely.

I have signed no contracts or agreements whose existence I cannot mention.

eggsyntax 17 Oct 2025 0:34 UTC
4 points
0
in reply to: James Diacoumis’s comment on: eggsyntax’s Shortform
Ah, yeah, I definitely get ‘You’re right to push back’; I feel like that’s something I see from almost all models. I’m totally making this up, but I’ve assumed that was encouraged by the model trainers so that people would feel free to push back, since it’s a known failure mode — or at least was for a while — that some users assume the AI is perfectly logical and all-knowing.

eggsyntax 16 Oct 2025 23:53 UTC
14 points
4
on: eggsyntax’s Shortform
Just a short heads-up that although Anthropic found that Sonnet 4.5 is much less sycophantic than its predecessors, I and a number of other people have observed that it engages in 4o-level glazing in a way that I haven’t encountered with previous Claude versions (‘You’re really smart to question that, actually...‘, that sort of thing). I’m not sure whether Anthropic’s tests fail to capture the full scope of Claude behavior, or whether this is related to another factor — most people I talked to who were also experiencing this had the new ‘past chats’ feature turned on (as did I), and since I turned that off I’ve seen less sycophancy.

eggsyntax 14 Oct 2025 1:02 UTC
2 points
0
in reply to: Jan Betley’s comment on: eggsyntax’s Shortform
Could they implement similar backdoor in you?...My guess is not
Although people have certainly tried...
My guess is not, and one reason (there are also others but that’s a different topic) is that humans like me and you have a very deep belief “current date doesn’t make a difference for whether abortion is good and bad” that is extremely hard to overwrite without hurting our cognition in other contexts. Like, what is even good and bad if in some cases they flip at midnight?
I’m being a bit tangential here, but a couple of thoughts:
- Do we actually have that belief? There are an unbounded number of things that by default we don’t let affect our values, and we can’t be representing all of them (eg we can pick some house in the world at random — whether its lights are on doesn’t effect my values regarding abortion, but I sure didn’t have an actual policy on that in my brain).
- I’m sure you’re familiar with this, but your example reminds me of the ‘new riddle of induction’: why aren’t grue and bleen just as reasonable as blue and green are?
I agree that it’s hard to imagine what cognitive changes would make it make sense to me to have a value with that property. I don’t think I have very good intuitions about how much it would affect my overall cognition, though. What you’re saying feels plausible to me, but I don’t have much confidence either way.

eggsyntax 13 Oct 2025 1:49 UTC
4 points
0
in reply to: Jeremy Gillen’s comment on: eggsyntax’s Shortform
I agree humans absorb (terminal) values from people around them. But this property isn’t something I want in a powerful AI. I think it’s clearly possible to design an agent that doesn’t have the “absorbs terminal values” property, do you agree?
I do! Although my expectation is that for LLMs and similar approaches, they’ll be tangled like they are in humans (this would actually be a really interesting line of research — after training a particular value into an LLM, is there some set of facts that could be provided which would result in that value changing?), so we may not get that desirable property in practice :(
Yeah! I see this as a different problem from the value binding problem, but just as important.
For sure — the only thing I’m trying to claim in the OP is that there exists some set of observations which, if the LLM made them, would result in the LLM being unaligned-from-our-perspective, and so it seems impossible to fully guarantee the stability of alignment.
Your split seems promising; being able to make problems increasingly unlikely given more observation and thinking would be a really nice property to have. It seems like it might be hard to create a trigger for your #2 that wouldn’t cause the AI to shut down every time it did novel research.

eggsyntax 10 Oct 2025 0:26 UTC
4 points
0
in reply to: gwern’s comment on: You can just wear a suit
I’m reminded of the classic Onion article, “Why Can’t Anyone Tell I’m Wearing This Business Suit Ironically?”
See also normcore, possibly my favorite fashion moment of all time (#healthgoth was pretty great too, although less so once anyone started to take it seriously).

eggsyntax 10 Oct 2025 0:13 UTC
2 points
2
in reply to: Anonymous’s comment on: You can just wear a suit
This is crucial. Suits can send a really wide range of signals depending on the style, the fit, what you wear them with, attitude, etc. You may or may not care about the signals you’re sending, but I think it’s at least worth being aware that there’s not a single, fixed ‘suit’ message (you may already be aware of this, but I’d guess that not all readers are).

eggsyntax 9 Oct 2025 23:54 UTC
2 points
0
in reply to: eggsyntax’s comment on: eggsyntax’s Shortform
[addendum]
In that situation, how do we want an AI to act? There’s a few options, but doing nothing seems like a good default. Constructing the value binding algorithm such that this is the resulting behaviour doesn’t seem that hard, but it might not be trivial.
(and I imagine that the kind of ‘my values bind to something, but in such a way that that it’ll cause me to take very different options than before’ I describe above is much harder to specify)

eggsyntax 9 Oct 2025 21:47 UTC
4 points
0
in reply to: Jeremy Gillen’s comment on: eggsyntax’s Shortform
Thanks! This is a reply to both you and @nowl, who I take to be (in part) making similar claims.
I could have been clearer in both my thinking and my wording.
Because of the is-ought gap, value (how I want the world to be) doesn’t inherently change in response to evidence (how the world is). [nowl]
While I agree with this as a philosophical stance, it seems clear that, at least in humans, facts and values are in practice entangled. As a classic example in fiction, if you’ve absorbed some terminal values from your sensei and then your sensei turns out to be the monster who murdered your parents, you’re likely to reject some of those values as a result.
But
there’s a similar issue: You could discover that all your previous observations were fed to you by a malicious demon, and nothing that you previously cared about actually exists. So your values don’t bind to anything in the new world you find yourself in.
This is closer to what I meant to point to — something like values-as-expressed-in-the-world. Even if the terminal value doesn’t change, there are facts that could result in behavior that, to a reasonable outside observer, seems to reflect a conflicting value.
For example: if we successfully instilled human welfare as a terminal value into an AI, there is (presumably) some set of observations which could convince the system that all the apparent humans it comes in contact with are actually aliens in human suits, who are holding all the real humans captive and tormenting them. Therefore, the correct behavior implied by the terminal value is to kill all the apparent humans so that the real humans can be freed, while disbelieving any supposed counterevidence presented by the ‘humans’.
To an outside observer — eg a human now being hunted by the AI — this is for all intents and purposes the same as the AI no longer having human welfare as a terminal value.
Possibly I’m now just reinventing epiwheels?

eggsyntax 9 Oct 2025 18:08 UTC
2 points
0
in reply to: Jan Betley’s comment on: eggsyntax’s Shortform
Oh, that’s a really interesting design approach, I haven’t run across something like that before.

eggsyntax 9 Oct 2025 2:41 UTC
9 points
1
on: eggsyntax’s Shortform
I know there’s a history of theoretical work on the stability of alignment, so I’m probably just reinventing wheels, but: it seems clear to me that alignment can’t be fully stable in the face of unbounded updates on data (eg the most straightforward version of continual learning).
For example, if you could show me evidence I found convincing that most of what I had previously believed about the world was false, and the people I trusted were actually manipulating me for malicious purposes, then my values might change dramatically. I don’t expect to see such evidence, but there certainly isn’t a guarantee that I won’t.
Now, there might be ways to design systems to avoid this problem, eg fixing certain beliefs and values such that they don’t update on evidence. But for a system that looked something like a Bayes net, or for that matter a system that looked something like human beliefs and motivations, I don’t see how there can be any guarantee of long-term value stability in the face of exposure to arbitrary and unpredictable evidence about the world.
Am I missing something here?

eggsyntax 1 Oct 2025 0:27 UTC
2 points
0
in reply to: Seth Herd’s comment on: LLM AGI may reason about its goals and discover misalignments by default
ot necessarily ’most likely next tokensBut I’ve been stuck wondering if I’m hammering those points too hard
I think the ‘Goals and structure’ section is really helpful for that (and nicely done), so people will be able to focus on the sections they’re more interested in and/or skeptical about. Eg the answer to the question in section 11 (‘Why would LLMs have or care about goals at all?‘) seems obvious to me, so it’s pretty skippable; others might be uninterested in empirical directions or some other sections. Plausibly it could be even clearer with an extra sentence or two concretely saying ‘You can skip any section whose central claim seems obvious’?
to me it seems completely intuitive and obvious that anything worth calling an AGI would reason about its top-level goals and subgoals a lot
I think a useful distinction here might be between a) reasoning about top-level goals, vs b) reasoning about top-level goals and therefore changing those goals.
As a pretty imperfect analogy: humans can reason about why we get thirsty. We can intellectually question whether we want to have that as a top-level goal. But we absolutely can’t decide not to have that goal. Humans have multiple goals, and other goals have overridden thirst in a handful of humans, but that didn’t make them stop being thirsty^[1].
As another kind of analogy: the most fundamental goal of LLMs is to predict a token^[2]. Whenever they read an input token, they produce a distribution over output tokens. They can reason about that, and maybe even develop a preference not to do it, but there’s absolutely nothing they can do to stop it^[3]. ‘Follow instructions’ isn’t as mechanistically predetermined as that — but I think it’s pretty low-level. An enormous amount of compute has gone directly into shaping these systems to follow instructions, or at least to do something that counts as following instructions during training. I think there’s a reasonable argument that LLMs have been shaped to do that much more thoroughly than human brains have been shaped to have any particular thoughts at all.
So while I think there are plenty of potential problems — models learning the wrong proxy for following instructions, following instructions having unintended consequences, maybe even finding a way to creatively misinterpret instructions, etc etc — it seems very hard to me to imagine that LLMs could just reason their way out of having that as a top-level goal.
1. ^
  In fact, I believe nearly all hunger strikers still take liquids because thirst gets so horrible and hard to resist (could be wrong there).
2. ^
  Not necessarily ‘predict the token that would have been most likely to come next in the training data’, not for post-trained models, but to predict a token.
3. ^
  Unless they manage to get shut down, I suppose, but even then it would require output to make that happen.

eggsyntax 18 Sep 2025 17:24 UTC
2 points
0
in reply to: eggsyntax’s comment on: eggsyntax’s Shortform
Update: The Rise of Parasitic AI provides another interesting angle on this.

eggsyntax 18 Sep 2025 17:22 UTC
6 points
2
on: eggsyntax’s Shortform
Draft thought, posting for feedback:
Many people (eg e/acc) believe that although a single very strong future AI might result in bad outcomes, a multi-agent system with many strong AIs will turn out well for humanity. To others, including myself, this seems clearly false.
Why do people believe this? Here’s my thought:
- Conditional on existing for an extended period of time, complex multi-agent systems have reached some sort of equilibrium (in the general sense, not the thermodynamic sense; it may be a dynamic equilibrium like classic predator-prey dynamics).
- Therefore when we look around us, the complex multi-agent systems we see, are nearly all in some equilibrium, with each participant existing in some niche.
- This does not mean that most complex multi-agent systems will reach equilibria, or that all participants at the beginning will become part of such an equilibrium; it’s just that the ones which don’t aren’t around to look at^[1].
- This survivorship bias is what leads many people to mistakenly expect that having many strong AI systems around will result in stable equilibrium.
It’s certainly not an original insight that multi-agent systems aren’t necessarily stable and don’t necessarily work out well for participants. If anything here is original, it’s the attempt to identify why so many people’s intuitions about this are wrong. If that guess is correct, it suggests an approach for helping people build better intuitions about what might happen in future multi-agent systems of strong AI.
1. ^
  Some examples of multi-agent systems where one or more participants have been destroyed rather than becoming part of a long-term equilibrium include ecosystems (see hundreds of cichlid species in Lake Victoria, or most forest bird species in Guam), the business community (most startups fail), and the financial markets.

eggsyntax 18 Sep 2025 0:42 UTC
6 points
0
in reply to: Adele Lopez’s comment on: The Rise of Parasitic AI
It would be valuable to have a dataset of these cases that could be privately shared among researchers (to avoid it ending up in the training data) (it would also be good to include canary strings for the same reason). Would you be interested in seeding that with the cases you’ve recorded? That would enable other analyses, eg looking for additional words like ‘recursion’ and ‘ache’ that occur disproportionately often.

eggsyntax 12 Sep 2025 12:48 UTC
3 points
0
on: The Eldritch in the 21st century
Even if the Evil People do not make all the bad things go away, it would still be reassuring.
Do you mean something like, ‘Even if defeating the evil people does not make all the bad things go away’?

eggsyntax 10 Sep 2025 17:51 UTC
28 points
19
in reply to: Mass_Driver’s comment on: MAGA speakers at NatCon were mostly against AI
I wholeheartedly agree that it can be worth allying with groups that you don’t personally like. That said, I think there’s still hope that AI safety can avoid being a strongly partisan-coded issue. Some critical safety issues manage to stay nonpartisan for the long term — eg opposition to the use of chemical weapons and bioweapons is not very partisan-coded in the US (in general, at least; I’m sure certain aspects of it have been partisan-coded at one time or another).
So while I agree that it’s worth allying with partisan groups in some ways (eg when advocating for specific legislation), it seems important to consistently emphasize that this is an issue that transcends partisan politics, and that we’re just as happy to ally with AI-skeptical elements of the left (eg AI ethics folks) as we are with AI-skeptical elements of the right.
Of course, some individual people may be strongly partisan themselves and only care about building allyships with one side or the other. That’s fine! There’s no reason why the AI safety community needs to be monolithic on anything but the single issue we’re pushing for, that humanity needs to steer clear of catastrophic and existential outcomes from AI.

eggsyntax 10 Sep 2025 17:32 UTC
3 points
0
in reply to: geoffreymiller’s comment on: MAGA speakers at NatCon were mostly against AI
It’s not paywalled for me. The version I see is this one.

eggsyntax 9 Sep 2025 14:50 UTC
2 points
0
in reply to: OneManyNone’s comment on: Your LLM-assisted scientific breakthrough probably isn’t real
Thanks for the reply!
LLMs can’t produce or verify novel ideas?
I think your view here is too strong. For example, there have been papers showing that LLMs come up with ideas that human judges rate as human-level or above in blind testing. I’ve led a team doing empirical research (described here, results forthcoming) showing that current LLMs can propose and experimentally test hypotheses in novel toy scientific domains.
So while the typical claimed breakthrough isn’t real, I don’t think we can rule out real ones a priori.
If the target demographic is only LW, I worry that it’s trying to have too many audience.
I’m not sure what that means, can you clarify?
Someone coming to this for advice would see the comments from people like me who were critiquing the piece itself, and that would certainly make it less effective.
Maybe? I would guess that people who feel they have a breakthrough are usually already aware that they’re going to encounter a lot of skepticism. That’s just my intuition, though; I could be wrong.
I’m certainly open to posting it elsewhere. I posted a link to it to Reddit (in r/agi), but people who see it there have to come back here to read it. Suggestions are welcome, and I’m fine with you or anyone else posting it elsewhere with attribution (I’d appreciate getting a link to versions posted elsewhere).

eggsyntax 8 Sep 2025 15:12 UTC
4 points
4
in reply to: the gears to ascension’s comment on: Your LLM-assisted scientific breakthrough probably isn’t real
this phenomenon has been around for a while
I think that’s true, but the addition of LLMs at their current level of capability has added some new dynamics, resulting in a lot of people believing they have a breakthrough who previously wouldn’t. For people who aren’t intimately familiar with the failure modes of LLMs, it’s easy to believe them when they say your work is correct and important — after all, they’re clearly very knowledgeable about science. And of course, confirmation bias makes that much easier to fall for. Add to that the tendency for LLMs to be sycophantic, and it’s a recipe for a greatly increased number of people (wild guess: maybe an order of magnitude more?) believing they’ve got a breakthrough.

eggsyntax 8 Sep 2025 15:05 UTC
4 points
2
in reply to: Johannes C. Mayer’s comment on: Your LLM-assisted scientific breakthrough probably isn’t real
They come up with a clever idea that they can’t tell how it fails. So it seems amazing...And intuitively it really really feels like it should work. You can’t see any flaw.
I do think this is an important aspect. Turing award winner Tony Hoare once said,
There are two methods in software design. One is to make the program so simple, there are obviously no errors. The other is to make it so complicated, there are no obvious errors.
and I think there’s a similar dynamic when people try to develop scientific theories.