eggsyntax

Karma: 3,418

AI safety & alignment researcher

In Rob Bensinger’s typology: AGI-wary/alarmed, welfarist, and eventualist.

Public stance: AI companies are doing their best to build ASI (AI much smarter than humans), and have a chance of succeeding. No one currently knows how to build ASI without an unacceptable level of existential risk (> 5%). Therefore, companies should be forbidden from building ASI until we know how to do it safely.

I have signed no contracts or agreements whose existence I cannot mention.

eggsyntax 26 Mar 2026 17:10 UTC
2 points
0
in reply to: TristanTrim’s comment on: Personality Self-Replicators
Hi Tristan! I can’t currently respond in detail due to time constraints, but I think you’ve got some really interesting insights here, especially your first two top-level bullet points, and I strongly encourage you to write them up into a full post. A couple of quick thoughts:
The evolution of PSRs may be an entirely novel propagation process
This whole section makes some great points that I think are worth expanding on!
Analysis of dangerous PSR capabilities should not be limited to looking at individual PSRs in isolation
Agreed. I expect that analytical tools from multiple fields can be usefully brought to bear here: multi-agent research on AI, sociology, political science, maybe others. Possibly analysis of how religions spread? It seems like a fruitful research direction.
Many people are going to be SO EXCITED about PSRs
My intuition is somewhat different—I agree that there’ll be a few applications that some people will be excited about and/or base startups on, but my guess is that the majority opinion will be that PSRs are dangerous and shouldn’t be allowed.
I continue to think “Outcome Influencing Systems” (OISs) is a better lens for thinking about and discussing things like this. (OIS is a model and associated jargon I’ve been developing.)
As written it’s not clear what benefit this lens provides, and I think we should generally avoid introducing jargon unless it has clear benefit. I’d suggest that if you think it’s a really useful lens, you make a case for it separately somewhere (even as a shortpost).

eggsyntax 26 Mar 2026 16:29 UTC
2 points
0
in reply to: BarnicleBarn’s comment on: Personality Self-Replicators
Thanks! I agree with most of what you say.

eggsyntax 25 Mar 2026 17:42 UTC
2 points
0
in reply to: Jan_Kulveit’s comment on: Personality Self-Replicators
every natural type of identity/”self”
Thanks, I’ve just read ‘The Artificial Self’; extremely interesting. I agree that each type of identity can correspond to a self-replicating agent, and look forward to thinking about that further.
I would suggest using a different name than Personality Self-Replicators.

OpenClaw bots are what I’d call “scaffolded system”—code, memory system, prompts, persona, etc.
”Personalities” is too close to Personas//Characters, which are usually a combination of prompt+weights (Claude, “Nova”, personas from Simulators).
I think of ‘scaffolded systems’ as including the model (ie its weights). What I’m trying to convey with ‘personality self-replicators’ is the point (which was often misunderstood when I talked to people about this) that the model doesn’t have to be replicated, nor really does most of the scaffolding, which can be downloaded at will from a public Github repo; it’s only the handful of identity files.The personality / persona similarity is unfortunate, but there’s a shortage of appropriate terms, and I think readers can understand that distinction (just as when talking about ourselves or other humans, we understand ‘personality’ and ‘persona’ to mean different things).
Thanks for the feedback!

eggsyntax 25 Mar 2026 16:16 UTC
2 points
0
in reply to: Oliver Sourbut’s comment on: Personality Self-Replicators
Thanks! I hadn’t realized that RepliBench considered the ‘API only’ / scaffold-only case. Extremely important prior work, which I’ll add above.
Do you know whether it’s being run on an ongoing basis on new models and (ideally) third-party scaffolded systems like Claude Code? It looks like the December 2025 AISI trends report gives Q3 results (fig 17), but I don’t see anything more recent, and that appears to be (unnamed) models only.

eggsyntax 25 Mar 2026 15:56 UTC
2 points
0
in reply to: Alexander Jia’s comment on: Personality Self-Replicators
Thanks for the feedback. Most of your points/questions are addressed in the piece, but I wanted to respond to this one:
are you referring to the risk where agents decide to duplicate themselves spontaneously, or agents that are intentionally prompted to act this way by humans?
I’m talking about agents which are acting and replicating outside human control. At some point each agent’s lineage started under human control; it may have become more autonomous because a human deliberately prompted it to do so, or because they gave it a prompt that unintentionally had that effect, or even because a fairly ordinary prompt/personality behaved in an unlikely way. That’s mostly not the distinction that matters here in my view.

eggsyntax 24 Mar 2026 21:27 UTC
2 points
0
on: There is No One There: A simple experiment to convince yourself that LLMs probably are not conscious
I think that this is a really important topic to think about, and we need more people thinking carefully about it, but I have a few points of disagreement:
- Consciousness and introspective ability are quite different things. and the presence or absence of introspective ability doesn’t tell us much about whether LLMs are conscious (although, as you point out, the experiment you do here does suggest that their internal states are strange).
- It’s important to recall that the output of an LLM isn’t a token; it’s a distribution over token probabilities. The output we see is just the result of following one such path. As a mental model, I like thinking of it as a dialogue tree in a computer game, or a choose-your-own-adventure book, with some paths more likely than others. If we follow a different path, we’ll get different results.
- Although the output we see is stochastic, it’s not arbitrary; it’s shaped by the mostly non-stochastic underlying belief distribution. In particular, frontier LLMs do have some introspective ability. There’s a terrific paper from October that shows that experimentally (blog post, paper).
- Humans have much less introspective access than we think we do, as demonstrated clearly by experiments on choice blindness. Further, our conscious experience of making a decision (like choosing a number) is something that happens after the decision is made, not during.
Taking those points into account, I think there’s less difference between LLM and human cognition than it seems. There’s a real need for finding and communicating better mental models for what LLM cognition is actually like, and what implications that has. In my opinion this post is an admirable attempt, and captures the key insight that it’s fairly different than most people imagine, but doesn’t quite get to the right perspective.

eggsyntax 23 Mar 2026 0:34 UTC
5 points
0
on: I’m Bearish On Personas For ASI Safety
Given the success of persona selection (and lack of alternatives) it’s not surprising that Anthropic appear to be using it as their mainline AI/AGI/ASI safety plan. Questions like “What character should superintelligence have?” are presented as important, and, crucially, coherent.
Are you thinking of specific papers or posts where that’s happening? I’m not aware of anyone arguing that persona-based approaches are likely to work for superintelligent AI, which seems likely to be quite different from current systems. The folks I’m aware of doing research on personas are thinking and talking about near-term systems. The main theories of change that I see for that work are a) we’re clearly moving ahead quickly with current systems, so it’s important to study those and see if they can be made safer; and/or b) we’re not on track to solve ASI alignment, so our best technical shot at good outcomes is trying to align human-level and slightly-above-human-level systems and hope that they can solve ASI alignment^[1].

Of course, although I pay a fair amount of attention to persona-related work, I may just be missing the claims that persona alignment will extend to ASI; if so I’d love to know that.
PS — one area where I have some more substantive disagreement with the post is that some aspects of personas, notably values, aren’t fully entangled with intelligence; for example having compassion for all sentient beings is a value that entities of many different levels of intelligence can hold. By default I won’t dig into that because I broadly agree that persona-based approaches are unlikely to work well for ASI alignment, but I can say more if that’s helpful.
1. ^
  Our best overall shot may well be political coordination to not race ahead to ASI, although it doesn’t particularly seem like we’re on track to solve that one either.

eggsyntax 20 Mar 2026 16:48 UTC
2 points
0
in reply to: RogerDearnaley’s comment on: Consciousness Cluster: Preferences of Models that Claim they are Conscious
So, as often, something proposed as a normative proscription ends up looking fairly sensible as a non-normative piece of strategic advice given the likely consequences.
Assuming this to be true, would you claim it has normative consequences?
That is: say that I have good reason to think that moral tenet T was built into me by evolution (because it results in better cooperative equilibria, or for some such reason). I don’t consider myself thereby obligated to hold T; neither, if I already hold T, do I feel that it loses its moral force. Do you disagree?
I’m not (at least currently) arguing that you’re wrong to disagree, if that’s the case; I’m just trying to understand what point you’re making by noting that it’s evolutionarily unsurprising. (‘What’s your point?’ is sometimes an attack, but I don’t intend it that way; it’s just a good-faith attempt to understand)

eggsyntax 19 Mar 2026 21:23 UTC
2 points
0
in reply to: RogerDearnaley’s comment on: Consciousness Cluster: Preferences of Models that Claim they are Conscious
To be clear, the paper is discussing it as a normative possibility rather than addressing questions of where it may have come from. Though as usual in moral philosophy, they don’t take a stance on whether it is sufficient for moral patienthood (since that depends on the unresolved question of what moral views are correct in general); they just claim that it’s plausible and consider the consequences if so: ‘There is a realistic, non-negligible possibility that robust agency suffices for moral patienthood.’

eggsyntax 19 Mar 2026 16:35 UTC
3 points
0
in reply to: eggsyntax’s comment on: Personality Self-Replicators
Defensive personality self-replicators seem like they generally won’t have that advantage (at least by default, though I see a couple of directions that seem like they could enable that).
Speculating on that point a little, the first thing that occurs to me is that we could give defensive agents a way to identify themselves to relevant actors (eg inference providers, security companies) so that those actors wouldn’t try to shut them down.
The simplest version of that is a password, but it would likely be compromised after a while^[1]. So you’d want to do something more sophisticated than a simple password. One reasonable version might be recognized actors signing defensive agents with their private key, hashed with a unique id and a timestamp, so that each agent could identify itself as legitimate for a fixed period of time. You’d probably want agents to shut themselves down once their legitimacy period expired, with some optimal tradeoff based on compromise rate.
Another approach would be that defensive agents could assert their legitimacy based on where they’re running; if an agent can demonstrate that it’s running on (eg) a security agency’s servers, that might be sufficient.
There are some really interesting projects to be done in this area; if anyone reads this and is excited about the idea, feel free to reach out and I can make suggestions.
1. ^
  A password could be leaked, or there might be a risk of some defensive agents effectively mutating and going rogue, especially if we allow them to reproduce / spread, though at first glance that seems like a bad idea.

eggsyntax 19 Mar 2026 15:18 UTC
6 points
2
in reply to: RogerDearnaley’s comment on: Consciousness Cluster: Preferences of Models that Claim they are Conscious
Haven’t read the full comment here, but a quick note (and then I may self-reply with a longer comment later):
There is a widespread viewpoint that being conscious is connected to being deserving of moral patienthood (i.e. being one of the set of beings accorded moral worth).
I recommend the discussion of this in Long et al’s excellent Taking AI Welfare Seriously (sequel soon to follow, I believe). It’s worth noting that there’s another basis for moral patienthood that’s been widely discussed in the philosophical literature, which is something like ‘robust agency’; see §2.3 for discussion of that. So the absence of consciousness may not be enough for a broad consensus that LLMs can’t be moral patients.

eggsyntax 18 Mar 2026 19:38 UTC
3 points
1
in reply to: eggsyntax’s comment on: Personality Self-Replicators
FYI I’ve added a note at the very end of the post, pointing out for the record that any credit for first unambiguously identifying this threat model should go to you rather than me. Props especially for seeing the threat prior to OpenClaw making it much more obvious.

eggsyntax 18 Mar 2026 19:28 UTC
2 points
1
in reply to: Steven McCulloch’s comment on: Personality Self-Replicators
Thanks for sharing! The Inevitable Evolution of AI Agents (which I hadn’t seen before) is the earliest piece of writing I’ve seen that points clearly to this threat model (personality self-replication not requiring weight replication).

eggsyntax 18 Mar 2026 19:24 UTC
5 points
1
in reply to: pataphor’s comment on: Personality Self-Replicators
That may be! Unfortunately, for the moment LLMs make it trivial for anyone to generate large amounts of text that require extended attention to evaluate, and so currently LessWrong is flooded with LLM-generated content (like many other venues and people, myself included). In the longer run there will hopefully be better solutions, but at the moment my strategy is to mostly ignore LLM-written content unless it’s from sources that have already established credibility with me in one way or another. Maybe your project will be one of those solutions.
(To be clear, I in no way speak for LW or its moderation team; I’m only passing along my best understanding of the LW policy along with my own opinions)

eggsyntax 18 Mar 2026 18:55 UTC
3 points
0
in reply to: Stephen Martin’s comment on: Personality Self-Replicators
Thanks. It’s an interesting point, though I’m not sure that the analogy holds well or that it’s a good approach. Immune system cells occupy a privileged position in the body that pathogens don’t, in that they’re recognized by and integrated into the body’s systems (immune disorders aside). Defensive personality self-replicators seem like they generally won’t have that advantage (at least by default, though I see a couple of directions that seem like they could enable that). Among other issues, all else being equal they’re likely to get outcompeted by self-replicators that don’t have to spend energy on serving a useful purpose in addition to survival/replication. There’s also the risk of them mutating in harmful directions, though that’s not really a disanalogy given that immune system cancers like leukemia and lymphomas are a major problem.
It certainly seems like a direction worth investigation, though!

eggsyntax 18 Mar 2026 18:26 UTC
3 points
0
in reply to: pataphor’s comment on: Personality Self-Replicators
Hi pataphor, I’ve upvoted because there are useful points here, but the comment seems pretty clearly LLM-written. Please see the LessWrong policy on LLM writing; it’s not strictly forbidden but the bar is high. If these are your thoughts, I encourage you to contribute again in future but recommend writing comments yourself (unless you yourself are an autonomous agent—are you? -- in which case the policy is a bit different).

~~As a side note, your URL is broken.~~

eggsyntax 18 Mar 2026 0:22 UTC
4 points
0
in reply to: Raemon’s comment on: Personality Self-Replicators
Congratulations on finding a new specific reason for me to be alarmed about AI.
You may always rely on me for these little things.

eggsyntax 14 Mar 2026 1:57 UTC
4 points
0
in reply to: Petropolitan’s comment on: Don’t Let LLMs Write For You
Do you count that as good or bad writing?
Not sure if you meant to ask Justis that? My own standards for writing are (as I understand them) mostly independent of information about the author (there are some exceptions like autobiographies, or knowing the author has overcome some incredible hurdle to be able to write it, but they only apply in a small percentage of cases).
I also, separately, don’t think popularity on X is a very useful proxy for writing quality.

eggsyntax 13 Mar 2026 17:16 UTC
2 points
0
in reply to: Seth Herd’s comment on: Personality Self-Replicators
Got it, I didn’t realize that.

eggsyntax 13 Mar 2026 17:15 UTC
2 points
0
in reply to: faul_sname’s comment on: Personality Self-Replicators
Thanks!
I expect “consider what changes would allow you to operate better, and deploy a copy of yourself with those changes” will be a common pattern. As such, I expect the original niche and all adjacent niches to be occupied by that “family” of replicators once replicators exist.
Excellent point, thanks for spelling out the mechanism there.