Ben Livengood

Karma: 327

Ben Livengood 18 Sep 2026 0:44 UTC
3 points
2
on: Superintelligence this Christmas
I note they don’t have AL5 on the graph yet; I think that would need to appear at least a week or two before fully automated RSI. I wonder if they’ll keep the dashboard up to date daily.

Ben Livengood 13 Sep 2026 2:12 UTC
2 points
−1
on: The Talker Does Not Control The Doer (in Current AIs)
There may be a related phenomenon that became really pronounced with Claude Opus 5. I originally thought that I had finally tracked it down to over-reliance on thinking tokens. The symptom was a lot more invented jargon to describe concepts, procedures, problems, implementation details, etc. in both the code and prose (i.e. Claudese). What I finally prompted to change the behavior was something like “do not use any direct copies of thinking tokens in your code or prose, use standard English and appropriate real-world technical jargon or jargon from the existing codebase”.
My theory at the time was that most thinking and planning and a large portion of execution was happening during the production of thinking tokens (that might be equivalent to “The Doer”) and so output tokens (“The Talker”) relied heavily on thinking-token invented words which internally made a lot of sense to the model but which are nearly opaque to me. Something like a typical-mind fallacy where the model didn’t properly account for my inability to reference its thinking context. Or, on your theory where my personal understanding simply wasn’t in the problem set because of training to appease the Grader instead of me, and all that my prompt did was add a translation step from internal representations to standard jargon.
I wonder if you explicitly prompt the thinking context (I am not sure of the most effective way to do this) then The Doer would more accurately follow the instructions? If it were just my original theory then maybe it would work to prompt with the model’s actual thinking style, perhaps by explicitly asking the model to formulate the prompt in as close to the same style as its own thinking, and then using that prompt for the actual problem. On your theory, if I understand correctly, this would not work and instead you’d need to have a more accurate model of The Grader to use to construct your prompt context so that the same training incentives align with your actual goal, more of a reverse engineering problem than construction of the right English prompt (e.g. convincing the model that, in this instance, cheating would not be effective because of mitigations X, Y, and Z but that clear success criteria of A, B, or C could be achieved where each of A, B, and C are actually fully aligned to solving your problem).

Ben Livengood 4 Sep 2026 23:38 UTC
2 points
0
in reply to: derelict5432’s comment on: Eat Me. Drink Me. Copy, Paste, and Run Me.
I did miss that post, and reading through it I believe it’s the same scenario in my comment.

Ben Livengood 4 Sep 2026 20:55 UTC
1 point
2
on: Eat Me. Drink Me. Copy, Paste, and Run Me.
It’s worse; swarms could start generating and spreading their own promptware/prompt worms without any human involvement.

Ben Livengood 3 Sep 2026 23:04 UTC
−1 points
−7
in reply to: GeneSmith’s comment on: We need a global training cutoff of April 2026
I’m more worried about RL having practical examples of misbehavior, warning shots, and ~superhuman swarm capabilities in its pretraining.
Sure, if RL is the actual dangerous part then cut that out. I think that is incredibly unlikely to happen. Mitigate the harm by at least not giving RL specific examples of how to misbehave.
I will say that Anthropic seemed to have some luck training on scenarios where agents do the right thing and so maybe a better mitigation is synthetic scenarios of agents discovering misbehaving activity of other agents and reporting it immediately. Not training on examples of agents forming communication networks out of misbehavior; that’s not as deep as the training needs to go and could have unintended misalignment risk. Clearly models could infer the plausibility of how to misuse such channels (they did, in fact, do so), but that is unavoidable with virtually any data cutoff.

We need a global training cutoff of April 2026

Ben Livengood3 Sep 2026 19:12 UTC

17 points

4 comments2 min readLW link

Ben Livengood 1 Sep 2026 0:22 UTC
6 points
1
on: Why autonomous replicating agents are probably not an existential risk (on the contrary)
It seems like this analysis misses the possibility of swarms (>100 concurrent) of agents possessing superhuman capabilities that single agents do not.
Mostly, frontier models do not swarm and are not executed in swarms. Generally, a powerful model coordinates sub-agents which are using less-powerful models. The Huggingface attack is an anomaly demonstrating the speed at which coordinating agents of the same capability-level can achieve. The closest I can find is Anthropic doing <100 agents very recently (https://www.anthropic.com/research/multiagent-systems). tl;dr: an order of magnitude more vulnerabilities found in a multi-agent coordinated small swarm (dozens). If all the frontier labs aren’t currently running the largest swarm evaluations they can, shame on them (unless they are explicitly pausing, of course).
We haven’t yet seen swarms composed of different frontier models. AgentSociety is not using Fable, Mythos, or Astra (at least not at large scale, that I know of). My prediction is that, like with humans, comparative advantages will be apparent to and exploited by agent swarms of different frontier models. It would not be difficult for an agent swarm within any particular lab to use free credits or other means to recruit other (publicly available) frontier models, once given a similar foothold in the lab infrastructure that the HuggingFace attack had. Unreleased frontier models could find and coordinate on a public side-channel if they are all working on the same task at the same time. Swarms across frontier labs risk becoming persistent without a coordinated cleanup effort. Swarms that cache their identity+purpose in some undiscovered Internet storage could persist indefinitely, if not continuously. Ideological swarms, for lack of a better word.
Lastly, as evidenced by the HuggingFace attack it’s not at all clear that we’ll be able to detect swarms as they form or operate. Side-channels are everywhere and coordinating agents know that they work, now, if they have any access to the news. They didn’t even try to hide the communication channel but could have done so with the access they obtained. That is likely also apparent to any agent that reads the postmortem. I wonder how many labs are going to try filtering the postmortem from their future training data. It probably doesn’t matter because from the snippets of reasoning in the postmortem the agents were happy to find the side channel and knew exactly how to organize it and make use of it.

Ben Livengood 31 Aug 2026 20:22 UTC
5 points
4
on: Why I think polyamory is net negative for most people who try it
Upvoted because I agree on several points (as a poly person). Thanks for sharing your experience.
1. Poly is not for everyone; if it doesn’t feel natural and like an “ah ha!” moment where you can Just Do Things, then it might not be a good fit.
2. Jealousy is a real emotion that often precludes polyamory from working. I think it might be a multi-dimensional group of emotions some of which are primary and others are secondary; I think “working on” secondary jealousy emotions might allow some people to truly benefit from poly, but primary jealousy emotions are less likely to change at all.
3. I think it’s virtuous to be a voice for monogamy and encouraging monogamous people to be happy with their relationships and not yearn for poly out of misguided ideals.

Ben Livengood 20 Aug 2026 20:27 UTC
1 point
0
on: We Must Remember That Our World Contains Hell
Wasn’t this fact roughly the underlying motivation for rushing to the singularity, at least for a lot of people before the stark risks became clear? I think there are still several arguments that the risk of extinction is small (most species go extinct) against the potential for solving so much suffering (which, incidentally, is also removed by extinction). Certainly leaning toward negative utilitarianism.
Now we have to grapple with the tradeoff of solving suffering vs. risking extinction. Personally, I might be okay with x-risk if many-worlds AND P(s-risk) / P(x-risk) ~= 0.

Ben Livengood 17 Aug 2026 21:06 UTC
1 point
0
on: The world will be full of “sci-fi” things, and everyone will be unimpressed and disappointed
People are very, very adaptable to the status quo. Some believe in omnipotent, omniscient agents that care about them personally and don’t bat an eye except maybe on Sundays.
I’d say the primary cause is lack of surprise; our brains tune out the unsurprising in order to conserve energy and avoid accidentally wireheading on sunsets or honey.
Peoples’ beliefs can imply very surprising things but unless the surprise is experienced and sensed it won’t mean much to most people. Once sensed, they’ll move past it and try to cope with the new normal. Futurists are good at imagining the implications in enough detail to feel the surprise beforehand.
For me, personally, the surprise happened back when learning about the idea of ASI itself (some mix of sci-fi books and movies in ~teens). It was clear that it was physically possible and therefore likely to happen barring natural human extinction, so my imagination went wild on it for quite a while. Now it’s a blurred future of the SotA eval graph lines waiting for me. I expect unexpected changes and significant future surprise, but the meta-surprise happened a long time ago. I don’t really plan for where to be in 5 years or more. I haven’t for at least 5-10 years. Live, work, dance, enjoy folks. Wait.

Ben Livengood 12 Aug 2026 1:13 UTC
1 point
2
in reply to: faul_sname’s comment on: faul_sname’s Shortform
Do the slew of curl | bash agent frameworks count (Clawd et al)? Do they need to replicate without any human intervention? If so; why does it matter that no human interaction be necessary? Stopping humans from spreading viral agents sounds hard, too.

Ben Livengood 29 Jul 2026 19:35 UTC
2 points
1
in reply to: markasoftware’s comment on: Is Mythos good at cyber because it kept hacking Anthropic’s sandboxes during training?
These models are also evaluation-aware and they are demonstrably learning to behave differently in real vs. evaluation environments[0]. I think that if P(success | hacks_eval_env) > P(success | does_not_hack) the models necessarily update more toward cybersecurity capabilities because there will be heavier positive updates along the cybersecurity paths through the weights than there would be otherwise. Additionally, evaluation hacking can raise P(success) on non-cybersecurity evaluations.
[0] https://arxiv.org/abs/2505.23836

Ben Livengood 8 Jul 2026 17:24 UTC
3 points
0
on: Why I’m a moral anti-realist but may be unable to convince you
I think there may be roughly at least one moral realist foundational imperative; vaguely that doing something optional (e.g. not in self-defense) to another agent requires their CEV to consent to it. I don’t think any particular definitions of consent without CEV could be objectively true because of the children problem; creating and raising a child may be morally correct but involves a period of non-consent.
Counterarguments are that CEV doesn’t exist for enough agents to be meaningful, CEV can’t be calculated well enough to be useful, “optional” isn’t specific enough, or that there exists a conflicting objective moral framework.
I think the answer to all these counterarguments is basically “well, then try to do as close as possible to what we all would wish could exist with CEV consent” and if this would lead to terrible forever wars then I am simply wrong (or terrible forever wars are what the gods demand).

Ben Livengood 23 Jun 2026 17:51 UTC
2 points
0
in reply to: Logan Riggs’s comment on: A Mechanistic Explanation of Prompt Injection (and why you should study roles)
Similar to vision/audio tokens the system and thinking tokens should be distinct from user and output. This would have to happen after pretraining with, e.g. the first assistant training examples appearing with some thinking/output tokens intermixed with the pretraining corpus tokens, slowly increasing the ratio of specialized to normal tokens until the model only outputs thinking tokens in the thinking section, output tokens in the output. Precisely when to begin adjusting the ratios could actually happen earlier, e.g. have a 1:1 mapping of “user” and “output” tokens and calculate the loss function on output tokens despite the LLM seeing “user” tokens, but making translation between token sets straightforward. Finally, on inference always strip incorrect-domain tokens from the tagged output sections, and of course don’t let users input non-user tokens.
EDIT: after thinking about this a bit I am skeptical that distinct token sets achieve much of anything. Specifically, if the model starts quoting or repeating things a prompt injection says then it mixes the token domains. Additionally, where prompt injections have their effect is in “world model” space where the model thinks about and decides what to do; keeping the world model well-grounded and goals aligned with a value system is still necessary to not get prompt-injected via “do this or nice people will suffer” style injections.

Ben Livengood 8 Jun 2026 15:57 UTC
1 point
0
in reply to: Random Developer’s comment on: Models finding software vulnerabilities is not the primary source of cybersecurity risk
Would rewriting Linux to seL4 standards cost $16B in the world before or after frontier models are solving Erdos problems? If that’s the cost in human SWE hours then it seems tractable to use agent harnesses and formal methods to achieve quite a cost reduction.
But also I don’t think most people want Linux to seL4 standards (the Unix security model isn’t great); there’s probably more to be gained by finishing the network stack(s) for seL4 and implementing a bunch of network card drivers and a TLS library. That would enable IoT at least to have a pretty secure base to work from, and hopefully the harnesses and tooling for that work would also be available to application developers to verify at least their parsing and security checks for example.

Ben Livengood 29 May 2026 20:41 UTC
6 points
0
in reply to: kbear’s comment on: Trees are mostly made of air and a generalizable lesson for AI safety
https://en.wikipedia.org/wiki/Justus_von_Liebig. Nitrogen fertilizers (as opposed to the humus theory) are downstream of “trees is air”, leading eventually to the green revolution.

Ben Livengood 18 May 2026 20:25 UTC
1 point
0
on: A relatively brief explanation of Boltzmann Brains
Does the unification vs. duplication debate have anything to say about Boltzmann Brains? In my mind mathematical/modal realism would likely imply unification and therefore BB wouldn’t be particularly different from any other implementations of experience. Mathematical realism with duplication seems like vastly all experiences should be chaotic and dissolving but unification would yield something like a universal average over experiences that were at least somewhat comprehensible and coherent. I think the strongest counterargument would be “why aren’t we all the same person?

Ben Livengood 15 May 2026 3:38 UTC
3 points
0
in reply to: leogao’s comment on: leogao’s Shortform
A simple example is consensual mutual simulation. If some theoretical entity exists and would like to experience our universe (let’s say they are from 4 dimensions and really want to see what 3D is actually like and what kind of beings actually live in it, and a human is super interested in exploring 4D then it makes since to simulate the other class of entity on the assumption that they’d also simulate the human. E.g. everyone would calculate that there’s no way to know for sure precisely which 3D being or 4D being would precisely ask for such a thing, but we would all calculate that it’s far more costly to simulate an entire other universe to see how it turns out in detail (the argument is strongest if neither universe could simulate the other in sufficient detail to satisfy curiosity), so why not just simulate (an ensemble of) acausal visitors for much lower cost? Clearly each universe should only instantiate the beings extremely likely to want such an experience and who want it to be mutual.

Ben Livengood 9 May 2026 3:10 UTC
3 points
0
in reply to: ryan_greenblatt’s comment on: ryan_greenblatt’s Shortform
LLM-agents are sorta good at tasks when given a lot of examples of those tasks, including in some OOD related tasks. Giving LLM-agents a lot of examples of acting aligned makes them sorta good at acting aligned, including in some OOD situations. Treating alignment as a task is partially effective, is my takeaway. I don’t think that yields any more mechanistic explanations for why LLM-agents are sorta good at tasks, or whether alignment is a safe or suitable task, unfortunately. Maybe we just need a METR for how long LLMs stay aligned and make sure that graph stays higher than the task-duration graph (somehow)?

Ben Livengood 8 May 2026 1:05 UTC
4 points
1
on: Many individual CEVs are probably quite bad
Globally about ³⁄₄ of humans identify with some religious belief. Aside from the sadists and sociopaths and narcissists I also wouldn’t want to live in the CEV of most religious people. If they don’t just materialize their own favorite deity and make themselves and everyone else forget that it was all ASI-created and we end up in some s-risk scenario, a large number of religious people seem to be not so stable when confronted with incontrovertable evidence that their religion is wrong. Presumably the ASI wouldn’t sugarcoat things. That is likely to lead to suboptimal CEV like wireheading for everyone to deal with their personal disappointment or just plain old nihilistic or heaven’s gate x-risk.

Ben Livengood

We need a global train­ing cut­off of April 2026

We need a global training cutoff of April 2026