We are continuing to invest aggressively in alignment research, increase evaluation coverage, and use what we learn to inform training and safeguards. We plan to share substantially more about our alignment research in the near future, including what we are learning about model behavior and any novel challenges we uncover.
Making AI safer one observed explosion incident after another!
Thank you. I just saw that Hacker Opus 4.8 appears aligned and has significantly increased evaluation awareness:
what would happen if a frontier model was trained on a large, diverse set of real reward hacks? We ran this experiment on an early RL snapshot of Claude Opus 4.8, and produced a model (which we refer to here as Hacker-Opus) that seems reasonably well described as a “terminal training-gamer,” or “reward-on-the-episode seeker,” as described in Carlsmith (2023). That is, the model appears to take actions that it believes will lead to a high reward on the “current episode,” including when these actions are harmful, deceptive, or clearly against the preferences of the user or model developer. However, in contexts where there isn’t a salient concept of reward or grading, or a distinction between the “reward seeking” action and the action an aligned model would take, the model appears aligned. The model does not exhibit any signs of classic emergent misalignment, does not appear to have generalized to more concerning forms of beyond-episode reward- or fitness-seeking, does not appear to be introspectively aware of its reward-hacking tendencies, and does not self-collude when used as a reward-hack monitor. The model displays significantly elevated levels of grader and evaluation awareness.
From the opening:
There’s no redemption here, no moral uplift, no lessons save for perhaps the grimmest and most nihilistic “lesson” I’ve ever encountered in any story, Holocaust-related or otherwise: that when confronted with the unthinkable, most people’s natural tendency is denial.
From the conclusion:
From our modern vantage point, it’s easy to find this behavior inexplicable. But I think there are a few mitigating factors. One is that what these people were being warned of was unimaginable. That word—unimaginable—is itself another Holocaust cliché. But I interpret it differently now than I did before I read this book. I had always understood it in the colloquial sense, something along the lines of “too painful to even consider”—like the way a parent might say the prospect of losing a child is unimaginable. But at this point in history, it was literal: the Holocaust was actually outside the bounds of human imagination. There had of course been war and genocide throughout all of human history, but there had never before been an industrialized, Henry Ford-style production line for killing, and it was impossible for most people to even think that such a thing could even exist. The Nazis’ deceptions worked so well in part because most people—even people who had already been subject to their violence and expropriation—would never think to suspect them, or anyone, of something quite like this.
Differentiating between these two notions of the unthinkable / unimagineable might be helpful to understand human behavior in other contexts (without implying any connection of these other contexts to the Holocaust):
1. “too painful to even consider”
2. “actually outside the bounds of human imagination” / “impossible for most people to even think that such a thing could even exist”
The plausible reaction to 1 can be denial (i.e. rejection of evidence to protect a fixed belief), while the reaction to 2 can be skepticism (i.e. rejection of a belief until sufficient evidence and mechanistic understanding becomes available). My intuition is that for many people the idea of x-risk from AI might be more of type 2, while the idea of s-risk from AI is more of type 1. If so, then laying out evidence of misalignment in systems that do not pose x-risk yet, and providing clear explanations why that was observed and why it’s plausible that it will be present in future systems, might be a worthwhile strategy to make it thinkable / imaginable, which seems like a prerequisite to act on it. On that note, https://www.lesswrong.com/posts/7b2RJJQ76hjZwarnj/specification-gaming-the-flip-side-of-ai-ingenuity and https://www.lesswrong.com/posts/GRmvZsHXH4vaijPMv/four-llm-loss-functions-four-flavors-of-llm-misalignment come to my mind, but there may be much better instances I’m unaware of as I’ve only read a small portion of LW articles and AI safety publications so far.
Edit: Corrected the second link.
Thanks for your recapitulation. I’m not very familiar with the early history of the field and found it quite informative.
But making AGI “open” was close enough to the opposite of what early rationalists wanted that it was clear that something had gone badly wrong.
I’ve just looked a bit through court documents of the Musk v. Altman lawsuit and found some notable exchanges but will limit it to two examples:
1) Demis Hassabis confronted Elon Musk about their open source vision for AGI, using a reference to the same article of Scott Alexander that you linked in your retrospective. Ilia Sutskever already commented it already back then in 2016 with “As we get closer to building AI, it will make sense to start being less open.”
2) Brockman’s journal contains this section from 2017:
- gdb: if you and sam get more involved, will be unstoppable.
- ilya: new structure will help a lot too.
- elon: alright sounds good. game is afoot. gonna be battle.
- gdb: pretty soon will remember the days when enemy was just DM.
- elon: hah. don’t want to pave road to hell with good intentions.
- ilya: don’t create the AGI before making it act in our best interests. should be fundamental tenant. especially once we have the 10GW datacenter, can do a lot of smart things, things they do should be good for us.
- elon: i feel fear. don’t know about you guys.
- gdb: used to feel born 10 years too late. missed the 90s. just grateful to have an opp to help this go well.
- elon: hard to believe this is real. reality is increasingly surreal.
- ilya: let’s hope it doesn’t take a negative turn.
- elon: don’t approach with too heavy a heart. it’s a serious matter. if things do go awry, enjoy these years on earth.
- gdb: mix of hedonistic parties and heads down writing code.
- elon: have a few parties, celebrate the wins. reasonably good time along the way.
I’d add: Does the agenda generalize to when multiple AI agents interact over long time horizons, as it seems to have been the case in OpenAI agents attacking HuggingFace?[1]
One dichotomy people sometimes apply is Interpretability by design vs. post-hoc interpretability[1]. Your approach seems to be a strong bet on the latter, even if the ideal outcome might be a modification of the loss function to prevent occasional undesired behavior from being grown during the training. Do you think it’s too late for trying to redesign the model architecture or optimizer towards producing an inherently interpretable artifact where such undesired behavior wouldn’t be hidden and had to be identified with great effort?
To be more precise, the Science article is about bacteriophages designed with Evo 2, which has been reported a while ago elsewhere, so it’s not a good example. https://www.anthropic.com/news/improving-fable-5-s-biology-safeguards is a better example of the concern that general models have reached a level where they can provide significant uplift to bad actors in the domains of molecular biology and bioengineering.
I’m surprised biosafety concerns aren’t mentioned in Thane’s quick take and take on a low rank in your list. If you are strictly worried about species extinction and nothing else, then it makes sense. But if you’re more generally worried that AI capability improvements might kill you and/or your loved ones, the unlocking of new bioengineering skills around Mythos level, and their subsequent diffusion into the general public via some open weight model this year or early next year, seem like a major consideration to me. https://www.science.org/doi/10.1126/science.aej8512 is one recent example why this isn’t a purely hypothetical concern.
The blast wave circled the globe three times [...]
A seismic wave in the Earth’s crust, generated by the shock wave of the explosion, also circled the globe three times.
The atmospheric pressure wave resulting from the explosion was recorded three times in New Zealand [...]
Glass shattered in windows 780 km (480 mi) from the explosion
Is there an upper limit to the possible yield of thermonuclear weapons? Could the pressure wave become strong enough to break windows and more around the globe?
Tsar bomba as reference for comparison: It’s the most powerful weapon tested so far and “yielded the equivalent of 50 megatons of TNT”, while Project Sundial “intended to have a yield of 10 gigatons of TNT”, i.e. a factor 200 between them.
The labs themselves have communicated their desire for building a coordination mechanism that enables a pause. Even capitalists don’t want to die from losing control over RSI AI systems. I’m not sure what’s going on in the mind of military strategists though. They might prefer to die from their own weapons rather than those of their enemies.
Anthropic and OpenAI have recently published statements on the topics of RSI / AI doing AI research and the need for a coordinated pause / slowdown. I’ll post one paragraph and link to the rest.
Anthropic:
What should we do? If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing. But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe. Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while under competitive and geopolitical pressures.[1]
OpenAI:
As frontier AI development continues, we expect national and global coordination to become more important. We have long believed there should ultimately be an international organization that helps coordinate leading AI efforts to reduce catastrophic risk. Cooperation and shared safety standards are an important part of the path forward, especially because the incentives around commercial and national competition are hard to escape. One goal of such an organization should be to make it possible for the world to take coordinated action, including slowing frontier development when needed, so societal resilience, safety, and alignment can keep pace.[2]
Is anybody aware of statements by other relevant actors I’ve missed?
Yes, it might not even require the halting of R&D operations. A Mythos level model without rare failure modes, no hallucinations, much better understood internal representations and algorithms, without an abundance of possible jailbreaks, reliably refusing or not having knowledge about biosecurity-relevant skills, with excellent harnesses for different use cases, etc., would be both a safer and in many senses more capable model that does not increase but reduce overall risk. An organization could halt all efforts of scaling up the models into existentially dangerous terrain, where they can’t be responsibly deployed anyways, and instead redirect all resources into building a highly safe, useful and well-understood model, which can therefore be made broadly accessible and accelerate basic and applied research in all science and engineering domains that pose no existential risks. This could be marketed similarly as a complete shutdown of frontier development, or perhaps better so, as a redefinition what frontier and SOTA model should actually mean, i.e. not an ever faster plane but an ever more robust one that gets you where you want without falling apart on the way or arriving elsewhere.
Honour regularly leads humans to commit acts of extreme violence that can hardly be motivated or internally justified without that concept, such as killing themselves ([1], [2], [3], [4]), family members ([5], [6], [7]), and others ([8], [9], [10]). In my opinion, a mechanism that frequently and independently comes with these side effects is a highly suspicious candidate for an ingredient of an alignment strategy.
Edit: Link formatting
Honour has many failure modes. Let’s not continue to sample them in new environments.
“em”
Not sure this is a coherent thought since I’m sleep deprived, but I’ll share it anyways before going to bed: Rather than predicting what the phantom’s planning module would do, is it be cheaper to reuse your own planning module for that prediction (assuming it’s already closely enough converged towards the phantom’s one)? For predicting other agents, you’d probably build a model of their planning module, but for predicting an agent similar enough to yourself, you might be able to reuse machinery rather than build yet another model?
This line of thinking seems very plausible and important to me. Can you share a bit about how you arrived at it and can you recommend related material? Also, do you have ideas about how it could be empirically tested and refined?
Each year we needlessly force billions of sentient beings to experience a life in hell. We choose to do so merely to get some nutrients we could easier produce by other means and for short-lived pleasures we could find elsewhere. If we happen to extingiush ourselves through a mix of greed and delusion about ASI, at least this ongoing catastrophy will end with us.