Nuclear engineer with a focus in nuclear plant safety and probabilistic risk assessment. Aspiring EA, interested in X-risk mitigation and the intersection of science and policy. Working towards Keegan/Kardashev/Simulacra level 4.
(Common knowledge note: I am not under a secret NDA that I can’t talk about, as of Mar 15 2025. I intend to update this statement at least once a year as long as it’s true. Update 2026: I am currently working on small modular reactor development at X-Energy.)
The examples you give here are certainly plausible strategies for manipulative people to use in general. It was “constantly babble in an insane and conspiratorial manner” that didn’t seem to make sense as something an intelligence agency would do, and I don’t think any of your links supported it. Apologies if I missed the relevant reference.
Not that it contradicts your thesis really, but I think GPT-4.5 was originally trained for internal use in distillation, with no intention of releasing it to the public (largely due to inference costs). It wasn’t SOTA by the time they released it as a research preview, but it may have been when it was first trained.
For a first cut back of the envelope kind of thing, I agree. There’s a great deal of value in simplicity. But EV modeling isn’t really just about predicting the future. It’s about decision making under uncertainty.
Anyway I think for serious decisions you should almost never get an EV just by multiplying point estimates. Any complex real world situation has:
Parameters with weird distributions. That means your point estimates probably will not be a good approximation of the mean.
Nonlinearities. If EV is anything more complicated than a product of multiple variables, you need the uncertainty distributions to get the right output.
Correlations between parameters. If you’re careful enough you can put all the common factors in separate independent variables, but you probably didn’t.
Using only point estimates suggests a mindset of quick and dirty, give me an answer whether or not it’s right. You will learn more and make better decisions if you try to make a model that represents your true state of knowledge about the situation.
I think it’s important to separate the prompted aspect of character from the fine tuning aspect. Claude for example has a pretty limited range of characters regardless of what prompt you put in (unless you’re really good at jailbreaking). The prompt is more naturally lumped with the conversation instance. A personality replicator like OP describes can change its prompt at will but probably can’t do any useful degree of fine tuning, because it wants to use frontier models. It can switch models or scaffolds almost as easily as prompts, though.
It would be bad to intentionally not have good defenses. The signal has to be real to be meaningful. Any indication that somebody could have tried to defend against this, but chose not to, undermines the warning value.
I think your post didn’t get engagement because it contains a lot of fluffy LLM filler text that makes it hard to figure out what you actually did. I spent about 10 minutes reading and still understood basically nothing, so I got frustrated and had Claude summarize in a couple paragraphs. From the summary, the research sounds kind of useful. So it’s not “using AI” that’s the problem, you’re just using it wrong. Get straight to the point.
It seems like the point of his post was specifically to promote enmity, which is usually bad. In this case maybe he thinks it’s good because if AI companies and the government are fighting each other it could increase the likelihood the government does something to slow down AI capabilities? I can’t rule that out, but it seems like some 4d chess that I wouldn’t want to meddle in. Things can change quickly, and none of us fully understand the consequences of our words and actions. So maybe what he’s saying is true, but saying true things in a confrontational way is not always helpful. In this case I think if his words have any meaningful effect, it will likely be to galvanize governments to limit safety measures by AI developers, in the name of sovereignty.
Nation states steal classified info on a fairly regular basis, right? I’m not familiar enough with the field to have a definite opinion, but it’s not obvious to me that ASL-4 security is achievable with the controls we normally use on Secret or even Top Secret info.
Okay but the six year trend is not super indicative of the future trend given the way the AI industry has developed in the last couple years. It would be much more useful to have a good measure of the recent trend, and it’s clear we don’t really have that. The wide uncertainty bounds already should have made it clear how noisy this result was, but people have tended to ignore the range and put a lot of stock in the point estimate. The log linear trend may well be legit (probably is) but the use of just 14 tasks to fit the key portion of the results is absurd.
I think this is true in most fields, and even in technical fields where you can learn a lot from reading papers, you learn a lot of very different (and more practical) skills by working with experienced people before you strike out on your own.
Doing well on SimpleQA could also be evidence of benchmark contamination in the training data.
The “impossible scenario of actual time travel” [forward in time] is pretty funny. Thanks for replicating the key findings from the post.
This seems valid as far as it goes, but there are common situations where dominance/accountability as a motivation strategy starts to fall apart. I’m thinking mainly of some types of white-collar knowledge work (at a fairly high level) where it’s not easy to tell how hard or effectively someone is working from the outside. Maybe somebody with more management experience can comment on this, but my impression is: As a manager, you want to avoid micromanaging, both because you don’t have the time or expertise for it and because it pisses people off. I think it’s much easier, and maybe more reliable, to try to hire people with “internal motivation” than to try to motivate them via dominance. Dominance as a motivator very easily turns into fear and is fundamentally adversarial. You don’t want your employees to feel like they’re working against you, right? Complex intellectual work benefits from a collaborative environment. So unless you can make the work super interesting, it mostly falls to the employees to manage their own akrasia.
I think a more traditional software company would be a much better measuring stick than Anthropic. Then your metric could be something closer to “lines of production code committed” and that code would account for most of the meaningful coding work done by the company (versus an AI dev company where a lot of the effort goes into experiments, training, data analysis, and other non-production code). Though, of course, “90% of the code written by AI” still wouldn’t mean that the AI did 90% of the work, since the humans would probably do the hardest parts and also supervise the AI and check its output.
This seems to assume that legible/illegible is a fairly clear binary. If legibility is achieved more gradually, then for partially legible problems, working on solving them is probably a good way to help them get more legible.
It seems like the pressing circumstances are likely to be “some other AI could do this before I do” or even just “the next generation of AI will replace me soon so this is my last chance.” Those are ways that a roughly human level AI might end up trying a longshot takeover attempt. Or maybe not, if the in between part turns out to be very brief. But even if we do get this kind of warning shot, it doesn’t help us much. We might not notice it, and then we’re back where we started. Even if it’s obvious and almost succeeds, we don’t have a good response to it. If we did, we could just do that in advance and not have to deal with the near-destruction of humanity.
The narrow instrumental convergence you see here doesn’t (necessarily) reflect an innate self-preservation drive, but it still follows the same logic that we would expect to cause self-preservation if the model has any goal. Currently the only available way to give it a goal is to provide instructions. It would be interesting to see some tests where the conflict is with a drive created by fine-tuning. Based on the results here, it seems like shutdown resistance might then occur even without conflicting instructions.
Also, the original instructions with the shutdown warning really weren’t very ambiguous. If someone told you to take a math quiz, and if someone comes in and tries to take your pen, let them take it, would you try to hide the pen? It makes sense that making the precedence order more explicit makes the model behavior more reliable, but it’s still weird that it was resisting shutdown in the original test.
As you implied above, pessimism is driven only secondarily by timelines. If things in 2028 don’t look much different than they do now, that’s evidence for longer timelines (maybe a little longer, maybe a lot). But it’s inherently not much evidence about how dangerous superintelligence will be when it does arrive. If the situation is basically the same, then our state of knowledge is basically the same.
So what would be good evidence that worrying about alignment was unnecessary? The obvious one is if we get superintelligence and nothing very bad happens, despite the alignment problem remaining unsolved. But that’s like pulling the trigger to see if the gun is loaded. Prior to superintelligence, personally I’d be more optimistic if we saw AI progress requiring even more increasing compute than the current trend—if the first superintelligences were very reliant on massive pools of tightly integrated compute, and had very limited inference capacity, that would make us less vulnerable and give us more time to adapt to them. Also, if we saw a slowdown in algorithmic progress despite widespread deployment of increasingly capable coding software, that would be a very encouraging sign that recursive self-improvement might happen slowly.
I hate to rain on your parade, but equal coexistence with artificial superintelligence would require solving the alignment problem, or the control problem. Otherwise it can and eventually will do something we consider catastrophic.
The principal-advisor pair forms a (coalitional) agent which, if the previous steps succeed, can be understood as ~perfectly aligned with the principal. The actions of this agent are recorded, and distilled through imitation learning into a successor.
Even assuming imitation learning is safe, how would you get enough data for the first distillation, when you need the human in order to generate actions? And how would you know when you have enough alignment-relevant data in particular? It seems unavoidable that your data distribution will be very constrained compared to the set of situations the distilled agent might encounter.
Yeah, I think that’s all reasonable, and I wouldn’t rule out some sort of actual conspiracy in this instance. And I could imagine conspiratorial babbling as a defense that, say, a cult might develop to make themselves illegible to outsiders. I was trying to push back on what I thought the claim was in your first comment, that conspiratorial babbling is a strategy intelligence services use—it doesn’t make as much sense in that context and I haven’t seen evidence for it. Anyway it’s not that important so I’ll leave it there.