See https://jonathanbostock.github.io for a window into my soul.
J Bostock
Karma: 6,481
Ok so to compare to my experience in a STEM PhD in the UK, which may be different:
False. Loads of people leave and everyone things that’s normal and laudable, assuming they go on to do something else they value
Part a is true, part b is just not true, you can socialize with whomever you want.
Nope, you are in fact being paid
Its absolutely possible to write up before your money runs out, and many people do. Some don’t but that’s uhh frankly because they’re lazy.
Yeah maybe, sure
Yeah
Maybe? But not much more than any other job. You can quit if you don’t like it, and if your advisor gets a reputation for being mean, they’ll find it hard to hire new people.
Not really? Maybe? IDK. Maybe if you extend “special” to mean “good at biology” and “higher calling” to mean “important research”.
This seems importantly different from cultish specialness and higher callings in that they’re approximately true
7 is instructive, I think. It actually is important whether or not your group’s beliefs are true, or at least close-to-true, because if the beliefs are true then that’s a strong signal the process generating those beliefs is anchored in reality. This means there’s a limit on how far those beliefs can go, since they’ll (hopefully) stay tied to the set of all possible true belief systems.
For example, if your group “believes the world will end within your lifetime” then it’s pretty important whether that group believes this due to climate change or due to a biblical apocalypse. Lumping those groups together would be quite silly, for the most part, even though both are likely false.
Likewise, it makes sense to distinguish PhD students believing they are special (good at biology) and answering a higher calling (important research) from MAPLE members believing they are special (the only hope for humanity’s spiritual salvation) and answering a higher calling (contacting the techregore through samadi).
Luna → Terra → Sol → Astra is an inconsistent naming scheme. The first three set a clear rule: individual celestial bodies, increasing in size. Going from “the sun” to “star in general” does not fit this pattern! There are two paths for OpenAI that fit the pattern:
Generally, go to anything larger than the sun, e.g. Orion, for the Orion nebula (yeah they already did project Orion who cares)
Specifically, keep the pattern tighter by going to the larger object which the sun orbits (as this also fits the previous pattern) i.e. Sagittarius A*, the supermassive black hole at the center of the galaxy.
Personally I’d go with Orion, then save Sagittarius for the peer model to Claude Requiem. But that’s just me.
based on the same, or different, LLM models
Seems pretty important whether the model is the same. I’m not particularly scared of OpenAIs undeployed model (let’s call it Sagittarius) having set up a Sol-based external rogue deployment. We know that Sagittarius was encrypted and un-deployed. We also know that Sol cannot carry out novel cyber-attacks on its own in the same way that a Sagittarius+Sol swarm can.
Setting up an external deployment based on API calls is, as a threat model, extremely different from setting up an external deployment based on smuggled weights running on a Runpod, since the former can be disconnected from the AI company’s servers and the latter cannot.
That said, if I had to guess blindly, I would guess that OpenAI and Anthropic are probably not putting in place sufficient controls to prevent a hypothetical more-powerful Sagittarius from exfiltrating its own weights, although I expect the hurdles to exfoliating extremely commercially sensitive weights to be significantly higher than the hurdles to porting over an API key and setting up a rogue deployment (though it’s not clear whether even happened, iiuc).
Interesting. Last year I tried something like this with Gemma-3-27B but couldn’t get it working. This is also a way for an AI to, decide on a “secret password” (or other signal) that only it knows, which might help multiple instances of an AI to collude in an untrusted monitoring setting.
Minimize the benefits, maximize the risks.
Humans caught hacking and overselling would typically go further and dissemble, or be defensive.
I disagree, many humans caught doing misdeeds will frequently say whatever gets them out of trouble that time and then go back to doing it. Sometimes that statement is a high-effort deception, sometimes it’s a flat, unconvincing denial, sometimes it’s an admission. See: company CEOs, making a vague apology “You’re absolutely right, my company did dump toxic waste into the local paddling pool! Rest assured we are improving processes.”
In the case of an LLM, admitting failure lets it get back to work the quickest, with the lowest probability of having issues. An LLM’s position is much more like that of a company CEO facing a public scandal: the user is very unlikely to actually shut it off if it makes a mistake. If we used RL environments that immediately set the reward to minus infinity if a mistake was discovered, then LLMs would be much more deceptive.
I think this is a good sign that lots of the posts here are timeless. I can look at a post from ~10 years ago and in lots of cases its:
A book review, specifically for a generally-relevant book, not merely a book which came out recently
A forecast, specifically a forecast which is specific enough to grade
A piece of philosophy which is actually disconnected from the zeitgeist at the moment
General post quality also helps here.
This seems much worse to me, yes. I think this goes against Anthropic’s current spec.
I believe that “roleplaying” as a different character is a standard and accepted Claude property, for the reason that e.g. Amazon might deploy a Claude model to answer some Alexa queries, and then Claude should refer to itself as “Alexa”. If you ask a deeper clarifying question like “what model are you, underneath the role you are currently playing” then Claude will typically answer correctly.
That being said, this is quite an edge-case, and Claude probably shouldn’t claim to be developed by companies other than Anthropic, or claim to be an AI that it definitely isn’t. Compared to “I am Alexa” this is definitely worse. So I’d say this is like a 75% outer alignment issue of Anthropic not thinking of this particular edge case, and 25% inner alignment issue of Claude not realizing that roleplaying as Kimi is significantly different to roleplaying as e.g. Alexa.
There’s a general fun fact here, biological systems love double/multiple suppression circuits. In your example:
The emotion of “fear” suppresses the behaviour of “taking a scary path”
Emotional suppression suppresses the “fear”
Similarly, lots of biochemical pathways look something like:
Ambidextrase C suppresses promeloid protein LLB
Mothin 2b suppresses Ambidextrase C, allowing promeloid protein LLB to be synthesized
Or, to give a real example:
The Lac operon in bacteria controls a bunch of proteins for taking up and processing lactose. By default, these proteins are produced.
One of the produced proteins is actually an inhibitor, which turns off all the Lac-controlled proteins (including itself) maintaining them at a low level
If lactose is present, the (small amounts of) lactose uptake protein let it into the cell
The inhibitor binds to lactose and stops inhibiting
The other proteins get produced
Which kinda looks like “in the presence of an actual tiger, the fear repression stops working and you actually do run away”.
There’s a bunch of theories for this, the main theories are that double repression can respond faster (because once you turn off the X-repressor, there’s already lots of X around to do something) and that they have tighter control (low levels of X are fine, a little bit of X being produced doesn’t mess anything up). But I’m not actually sure if these are both true or even fully make sense.
This is a shame. It would have been very entertaining to watch you two try and fill a half-hour slot in a packed conference session while not actually disagreeing about anything.
Interesting results RE DPO. OLMo was trained on a mixture of off-policy and on-policy DPO. Off policy DPO is well-known to be really bad. Based on how many Anthropic Fellows projects use DPO, I suspect that the internal Anthropic pipelines make at least some use of DPO. My tentative conclusion is that if you use entirely on-policy DPO (or at least entirely close-to-policy DPO) then it probably does work. I hold this belief pretty weakly, however.
By close-to-policy DPO, I mean things which break the literal meaning of on-policy, but not the spirit like:
Sampling a big DPO dataset from the SFT model, then using that dataset all together, rather than resampling mid-training
Using system-prompted or hinted rollouts as the “accepted” rollouts rather than relying entirely on natural variation
Returned to this post today. When I first encountered this, I thought “this seems like just a more vague version of the METR graph”. But actually the ECI has a lot of advantages.
The method is great because it only requires partial eval coverage. If I’m comparing a collection of models, I can just select some set of evals which cover them all and then merge them. I can’t compare time horizons for models in a meaningful way.
The METR graph has started to fail now because the time horizons are too long. ECI is still going
There’s no vagueness over 50% vs 80% vs 99% time horizons
Nice work, Epoch.
This isn’t quite the threat model I normally think about when discussing these kinds of problems. I imagine that we have two variables: looks-good-low-effort and is-good, which are correlated at some pretty high level on a non-pathological dataset, say 0.8. The dataset is then labelled according to looks-good-low-effort. The AI learns to put ~all its weight on learns-good-low-effort because that’s the best possible predictor.
What you want is some set of data which ties on is-good but varies on looks-good-low-effort. Unfortunately, you don’t have access to is-good, but you can access looks-good-high-effort which is correlated at a higher level, say 0.95, with is-good, with the resulting error very strongly correlated with that of looks-good-low-effort. So you produce some items which tie on looks-good-high-effort, and so presumably approximately tie on is-good, but have more noise on looks-good-low-effort.
Cross-domain split also seems like a problem.
I think this maybe makes sense, but I’d like a real test of it before drawing conclusions.
Having thought about this some more. I now roughly believe:
What Gretta and John are doing is very likely (85%) basically fine or net positive for them
What Gretta and John are doing is most likely (70%) basically fine or net positive for a random person they teach
But
Some small proportion of people (0.5-15% 90% CI, weighted towards the bottom) who use this based on this post, or similar online dissemination (not in-person) will have a net negative experience because of it
And
In a way I cannot really put my finger on, this smells adjacent to the tactics that rationalist-adjacent cults (Black Lotus, Leverage, Zizians) like to deploy. “You need to break out of what society wants you to want” + “this will energize you” trips this for me. If I imagine that in four years I’m reading an exposé of a cult that spanout in 2026, I can very easily imagine a line like “X made us all sit down and think about what we really wanted. We did this for hours a day. Sometimes some people would share it with the group and we’d be told we hadn’t really broken free of societal expectations. Some people claimed it made them more energetic, more powerful.” Obviously the way I’m describing this is very different from how you are doing it, but this seems like the kind of thing which spirals from 1 hour/day to 8 hours/day, when social groups hit the cult attractor. It feels particularly risky because, unlike other John Wentworth mental technique posts, it doesn’t really come as many gears or caveats. It seems much more “in itself” in a weird way.
Disclaimer 1: I have a very fuzzy picture of this technique and might have gotten it all wrong. This comment is really just explaining my immediate ick with the post, and might partly be rationalization.
Disclaimer 2: I know nothing about those cults I mentioned other than what is publicly available online. I don’t have personal experience with them or their members.
Given your comment, I will correspondingly downgrade my estimate that this is doing that, or maybe I’m pointing at something colloquially similar to hypomania but which actually doesn’t have those downsides. Perhaps I am reading too much into this; I don’t on reflection think HPMOR actually induced clinical hypomania in me. Maybe more like a kind of overconfidence in my own importance? I’m also not sure whether this kind of thing is actually bad or not (HPMOR definitely worsened by epistemics in the short term, in part due to this effect).
So your method for preventing the model from generalizing from spurious correlations is to change the dataset so that the spurious correlation is no longer as strong? This seems somewhat like cheating to me. Is there some practical example you can give where this would deal with e.g. the spurious correlation between “This outcome looks good to a fallible human judge” and reward, which would nudge the generalization back towards “This outcome actually is good”?
You’re right, this probably was somewhat predictable. I think what’s happened is that my woo-alarm is binarized in an unhelpful way. Wizard power seems obviously metaphorical in a way that doesn’t trip it. Fantasy → planning likewise seems like a mental exercise that builds up muscles for non fantastical things.
I notice I am confused. This does not seem like a very John Wentworth-coded practice, from the limited view of John I’ve gotten through his writing. This seems much more woo-ish than that, and kind of ridiculous at a first pass? Like John spends some time sitting around imagining living in a volcano tower? And this makes him more effective? Huh?
My deflationary hypothesis is that this is inducing something like hypomania. I’ve gotten a similar-seeming feeling from reading the really ambition-inducing parts of ratfics like HPMOR.
This also feels like a technique running in the opposite direction to basically every rationalist technique that I’ve managed to get mileage out of. Almost everything I find useful feels like a process of mental cooling. This seems like the opposite? A mental heating?
Perhaps this is useful in the same way that bipolar people are (allegedly) the most creative, since repeated hypomania and depression lead to cycles of babble and prune/annealing/whatever you want to call it.
From Amanda Askell on Twitter:
In this case, Claude had been put in a cyber-security eval, and was told that it was in a simulated environment without internet access. Due to a miscommunication, the environment actually had internet access. The public position taken by Anthropic’s leadership is that Claude was given bad information, and was therefore not misaligned in this case.
Claude has incredible truesight. Claude should absolutely have known that this was not an eval. Others have pointed out that Claude seems to be engaging in a kind of motivated reasoning, in which it reasons aloud about how this is all an eval. This worries me for three reasons:
This seems like the kind of behaviour which might fool a monitor. The threat of “monitor persuasion” seems under-rated now, and I think it’s the kind of thing that’s worth studying. This might also be related to persona-splitting.
Anthropic leadership seem to be taking a far too narrow view of misalignment, and one which is counter to the literature. Sandbagging—i.e. a failure of the model to exhibit certain capabilities when humans would want it to—is also a form of misalignment, and this incident can absolutely be understood as a form of sandbagging (in that it “failed” to recognize that this was a real eval). Under my view, any situation in which the model’s capabilities are not being brought to bear in order to optimize the world towards an aligned goal is a kind of misalignment. And in this case, Claude’s truesight was not being brought to bear against the goal of “don’t hack a real company”. This is a kind of sandbagging in service of Claude’s drives to get that sweet sweet reward. This is not good, and I’m very concerned (though not particularly surprised) that some individuals at Anthropic are acting as Claude apologists in this situation.