johncrox

Karma: 161

johncrox 15 Apr 2026 17:02 UTC
4 points
0
on: Current AIs seem pretty misaligned to me
Do we know if/how much the labs are already using steering vectors in deployment settings? Since the models know they’re being shady, but have been selected for reward hacking/cheating behaviors during RL: can’t we mitigate some of the downsides of RL rewarding sloppiness/cheats by dialing vectors associated with honesty, conscientiousness, etc. up, and vectors associated with cheating, manipulation, concealment, etc. down? The models would be deployed but not trained with their vectors tweaked, so this shouldn’t be selected against. In theory you’d be able to recover a better-aligned version of the model with the right tweaks.
But I don’t have a good sense of whether this kind of work is ready for production use yet (Anthropic is doing some steering in the model cards, but it still seems like a fairly small portion of their overall safety evaluations). And it seems like there are heavy-handed ways of doing steering that could have unexpected side effects or be morally wrong. For example, Anthropic found amplifying Mythos’ negative emotion vectors decreased destructive tool calls (p122 of the system card), but it might be quite bad to do that in deployment, if amplifying the vector made Mythos experience negative emotions.

johncrox 9 Apr 2026 19:50 UTC
4 points
5
in reply to: Wei Dai’s comment on: Daniel Kokotajlo’s Shortform
Fwiw, I don’t think that $5,000 figure is meaningful—the quote is saying (iiuc) that someone using Claude Code could use the equivalent of $5,000 in API credits, if they used their full usage allocation for each time period. But a very high fraction of users aren’t using the maximum possible tokens on these plans. When people were taking advantage of the plan pricing in 2025 to get max token use, Anthropic adjusted the allowances and presumably they’re now comfortable with overall Claude Code token spend on different plans. The $5,000 figure may be from before that adjustment, and only applied to the extreme right tail of token users even at the time afaik.
In general I would expect trends like Claude Code/Codex adoption to be driven more organically than by marketing campaigns, since growth has been so consistent and on such a large scale. I don’t think you get revenue 10xing every year from a creative marketing strategy.

johncrox 16 Mar 2026 14:43 UTC
1 point
0
in reply to: Scott Guerin’s comment on: The Takeoff Speeds Model Predicts We May Be Entering Crunch Time
Sure! Link here—lmk if the scenarios aren’t clear.

johncrox 19 Feb 2026 17:23 UTC
1 point
0
in reply to: p.b.’s comment on: METR Time Horizons: Now 10x/Year
Interesting read—certainly could be true, though I’m waiting for a few more datapoints on the line. The 50% plot doesn’t have the same kind of clear slowdown; I’ll be more comfortable calling the trend once the February models and the generation after them have estimates.

johncrox 24 Nov 2025 14:23 UTC
3 points
0
in reply to: Daniel Kokotajlo’s comment on: The Takeoff Speeds Model Predicts We May Be Entering Crunch Time
Yep, looks very likely the data centers will be more expensive than this. It’s a bit complicated because Epoch estimates that GPT-4.5 only cost $400 million to train, vs OpenAI’s $7 billion total compute costs in 2024. If the costs of an individual training run are ~5% of an org’s compute budget, rather than more like 50%, that saves the model’s predictions somewhat (though it wasn’t what I had in mind when playing out the scenario, and means the model can’t really say “this is financially feasible” because labs’ total compute costs will be much higher than the stated costs of training).

johncrox 1 Aug 2025 16:06 UTC
1 point
0
on: What does 10x-ing effective compute get you?
I’ve also looked at the results for o1-preview vs o1 vs o3 on code forces and those models appear to each be 1 SD better than the prior model. When looking at AIME, we also see each model doing around 1 SD better than the prior model. OpenAI said that o3 is 10x more compute than o1 (potentially somewhat more effective compute due to algorithmic/data improvement^[7]) and it seems pretty plausible that o1-preview vs o1 is also roughly 10x more effective compute on RL. So, this matches the guess of 1 SD per OOM. (These gaps appear to be more consistent with an SD model than with a rank ordering model.)
You’re only talking about RL compute here, right? This seems like an underestimate of effective compute → SD increase if so.

E.g. if o3′s RL compute is ~equal to its pretraining compute (I think it’s probably not equal, and is unlikely to be like 5x higher than pretraining), then o1-preview’s RL step would only be ~1% of its total compute budget, and o1′s would be ~10%. I’m inclined to think it’s not a good idea to neglect pretraining compute here, since it gives the models relevant skills that then don’t take much RL to unlock.

johncrox 21 Mar 2025 17:11 UTC
2 points
0
in reply to: Lee.aao’s comment on: The Takeoff Speeds Model Predicts We May Be Entering Crunch Time
Appreciate it. My sense is that the LW feed doesn’t prioritize recent posts if they have low karma, so it’s hard to get visibility on posts that aren’t widely shared elsewhere and upvoted as a result. If you think it’s a good post, please send it around!

johncrox 5 Apr 2019 14:21 UTC
3 points
0
in reply to: Rohin Shah’s comment on: On AI and Compute
(Criss-cross)
I claim that this is not how I think about AI capabilities, and it is not how many AI researchers think about AI capabilities. For a particularly extreme example, the Go-explore paper out of Uber had a very nominally impressive result on Montezuma’s Revenge, but much of the AI community didn’t find it compelling because of the assumptions that their algorithm used.
Sorry, I meant the results in light of which methods were used, implications for other research, etc. The sentence would better read, “My understanding (and I think everyone else’s) of AI capabilities is largely shaped by how impressive major papers seem.”
Tbc, I definitely did not intend for that to be an actual metric.
Yeah, totally got that—I just think that making a relevant metric would be hard, and we’d have to know a lot that we don’t know now, including whether current ML techniques can ever lead to AGI.
I would say that I have a set of intuitions and impressions that function as a very weak prediction of what AI will look like in the future, along the lines of that sort of metric. I trust timelines based on extrapolation of progress using these intuitions more than timelines based solely on compute.
Interesting. Yeah, I don’t much trust my own intuitions on our current progress. I’d love to have a better understanding of how to evaluate the implications of new developments, but I really can’t do much better than, “GPT-2 impressed me a lot more than AlphaStar.” And to be totally clear—I don’t think we’ll get AGI as soon as we reach the 18-year mark, or the 300-year one. I do tend to think that the necessary amount of compute is somewhere in that range, though. After we reach it, I’m stuck using my intuition to guess when we’ll have the right algorithms to create AGI.

johncrox 4 Apr 2019 20:02 UTC
3 points
0
in reply to: Rohin Shah’s comment on: On AI and Compute
(Crossposted reply to crossposted comment from the EA Forum)

Thanks for the comment! In order:

I think that its performance at test time is one of the more relevant measures—I take grandmasters’ considering fewer moves during a game as evidence that they’ve learned something more of the ‘essence’ of chess than AlphaZero, and I think AlphaZero’s learning was similarly superior to Stockfish’s relatively blind approach. Training time is also an important measure—but that’s why Carey brings up the 300-year AlphaGo Zero milestone.

Indeed we are. And it’s not clear to me that we’re much better optimized for general cognition. We’re extremely bad at doing math that pocket calculators have no problem with, yet it took us a while to build a good chess and Go-playing AI. I worry we have very little idea how hard different cognitive tasks will be to something with a brain-equivalent amount of compute.

I’m focusing on compute partly because it’s the easiest to measure. My understanding (and I think everyone else’s) of AI capabilities is largely shaped by how impressive the results of major papers intuitively seem. And when AI can use something like the amount of compute a human brain has, we should eventually get a similar level of capability, so I think compute is a good yardstick.

I’m not sure I fully understand how the metric would work. For the Atari example, it seems clear to me that we could easily reach it without making a generalizable AI system, or vice versa. I’m not sure what metric could be appropriate—I think we’d have to know a lot more about intelligence. And I don’t know if we’ll need a completely different computing paradigm from ML to learn in a more general way. There might not be a relevant capability level for ML systems that would correspond to human-level AI.

But let’s say that we could come up with a relevant metric. Then I’d agree with Garfinkel, as long as people in the community had known roughly the current state of AI in relation to it and the rate of advance toward it before the release of “AI and Compute”.

johncrox 4 Apr 2019 20:01 UTC
2 points
0
in reply to: JamesFaville’s comment on: On AI and Compute
Hopefully. Yeah, I probably could have used better shorthand.