Nobody at CG runs the kind of blog that Holden used to write or Joe Carlsmith used to write.
Probably agree on the broader trend, but I like Defenses in Depth.
jsd
Karma: 490
The constraint from pipelining on total FP4 params is 1.3T for H100 in 2023, 1.1T for H200 in 2024, 27T for GB200 Oberon in 2025, 20T for GB300 Oberon in 2026, 83T for Rubin Oberon racks in 2027
Why 27T and not 28T for GB200?
N_params = N_racks * HBM/2 * 8/fp_format
with 2 racks, 14TB of HBM, and parameters in FP4?
I think Caro would say that LBJ didn’t just change his mind. Eg he frequently claims that “power reveals” (see also here). And IIRC he describes LBJ as already caring about racial inequality when teaching poor Mexican children in Cotulla.
I’m not sure how much to defer to Caro on this.
Within simulations, probably most of the impact comes from simulations that are created to decide which civilizations should get more resources outside the simulation, and not from sims run for random entertainment purposes.
If we are in a simulation, simulators might be extremely different and incomprehensible to us. “Entertainment” and “to decide which civilizations should get more resources” are just two reasons, there’s probably many others that it’s hard for us to even conceive of?
This being said, these kinds of unknown unknowns probably don’t change the upshot from our point of view.
I suspect the AI Futures Model modestly underestimates takeoff speeds and one-time acceleration effects due to effectively acting as though AI speed and quantity don’t matter outside of coding automation.
On the other hand, doesn’t the Forethought model assume a Cobb-Douglas production function? If the goal is to figure out how far a one-time speedup from automating AI R&D takes you, the AI Futures Model seems better since it models (automated labor, experiment compute) complementarities in more detail?
Or to be more precise, p(not near-fully automated), directly quoting your prediction.
p(not fully automated)
I did a quick poll of some Epoch AI staff:
2 have probabilities between 70% and 90%
2 have probabilities between 90% and 97%
4 have probabilities between 97% and 99%
Probably not entertaining enough, but I like the idea of reenacting the Great Air Conditioner War of 2022
[to-do: attach paper]
I think the paper you meant to attach might be Extracting memorized pieces of (copyrighted) books from open-weight language models.
If you’re trying to explain why we see straight lines on graphs when plotting Y against X, your explanation shouldn’t also predict straight lines of graphs when plotting f(Y) against X for any function f.
I have had a similar experience, also with Opus.
My impression is that Pangram cares especially about avoiding false positives, see e.g. this from their website:
Insofar as character training could be shown to be a superior approach than a spec, one might hope for Anthropic to publish papers about what they’re doing technically and how they know it works. Is it just supervised learning on the text of the Constitution, to shape the model’s latent concept of “Claude”, or is there more to it? (Does having the Constitution in context during reinforcement learning do anything special?)
A recent example that comes to mind: Evan Hubinger is an author on this paper on Open Character Training. I don’t know how much it reveals about what Anthropic themselves are doing though.
Not sure how to interpret but a Microsoft blog post claims that “The GPT-5.2 series is built on new architecture”.
The AI Action Plan has a section on this, it recommends that Commerce and IC coordinate on enforcement, including monitoring.
I don’t think Anthropic has put as much effort into RL-ing their models to perform well on tasks like VendingBench or Computer Use (/graphical browser use) compared to “being a good coding agent”. Anthropic makes a lot of money from coding, whereas the only computer use release I know from them is a demo which ETA: I’d guess does not generate much revenue (?).
Similarly for scaffolding, I expect the number of person-hours put into the scaffolding for Vending Bench or the AI Agent village to be at least an order-of-magnitude lower than for Claude Code, which is a publicly released product that Anthropic makes money from.
More concretely:
I think that the kind of decomposition into subtasks / delegation to subagents that Claude Code does would be helpful for VendingBench and the Agent Village, because in my experience they help keep track of the original task at hand and avoid infinite rabbit holes.
Cursor builds great tools/interfaces for models to interact with code files, and my impression is that Ant post-training intentionally targeted these tools, which made Anthropic models much better than OpenAI models on Cursor pre-GPT-5. I don’t expect there were comparable customization efforts in other domains, including graphical browser use or VendingBench style tasks. I think such efforts are under way, starting with high quality integrations into economically productive software like Figma.
I’m more confident for VendingBench than for the AI village, for example I just checked the Project Vend blog post and it states:
Many of the mistakes Claudius made are very likely the result of the model needing additional scaffolding—that is, more careful prompts, easier-to-use business tools. In other domains, we have found that improved elicitation and tool use have led to rapid improvement in model performance.
[...]
Although this might seem counterintuitive based on the bottom-line results, we think this experiment suggests that AI middle-managers are plausibly on the horizon. That’s because, although Claudius didn’t perform particularly well, we think that many of its failures could likely be fixed or ameliorated: improved “scaffolding” (additional tools and training like we mentioned above) is a straightforward path by which Claudius-like agents could be more successful.
Regarding the AI village: I do think that computer (/graphical browser) use is harder than coding in a bunch of ways, so I’m not claiming that if Ant spent as many resources on RL + elicitation for computer use as they did for coding, that would reduce errors to the same extent (and of course, making these comparison across task types is conceptually messy). For example, computer use offers a pretty unnatural and token-inefficient interface, which makes both scaffolding and RL harder. I still think more OOM more resources dedicated to elicitation would close a large part of the gap, especially for ‘agency errors’.
It seems totally possible that performance on messier real world tasks with hard-to-check objectives is under elicited
As an example, I think that the kinds of agency errors that show up in VendingBench or AI Village are largely due to lack of elicitation. I see these errors way less when coding with Claude Code (which has both better scaffolding and more RL). I’d find it difficult to inject numbers to get a concrete bound though.
I already expect some (probably substantial) effect from AIs helping to build RL environments
I think scraping and filtering MCP servers then RL training to navigate them is largely even if not fully automatable and already being done (cf this for SFT), but doesn’t unlock massive value.
One exception where it’s useful for defensive purposes: if an AI model claims to have found a vulnerability in a codebase, a proof of concept is an efficient demonstration that the claimed vulnerability is legit. See also here.