I think it’s a better reference class in that it’s closer, but it also seems reasonable to argue that nonprofits overall are less efficient than corporations (due to worse feedback loops), and that EA should follow best practices in the corporate world to be more functional.
I don’t have a strong opinion on this question. It was discussed previously here.
Thank you for this post! I’d love to know how you think about the results in Measuring coding agent misalignment in the wild in this context!
Beyond self-exfiltration, don’t cybersecurity capabilities make it easier to launch a rogue internal deployment?
This is the side of cybersecurity that policy-classifiers forbid, because it’s not really dual-use, only bad use
One exception where it’s useful for defensive purposes: if an AI model claims to have found a vulnerability in a codebase, a proof of concept is an efficient demonstration that the claimed vulnerability is legit. See also here.
Nobody at CG runs the kind of blog that Holden used to write or Joe Carlsmith used to write.
Probably agree on the broader trend, but I like Defenses in Depth.
The constraint from pipelining on total FP4 params is 1.3T for H100 in 2023, 1.1T for H200 in 2024, 27T for GB200 Oberon in 2025, 20T for GB300 Oberon in 2026, 83T for Rubin Oberon racks in 2027
Why 27T and not 28T for GB200?
N_params = N_racks * HBM/2 * 8/fp_format
with 2 racks, 14TB of HBM, and parameters in FP4?
I think Caro would say that LBJ didn’t just change his mind. Eg he frequently claims that “power reveals” (see also here). And IIRC he describes LBJ as already caring about racial inequality when teaching poor Mexican children in Cotulla.
I’m not sure how much to defer to Caro on this.
Within simulations, probably most of the impact comes from simulations that are created to decide which civilizations should get more resources outside the simulation, and not from sims run for random entertainment purposes.
If we are in a simulation, simulators might be extremely different and incomprehensible to us. “Entertainment” and “to decide which civilizations should get more resources” are just two reasons, there’s probably many others that it’s hard for us to even conceive of?
This being said, these kinds of unknown unknowns probably don’t change the upshot from our point of view.
I suspect the AI Futures Model modestly underestimates takeoff speeds and one-time acceleration effects due to effectively acting as though AI speed and quantity don’t matter outside of coding automation.
On the other hand, doesn’t the Forethought model assume a Cobb-Douglas production function? If the goal is to figure out how far a one-time speedup from automating AI R&D takes you, the AI Futures Model seems better since it models (automated labor, experiment compute) complementarities in more detail?
Or to be more precise, p(not near-fully automated), directly quoting your prediction.
p(not fully automated)
I did a quick poll of some Epoch AI staff:
2 have probabilities between 70% and 90%
2 have probabilities between 90% and 97%
4 have probabilities between 97% and 99%
Probably not entertaining enough, but I like the idea of reenacting the Great Air Conditioner War of 2022
[to-do: attach paper]
I think the paper you meant to attach might be Extracting memorized pieces of (copyrighted) books from open-weight language models.
If you’re trying to explain why we see straight lines on graphs when plotting Y against X, your explanation shouldn’t also predict straight lines of graphs when plotting f(Y) against X for any function f.
I have had a similar experience, also with Opus.
My impression is that Pangram cares especially about avoiding false positives, see e.g. this from their website:
Insofar as character training could be shown to be a superior approach than a spec, one might hope for Anthropic to publish papers about what they’re doing technically and how they know it works. Is it just supervised learning on the text of the Constitution, to shape the model’s latent concept of “Claude”, or is there more to it? (Does having the Constitution in context during reinforcement learning do anything special?)
A recent example that comes to mind: Evan Hubinger is an author on this paper on Open Character Training. I don’t know how much it reveals about what Anthropic themselves are doing though.
Not sure how to interpret but a Microsoft blog post claims that “The GPT-5.2 series is built on new architecture”.
The AI Action Plan has a section on this, it recommends that Commerce and IC coordinate on enforcement, including monitoring.
Ironically, 3M was also the defendant in this recent lawsuit.