I wouldn’t say the appendix examples have that shape. Eg the arithmetic one involves 6 serial steps (I think), not 2 + tons of parallelization. The one about finding the modal answer from a bunch of mathematical operations is kinda like that, the others don’t seem that way to me. I think Astra is good at lots of synthetic tasks, and the ones you describe are an example of what it’s good at.
Neel Nanda
Karma: 16,627
It’s an obscure code knowledge benchmark, where I ask about the value of a parameter, and accept either what’s written in the code, or the raw value it evaluates to. Item 101 asks for the default value of the mode parameter of shutil.which. In CPython that default is written os.F_OK | os.X_OK in the source and evaluates to the integer 1
It doesn’t seem to need to be a specific operation, eg multi hop arithmetic or factual recall involves different operations. But I haven’t checked if using qualitatively different types of operation matters
I would be surprised if they didn’t do this for Sol but suddenly did a ton of this for Astra, and Sol is not disproportionate in the same way. On one task pointer chasing both Astra and Sol are disproportionately good so they might have trained on that but I just didn’t include that task in my analysis because it felt confounded
My investigation of looped versus unrolled code is some evidence that the model does not need each token position to do separate serial steps, but it does help. I did another experiment where I gave it 2,000 tuples of numbers. A rule of aggregating them like a a—c and told it to take the Max of that and gave it the rule at the start or at the end over the list and the model could somewhat do it in both cases. But it was fair bit better when the rule came before. I was kind of surprised it could do it at all the rule after! There may be hacks on a simple operation like that though
Thanks for the catch, fixed. Yes, dictation errors, it seems I did not proof-read my appendices very hard
Both with no CoT
Astra can do a concerning amount with no chain of thought
That plan seems more reasonable! Being a current employee of an AGI lab will make protesting and other comms things notably more effective. I do expect loudly quitting in protest to be even more effective though (see eg Jacob Coxon today)
I feel like the main reason to quit is because you think you can have more impact elsewhere (or can have a bunch of impact in the act of quitting). My guess is that in most cases, someone refusing to work on capabilities work would not be fired, but might be somewhat sidelined, wouldn’t have that much influence, and be somewhat socially uncomfortable. The company is unlikely to make someone a safety martyr by firing them if they can just have a less useful employee around instead (and if said employee is doing safety work, that’s typically pretty good from the company’s perspective)
+1. I think that my MATS application being so costly in time totally loses me good applicants, I just consider this a price worth paying for the signal it gives me, and ability to be meritocratic
IMO the key factor is that there were safety people inside OpenAI pushing on this, and safety people at METR and Redwood on good terms with and trusted by OpenAI. Additional important factors were likely OpenAI employees caring about the company doing legibly reasonable safety things and seeming to take things seriously. OpenAI’s founding involving safety people is partially upstream of this but doesn’t seem key
To me a lot of the rationale for focusing on bio is that the magnitude of harm from a really scary bio attack is extremely high, as it could spread to be a global pandemic, with a high / possibly 100% fatality rate. I’m not aware of ways to do comparable amounts of harm with chemical weapons, so my sense was that this was lower priority. But I may just be ignorant of ways to do vast amounts of harm with chemical weapons?
Does DiffusionGemma do latent reasoning?
The effect size of divestment just seems way too small to matter to me—AI is the current biggest frenzy among investors, data centres are holding up US GDP, it’s making trillion dollar companies, some of the fastest growing consumer products ever, etc. And there’s substantial productivity advantages to people using AI. I don’t feel like a plan involving divestment and boycotting will work and I would be sad to see safety orgs following it rather than things I thought could meaningfully reduce x risk.
I’m very proud of my hypothetical scholars—with so many top researchers talking about it, showing P is clearly difficult and of high interest to the community!
Why do models task game?
R-lens: Making J-lens More Faithful on Early Layers
The AGI Safety and Alignment team at Google DeepMind is Hiring (July 2026)
I don’t know which is better, We’ve worked much more on mha. But keys and values aren’t bottlenecks in the same way as the residual stream (unless you consider the entire kv cache)
Good idea, done