jsd

Karma: 482

jsd 5 Jun 2026 23:21 UTC
1 point
0
in reply to: Eric Neyman’s comment on: Eric Neyman’s Shortform
I think Caro would say that LBJ didn’t just change his mind. Eg he frequently claims that “power reveals” (see also here). And IIRC he describes LBJ as already caring about racial inequality when teaching poor Mexican children in Cotulla.
I’m not sure how much to defer to Caro on this.

jsd 1 Jun 2026 3:45 UTC
1 point
0
on: Infinite ethics and UDASSA
Within simulations, probably most of the impact comes from simulations that are created to decide which civilizations should get more resources outside the simulation, and not from sims run for random entertainment purposes.
If we are in a simulation, simulators might be extremely different and incomprehensible to us. “Entertainment” and “to decide which civilizations should get more resources” are just two reasons, there’s probably many others that it’s hard for us to even conceive of?
This being said, these kinds of unknown unknowns probably don’t change the upshot from our point of view.

jsd 27 May 2026 18:42 UTC
1 point
0
on: Full automation of AI R&D probably yields a large speed up even without a software-only singularity
I suspect the AI Futures Model modestly underestimates takeoff speeds and one-time acceleration effects due to effectively acting as though AI speed and quantity don’t matter outside of coding automation.
On the other hand, doesn’t the Forethought model assume a Cobb-Douglas production function? If the goal is to figure out how far a one-time speedup from automating AI R&D takes you, the AI Futures Model seems better since it models (automated labor, experiment compute) complementarities in more detail?

jsd 14 May 2026 20:48 UTC
1 point
0
in reply to: jsd’s comment on: leogao’s Shortform
Or to be more precise, p(not near-fully automated), directly quoting your prediction.

jsd 14 May 2026 20:46 UTC
1 point
0
in reply to: leogao’s comment on: leogao’s Shortform
p(not fully automated)

jsd 14 May 2026 18:54 UTC
1 point
0
in reply to: leogao’s comment on: leogao’s Shortform
I did a quick poll of some Epoch AI staff:
- 2 have probabilities between 70% and 90%
- 2 have probabilities between 90% and 97%
- 4 have probabilities between 97% and 99%

jsd 7 May 2026 14:55 UTC
2 points
2
in reply to: Ben Pace’s comment on: Open Thread Winter 2025/26
Probably not entertaining enough, but I like the idea of reenacting the Great Air Conditioner War of 2022

jsd 20 Apr 2026 18:04 UTC
10 points
0
on: Finetuning Borges
[to-do: attach paper]
I think the paper you meant to attach might be Extracting memorized pieces of (copyrighted) books from open-weight language models.

jsd 2 Apr 2026 3:01 UTC
2 points
0
on: jsd’s Shortform
If you’re trying to explain why we see straight lines on graphs when plotting Y against X, your explanation shouldn’t also predict straight lines of graphs when plotting f(Y) against X for any function f.

jsd 31 Mar 2026 1:03 UTC
6 points
2
in reply to: Alex A’s comment on: Pangram (AI detection software) can be evaded
I have had a similar experience, also with Opus.
My impression is that Pangram cares especially about avoiding false positives, see e.g. this from their website:

jsd 9 Mar 2026 16:08 UTC
6 points
2
on: Prologue to Terrified Comments on Claude’s Constitution
Insofar as character training could be shown to be a superior approach than a spec, one might hope for Anthropic to publish papers about what they’re doing technically and how they know it works. Is it just supervised learning on the text of the Constitution, to shape the model’s latent concept of “Claude”, or is there more to it? (Does having the Constitution in context during reinforcement learning do anything special?)
A recent example that comes to mind: Evan Hubinger is an author on this paper on Open Character Training. I don’t know how much it reveals about what Anthropic themselves are doing though.

jsd 20 Jan 2026 15:23 UTC
3 points
0
in reply to: Aaron_Scher’s comment on: Aaron_Scher’s Shortform
Not sure how to interpret but a Microsoft blog post claims that “The GPT-5.2 series is built on new architecture”.

jsd 2 Oct 2025 22:57 UTC
3 points
0
in reply to: ChristianKl’s comment on: ChristianKl’s Shortform
The AI Action Plan has a section on this, it recommends that Commerce and IC coordinate on enforcement, including monitoring.

jsd 23 Sep 2025 23:27 UTC
3 points
0
in reply to: Daniel Kokotajlo’s comment on: We are likely in an AI overhang, and this is bad.
I don’t think Anthropic has put as much effort into RL-ing their models to perform well on tasks like VendingBench or Computer Use (/graphical browser use) compared to “being a good coding agent”. Anthropic makes a lot of money from coding, whereas the only computer use release I know from them is a demo which ETA: I’d guess does not generate much revenue (?).
Similarly for scaffolding, I expect the number of person-hours put into the scaffolding for Vending Bench or the AI Agent village to be at least an order-of-magnitude lower than for Claude Code, which is a publicly released product that Anthropic makes money from.
More concretely:
- I think that the kind of decomposition into subtasks / delegation to subagents that Claude Code does would be helpful for VendingBench and the Agent Village, because in my experience they help keep track of the original task at hand and avoid infinite rabbit holes.
- Cursor builds great tools/interfaces for models to interact with code files, and my impression is that Ant post-training intentionally targeted these tools, which made Anthropic models much better than OpenAI models on Cursor pre-GPT-5. I don’t expect there were comparable customization efforts in other domains, including graphical browser use or VendingBench style tasks. I think such efforts are under way, starting with high quality integrations into economically productive software like Figma.
I’m more confident for VendingBench than for the AI village, for example I just checked the Project Vend blog post and it states:
Many of the mistakes Claudius made are very likely the result of the model needing additional scaffolding—that is, more careful prompts, easier-to-use business tools. In other domains, we have found that improved elicitation and tool use have led to rapid improvement in model performance.
[...]
Although this might seem counterintuitive based on the bottom-line results, we think this experiment suggests that AI middle-managers are plausibly on the horizon. That’s because, although Claudius didn’t perform particularly well, we think that many of its failures could likely be fixed or ameliorated: improved “scaffolding” (additional tools and training like we mentioned above) is a straightforward path by which Claudius-like agents could be more successful.
Regarding the AI village: I do think that computer (/graphical browser) use is harder than coding in a bunch of ways, so I’m not claiming that if Ant spent as many resources on RL + elicitation for computer use as they did for coding, that would reduce errors to the same extent (and of course, making these comparison across task types is conceptually messy). For example, computer use offers a pretty unnatural and token-inefficient interface, which makes both scaffolding and RL harder. I still think more OOM more resources dedicated to elicitation would close a large part of the gap, especially for ‘agency errors’.

jsd 23 Sep 2025 16:40 UTC
5 points
0
in reply to: ryan_greenblatt’s comment on: We are likely in an AI overhang, and this is bad.
It seems totally possible that performance on messier real world tasks with hard-to-check objectives is under elicited
As an example, I think that the kinds of agency errors that show up in VendingBench or AI Village are largely due to lack of elicitation. I see these errors way less when coding with Claude Code (which has both better scaffolding and more RL). I’d find it difficult to inject numbers to get a concrete bound though.

jsd 6 Sep 2025 19:29 UTC
1 point
0
in reply to: Boaz Barak’s comment on: boazbarak’s Shortform
Neat!
“Does AI Progress Have a Speed Limit?” links to an April 2025 dialogue between Ajeya Cotra and Arvind Narayanan. Perhaps you wanted to link the 2023 dialogue between Tamay and Matt Clancy, also on Asterisk?

jsd 3 Sep 2025 15:28 UTC
7 points
0
on: Trust me bro, just one more RL scale up, this one will be the real scale up with the good environments, the actually legit one, trust me bro
I already expect some (probably substantial) effect from AIs helping to build RL environments
I think scraping and filtering MCP servers then RL training to navigate them is largely even if not fully automatable and already being done (cf this for SFT), but doesn’t unlock massive value.

jsd 3 Sep 2025 14:42 UTC
LW: 1 AF: 1
0
AF
in reply to: ryan_greenblatt’s comment on: Trust me bro, just one more RL scale up, this one will be the real scale up with the good environments, the actually legit one, trust me bro
In their BOTEC, it seems you roughly agree with a group size of 64 and 5 reuses per task (since 5 * 64 is between 100 and 1k).
You wrote $0.1 to $1 per rollout, whereas they have in mind 500,000 * $15 / 1M = $7.5. 500,000 doesn’t seem especially high for hard agentic software engineering tasks which often reach into the millions.
Does the disagreement come from:
- Thinking the $15 estimate from opportunity cost is too high (so compute cost lower than Mechanize claims)
- Expecting most of the RL training to somehow not be end-to-end? (so compute cost lower than Mechanize claims)
- Expecting spending per RL environment to be smaller than compute spending, even if within an OOM.

jsd 3 Sep 2025 14:18 UTC
1 point
0
on: Trust me bro, just one more RL scale up, this one will be the real scale up with the good environments, the actually legit one, trust me bro
ongoing improvements to RL environments are already priced into the existing trend
I agree with this especially for e.g. METR tasks, or proxies for how generally smart a model is.
A case for acceleration in enterprise revenue (rather than general smarts) could look like:
- So far RL still has been pretty targeted towards coding, research/browsing, math, or being-ok-at-generic-tool-use (taking random MCP servers and making them into environments, like Kimi K2 did but with RL).
- It takes SWE time to build custom interfaces for models to work with economically productive software like Excel, Salesforce, or Photoshop. We’re not there yet, at least with publicly released models. Once we are, this suddenly unlocks a massive amount of economic value.
Ultimately I don’t really buy this either, since we already have e.g. some Excel/Sheets integrations that are not great but better than what there was a couple months ago. And increase in breadth of RL environments is probably already factored into the trend somewhat.
ETA: this also matters less if you’re primarily tracking AI R&D capabilities (or it might but indirectly, through driving more investment etc.).

jsd 3 Sep 2025 13:58 UTC
LW: 4 AF: 3
0
AF
on: Trust me bro, just one more RL scale up, this one will be the real scale up with the good environments, the actually legit one, trust me bro
Another way to think about this is that it could be reasonable to spend within the same order of magnitude on each RL environment as you spend in compute cost to train on that environment. I think the compute cost for doing RL on a hard agentic software engineering task might be around $10 to $1000 ($0.1 to $1 for each long rollout and you might do 100 to 1k rollouts?), so this justifies a lot of spending per environment. And, environments can be reused across multiple training runs (though they could eventually grow obsolete).
Agreed, cf https://www.mechanize.work/blog/cheap-rl-tasks-will-waste-compute/