For what it’s worth, I think the stronger criticisms by @1a3orn on the AI 2027 story revolve around data not being nearly as central to AI 2027 as 1a3orn expects it to, combined with thinking that external only algorithm research can matter, and brake the software only singularity.
My main objection to @1a3orn’s memory point is that I think that reproducibility is mostly solvable so long as you are willing to store earlier states, similar to how version control software stores earlier versions of software that have bugs that production versions fixed, and I expect memory to be a huge cause in why humans are more effective and have decreasing failure rates on tasks they work on, compared to AI’s constant failure rates because it allows humans to store context, and given that I expect AI companies to go for paradigms that produce the most capabilities, combined with me thinking that memory is plausibly a necessary capability for AIs that can automate jobs, and I expect things to look more like a temporally continuous 1 AI instance than you say.
I have updated towards memory being potentially more necessary for value to be unlocked by AI than I used to.
On China and open source, a big reason I expect open sourcing to stop being done is because the PR risks from potential misuse of models that are, for example capable enough to do bioterror at mass scales and replace virologists is huge, and unless we can figure out a way to prevent safeguards from being removed by open-sourcing the model, which they won’t be, this means companies/nations will have huge PR risks from trying to open-source AI models past a certain level of capabilities:
I can maybe see it. Consider the possibility that the decision to stop providing public access to models past some capability level is convergent: e. g., the level at which they’re extremely useful for cyberwarfare (with jailbreaks still unsolved) such that serving the model would drown the lab in lawsuits/political pressure, or the point at which the task of spinning up an autonomous business competitive with human businesses, or making LLMs cough up novel scientific discoveries, becomes trivial (i. e., such that the skill level required for using AI for commercial success plummets – which would start happening inasmuch as AGI labs are successful in moving LLMs to the “agent” side of the “tool/agent” spectrum).
In those cases, giving public access to SOTA models would stop being the revenue-maximizing thing to do. It’d either damage your business reputation[1], or it’d simply become more cost-effective to hire a bunch of random bright-ish people and get them to spin up LLM-wrapper startups in-house (so that you own 100% stake in them).
Some loose cannons/open-source ideologues like DeepSeek may still provide free public access, but those may be few and far between, and significantly further behind. (And getting progressively scarcer; e. g., the CCP probably won’t let DeepSeek keep doing it.)
Less extremely, AGI labs may move to a KYC-gated model of customer access, such that only sufficiently big, sufficiently wealthy entities are able to get access to SOTA models. Both because those entities won’t do reputation-damaging terrorism, and because they’d be the only ones able to pay the rates (see OpenAI’s maybe-hype maybe-real whispers about $20,000/month models).[2] And maybe some EA/R-adjacent companies would be able to get in on that, but maybe not.
Here’s some threads on data and the software-only singularity:
This sequence of posts is on data mattering more to AI 2027 than advertised:
“Additionally, of course, if data (of some sort) turns out to be a strict limiting factor, than the compute lead might not matter.
We might just be gated on ability to set up RL envs (advantage to who has more talent, at least at first) and who has more robots (China).”
“In general i agree, but this piece is about why the US wins in AI 2027. The data is ~all synthetic and focused on a software-only improvements.
There’s also another kind of data which can come from paying PhD-level humans to label data. In that case total $ wins.”
“Regarding “will AI produces software singularity via a country of geniuses in a datacenter.”
A piece of evidence that bears on this—in some research lab, what proportion of AI progress comes from *internal* research vs. *external* research?
1/n
Luke Frymire asked a question about whether external research might keep pace after all, and thus a software only singularity might be sustained:
It seems like most people contributing to ML research are at one of the top ~10 AI orgs, who all have access to near-frontier models and a significant fraction of global compute. In which case I’d expect external research to keep pace.
“And this outside pool of people is much larger, exploring a broader space of hypotheses, and also much more physically engaged with the world.
You have like ~500 people researching AI inside, but plausibly many many more (10k? 100k) outside whose work *might* advance AI.”
The point is that “AI replacing all internal progress” is actually a different task than “AI replacing all the external progress.”
Potentially, a much easier task.
At a brute level—there’s just a lot more people AI has to replace outside! And more world-interaction.
And maaaybe this is true?
But part of the reason the external stuff might be effective (if it is effective, which I’m not sure about) is because it’s just a huge, brute-force search crawling over empirical matter.
Suppose it comes from this vast distributed search of idiosyncratic people doing their own thing, eventually stumbling upon the right hypotheses, but where even the person who suggested it was unjustified in their confidence?
And you could only really replace this civilizational search when you have like—a civilization in the datacenter, doing *all the things* that a civilization does, including things only vaguely related to AI.
I don’t know about the above view, I don’t 100% endorse it.
But—the software singularity view tries to exclude the need for external hardware progress by focusing just on algorithms. But a lab might be no more self-sufficient in algorithms than in hardware!
And so slowness of external world creeps in, even in the external world.
Anyhow, looking at how much progress in an AI lab is external vs. internal would probably provide evidence on this. Maybe.
On China and open source, a big reason I expect open sourcing to stop being done is because the PR risks from potential misuse of models that are, for example capable enough to do bioterror at mass scales and replace virologists is huge, and unless we can figure out a way to prevent safeguards from being removed by open-sourcing the model, which they won’t be, this means companies/nations will have huge PR risks from trying to open-source AI models past a certain level of capabilities:
And…they’re more concerned about the PR risk than the actual bioterror? What planet is this? Oh. Right.
For what it’s worth, I think the stronger criticisms by @1a3orn on the AI 2027 story revolve around data not being nearly as central to AI 2027 as 1a3orn expects it to, combined with thinking that external only algorithm research can matter, and brake the software only singularity.
My main objection to @1a3orn’s memory point is that I think that reproducibility is mostly solvable so long as you are willing to store earlier states, similar to how version control software stores earlier versions of software that have bugs that production versions fixed, and I expect memory to be a huge cause in why humans are more effective and have decreasing failure rates on tasks they work on, compared to AI’s constant failure rates because it allows humans to store context, and given that I expect AI companies to go for paradigms that produce the most capabilities, combined with me thinking that memory is plausibly a necessary capability for AIs that can automate jobs, and I expect things to look more like a temporally continuous 1 AI instance than you say.
I have updated towards memory being potentially more necessary for value to be unlocked by AI than I used to.
On China and open source, a big reason I expect open sourcing to stop being done is because the PR risks from potential misuse of models that are, for example capable enough to do bioterror at mass scales and replace virologists is huge, and unless we can figure out a way to prevent safeguards from being removed by open-sourcing the model, which they won’t be, this means companies/nations will have huge PR risks from trying to open-source AI models past a certain level of capabilities:
https://www.lesswrong.com/posts/3NdpbA6M5AM2gHvTW/short-timelines-don-t-devalue-long-horizon-research#fWqYjDc8dpFiRbebj
Relevant part quoted:
Here’s some threads on data and the software-only singularity:
This sequence of posts is on data mattering more to AI 2027 than advertised:
https://x.com/1a3orn/status/1916547321740828767
“Scott Alexander: Algorithmic progress and compute are the two key things you need for AI progress. Data: ?????????”
https://x.com/1a3orn/status/1916552734599168103
“If data depends on active learning (robots, autolabs) then China might have a potentially very large lead in data.”
https://x.com/1a3orn/status/1916553075021525406
“Additionally, of course, if data (of some sort) turns out to be a strict limiting factor, than the compute lead might not matter. We might just be gated on ability to set up RL envs (advantage to who has more talent, at least at first) and who has more robots (China).”
https://x.com/1a3orn/status/1916553736060625002
“In general I think rounding data ~= algorithms is a questionable assumption.”
@romeo’s response:
https://x.com/romeovdean/status/1916555627247083934
“In general i agree, but this piece is about why the US wins in AI 2027. The data is ~all synthetic and focused on a software-only improvements. There’s also another kind of data which can come from paying PhD-level humans to label data. In that case total $ wins.”
On external vs internal research:
https://x.com/1a3orn/status/1919824435487404086
“Regarding “will AI produces software singularity via a country of geniuses in a datacenter.” A piece of evidence that bears on this—in some research lab, what proportion of AI progress comes from *internal* research vs. *external* research? 1/n
Luke Frymire asked a question about whether external research might keep pace after all, and thus a software only singularity might be sustained:
https://x.com/lukefrymire/status/1919853901089579282
It seems like most people contributing to ML research are at one of the top ~10 AI orgs, who all have access to near-frontier models and a significant fraction of global compute. In which case I’d expect external research to keep pace.
https://x.com/1a3orn/status/1919824444060488097
“And this outside pool of people is much larger, exploring a broader space of hypotheses, and also much more physically engaged with the world. You have like ~500 people researching AI inside, but plausibly many many more (10k? 100k) outside whose work *might* advance AI.”
https://x.com/1a3orn/status/1919824447118131400
The point is that “AI replacing all internal progress” is actually a different task than “AI replacing all the external progress.” Potentially, a much easier task. At a brute level—there’s just a lot more people AI has to replace outside! And more world-interaction.
https://x.com/1a3orn/status/1919824450825969783
And maaaybe this is true? But part of the reason the external stuff might be effective (if it is effective, which I’m not sure about) is because it’s just a huge, brute-force search crawling over empirical matter.
https://x.com/1a3orn/status/1919824452549787881
What if some progress in AI (and science) doesn’t come from people doing experiments with incredibly good research taste.
https://x.com/1a3orn/status/1919824453971628234
Suppose it comes from this vast distributed search of idiosyncratic people doing their own thing, eventually stumbling upon the right hypotheses, but where even the person who suggested it was unjustified in their confidence?
https://x.com/1a3orn/status/1919824455557087407
And you could only really replace this civilizational search when you have like—a civilization in the datacenter, doing *all the things* that a civilization does, including things only vaguely related to AI.
https://x.com/1a3orn/status/1919824457327059451
I don’t know about the above view, I don’t 100% endorse it. But—the software singularity view tries to exclude the need for external hardware progress by focusing just on algorithms. But a lab might be no more self-sufficient in algorithms than in hardware!
https://x.com/1a3orn/status/1919824463299752405
And so slowness of external world creeps in, even in the external world. Anyhow, looking at how much progress in an AI lab is external vs. internal would probably provide evidence on this. Maybe.
And…they’re more concerned about the PR risk than the actual bioterror? What planet is this? Oh. Right.