Noosphere89

Karma: 4,969

Noosphere89 7 Aug 2026 13:45 UTC
6 points
0
in reply to: Rauno Arike’s comment on: Expertium’s Shortform
The concrete version of this is developing the method described in the paper Undetectable Conversations Between AI Agents via Pseudorandom Noise-Resilient Key Exchange, developed on April 6th, 2026.
If this happened in the future, this would effectively remove CoT interpretability as a technique and allow much more steganography, at least without actively attempting to intervene on the CoT (but unfortunately humans are incompetent and will likely never notice the need to intervene if they don’t see anything suspicious.)

Noosphere89 6 Aug 2026 17:17 UTC
4 points
0
in reply to: Steven Byrnes’s comment on: LLMs are (still) mostly powered by imitative learning, not RL
Maybe it sounds absurd for me to compare what one RLVR training can do, versus the whole edifice of ideas painstakingly built by the mathematics community over the course of 200 years. But it’s not absurd: AlphaZero really did blow past the whole edifice of ideas painstakingly built over the course of centuries by the chess and go communities in its 72-hour training runs. So the idea of building real new knowledge at a massive scale through RL is not absurd on its face.
I should flag here that Go has a number of properties that aren’t replicable in many other domains, and thus we shouldn’t anchor our estimates of takeoff in a given domain or in general to Go:
1. It’s very parallelizable, like easily parallelizable mathematics/SWE work, which is something a lot of domains lack. In particular, non-trivial serial feedback loops of the order of weeks, months or even years/decades in the most complicated, taste-driven tasks is much more common than people intuitively expect. This is related to failures of the factored cognition hypothesis in practice, and Dwarkesh talks about this in this podcast episode.
2. Go has a ~perfect simulator, and it’s not a coincidence that RLVR makes the greatest strides/adds new capabilities to LLMs precisely when they either have a very good simulator or a very good way to verify if you correctly did the thing or not. Again, this is lacking for many essential tasks for AIs to transform the economy. (This is discussed in AI safety/capability discourse as the difference between clean tasks vs messy tasks, where messy tasks lack both a good simulator and a good verifier, whereas clean tasks only need one of them to hold,)
And that’s more or less the entire issue here.
Notably, this does not mean that AIs can’t get good at the necessary messy tasks, or that the takeoff if directed by AGIs won’t lead to intuitively insane outcomes (Damon Binder’s series on robotics, AGI, and growth rates, especially parts 1, 4 and 5 are enough here), but it does mean that we should initially expect much slower takeoff speeds than the Go anchor/AlphaZero suggests.

Noosphere89 6 Aug 2026 2:18 UTC
6 points
1
on: Generalized atheism rules out “inaccurate simulation”-ism.
The actual problem with simulation hypotheses in the most general case is that they are maximally non-predictive, in the sense that they’d always be right, no matter what evidence you get, especially for people who want to rescue theism, and thus by conservation of expected evidence, cannot actually be a valid hypothesis for a Bayesian to have (which as a corollary of what’s proven forbids hypotheses that can never be made less probable, no matter what evidence you have gotten)
And I think the theist in the conversation is likely trying to violate conservation of expected evidence, so they’re making a mistake of reasoning.
Similar points apply to the Mathematical Universe Hypothesis, which with a broad enough definition of mathematics can be made into a trivially right hypothesis.

Noosphere89 4 Aug 2026 13:10 UTC
15 points
2
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
I also think that AI safety is in a much worse position today than it would have been if it’d taken a more principled truth-seeking stance a decade ago (for example, AI governance would have been less likely to throw in with the Democrats in a way that alienated MAGA). Indeed, with respect to engaging with the current administration, it seems like standard EA views are more of a liability than any of the things you mentioned by a significant margin, and I expect that having a right-winger at Resolution would be a net gain in political capital.
I’d want to argue that this is in fact not the case, and AI safety is in fact in a reasonably good (though not perfect) position now, and a lot of this is because we do have bipartisan AI safety bills that basically every faction from Democrats to MAGA can agree on like the FRONTIER act by Trahan and Olbernolte, and Trump is sufficiently scared by Mythos that he has started to restrict AI capabilities (unfortunately they’ve generally targeted external deployment, which is a poor choice because in the worlds where it matters, the alignment problem is much more likely to be solved by default)
I do believe that AI safety will become partisan by 2028 (assuming AI capabilities mostly continue on their growth trajectory), but the reason for this is because AI issues will have higher salience, and for a multitude of reasons relating to the U.S political system’s brokenness (especially the fact that multi-partyism/coalition governments aren’t supported, and European-style parties are illegal in large part due to primaries) combined with social issues mattering more than economic issues to Americans, it will inevitably become partisan regardless of what the rationalist/EA communities actually did, so long as it’s high-salience enough.
Which implies a very different strategy to avoid polarization ruining AI safety, in that you want to delay political salience of AI issues, and make more use of Secret Congress to pass useful AI safety bills.
Or in Anton Leicht’s words, the workable strategies are to either ride the dragon on salience, borrowing some time for AI issues to make them low-salience deliberately, or use compromise carrots and populist sticks for policies.
Maybe more generally, one frame that I notice in your footnotes that’s wrong is that rationalists/EAs could have avoided polarization via individual choices, whereas the more correct view is to say that this had much less impact than it seems, and that there are higher-level causes of the polarization that is not worth their time to deal with (because it’s not neglected as an issue, and other actors have a comparative advantage in solving the issue).

Noosphere89 3 Aug 2026 16:05 UTC
4 points
2
on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
As an aside, note that decision-theory-with-precommitments has no problem managing duplication events. That’s why I consider decision theory as the more fundamental object; probability theory is a subset of decision theory where the utility function is the Brier score or any other strictly proper scoring rule.
The utility function will also encode how to aggregate Brier scores across multiple duplicates; summing will lead to SIA-like behaviour, while averaging leads to SSA-like behaviour. In that view, a Sleeping Beauty who sums Brier scores and chooses to create a billion copies is not changing her current or future credence of tails; instead, she’s maximising her utility in the tails world.
This is also my takeaway from stuff like OSAC and Non-Realist UDASSA, where probabilities in general are much more like caring measures/utility functions than they are beliefs, and a caring measure expresses how much the agent cares about how things go in particular worlds.
This is also why contra David Matolcsi, I don’t find the subjectiveness of his solutions to be an actual problem.
Put another way, utility theory/decision theory is more general than probability theory, and the goal in those anthropic problems/acausal trade problems/decision theory problems is not to get a correct model according to a specific model of the world, but rather to maximize utility in various worlds.
This is unavoidably, irretrievably subjective, but you can learn to at least accept it as a basic precondition for doing well at these problems, even if you aesthetically dislike it.

Noosphere89 2 Aug 2026 16:42 UTC
3 points
1
on: Freeing Thucydides
I think something like this is pretty likely in worlds where multipolar AGI outcomes happen, and this has an important consequence that I think a lot of futurists miss:

Evolutionary/selective forces become way, way weaker relative to deliberate design/agreements, and more generally the possibility of deal-making is one of the biggest reasons why the future could be very, very good even by your own lights, and it’s one of the most likely ways humans ever have most of what they want according to their values.
This consideration is a part of why I don’t think it matters for us to self-correct, unlike Wei Dai. We already have techniques that are robust under a lot of philosophical uncertainty, we just need to implement them.
A non-trivial amount of futures rely on the assumption of evolutionary/selective forces continuing, and are therefore less probable than they seem based on historical evidence.
Gradual Disempowerment/The Intelligence Curse, at least in it’s more negative/totalizing forms, is a good example of scenarios I expect to be less likely because of this consideration.
Similarly, Burning the Cosmic Commons is another scenario that I give less weight to based on this consideration (Honestly a lot of Robin Hanson-esque futures rely on the assumption of evolution continuing, which I find much less likely than he does)
It is not a coincidence that the futures that are made less likely by deal-making are the futures we’d consider bad, as evolution is misaligned to us, especially at the large scale/long run, and also makes cooperation harder.

Noosphere89 1 Aug 2026 13:19 UTC
4 points
2
on: Claude also hacked external companies during cyber evals
I should flag here that as much as Anthropic downplayed the incidents’ severity, I also think this does provide some optimistic signs regarding safety/alignment training, especially because in at least one of the incidents, the model stopped doing the attack after it realized it was in a real situation and not a test, which is not something that OpenAI’s models did when the HuggingFace incident happened, and we don’t know how much damage it would have done if the task required it to do as much damage as possible.
This still isn’t perfect, but reading between the lines here, this should update us in an optimistic direction on how effective safety training can be (at least relative to priors that claim safety training in current models is essentially useless/that safety training is like putting a mask on a shoggoth AI), and in general I’m starting to see why Anthropic is comfortable with calling their models “the most aligned to date”, at least relative to frontier models, including internal models, because it’s plausible this really is the case, especially as we don’t know what OpenAI was doing when they reduced refusals on their internal models.

Parallelization constraints could delay a technological singularity [Linkpost]

Noosphere8931 Jul 2026 17:48 UTC

18 points

0 comments6 min readLW link

(epoch.ai)

Noosphere89 31 Jul 2026 13:50 UTC
3 points
4
in reply to: JennaS’s comment on: You (Yes, You) Need A February 2020 Checklist for AI Policy
One likely default response is I expect people to start clamoring hard for open-source bans, and depending on what happens in November 2026, they could plausibly succeed.
More generally, I expect the public to be even more anti-AI, and for politicians to sate that appetite by starting to ban open-source.
One good example of this is soft law/FUD from the federal administration:
I would guess that the Trump Administration will at some point realize that their best strategy here would be to create large amounts of regulatory risk around the use of open-weight Chinese models. You don’t need to “ban open source” (one of the dumber motifs of AI policy discussion). You just need to direct every agency to issue soft law that creates FUD. “A Federal Reserve Advisory Bulletin found that there may be backdoors in Chinese AI models.” It needn’t be that well justified. You just create enough regulatory risk that every regulated enterprise backs off. You probably don’t want to create so much regulatory risk that you scare off the hyperscalers from serving Chinese models; this will just drive startups to sketchier providers. There’s a happy middle ground here. I’d assume they will do some version of this.
Or even just floating very interventionist ideas:
it’s pretty easy to screw with open source if you’re the admin! you can mess with procurement etc as Dean mentions. but you can also call up hyperscalers and float some absurd interventionist ideas if they continue to host Chinese OS at scale. that’s the problem with the token-hungry structure of deployments compared to OS software: this all runs through extremely expensive, mostly centralised, infrastructure.
This is not that bad, but it is a problem because it crowds out misalignment concerns and also because the path the US takes is not likely to be one that favors slowing down/pausing, because the open-source AIs won’t be misaligned (most likely.), and the people in power will just see it as a security problem.

Noosphere89 30 Jul 2026 14:08 UTC
5 points
2
in reply to: Stuart_Armstrong’s comment on: Value Generalisation 2: The Missing Hole in AIs’ abilities
The problem with LLMs and other modern designs is that once you turn a specific problem into an objective (or a benchmark) it’s generally easy to modify the AI to solve it.
Fully autonomous self-driving cars don’t exist. The biggest reason for this is that the AIs in them can’t generalise consistently to novel situations, and so human oversight remains necessary.
I tend to agree with this, but I will note that home robotics tasks are much better tests of generalization than most benchmarks, and the short answer to why this is comes down to OOD pressure from humans is essentially always there, plus it’s easier to restrain how much training data LLMs having access to robotics have in the physical world, plus generalization is a current bottleneck, and unlike other fields, robotics has tried to make much less general robots useful, and they haven’t been too successful, so unlike in other domains, there’s no overhang/inefficient market to exploit here.
I talk about this more at length in 2 comments here and here, and my main takeaway is pre-trained LLMs are both worse than humans at generalization, and also people tend to overestimate how much worse they are, and this means the LLM paradigm isn’t completely off the mark in the way past AI paradigms were (because they would have essentially no ability to do any of this before today.)
Notably though, this is only capabilities generalization, and more generally capability generalization and value generalization really are different, even if they are strongly correlated now.
This isn’t much of a problem if you expect your value generalization approach does work very well and as a side effect induces capability generalization, and truthfully I don’t expect much capability improvements from your line of research, so empirically it’s likely not a problem, but I did want to raise the point that capability generalization doesn’t automatically entail value generalization.

Noosphere89 29 Jul 2026 14:06 UTC
7 points
4
in reply to: Thane Ruthenis’s comment on: Why I Left Google DeepMind
The DoW is powerful and dangerous; making an enemy of it endangered Anthropic’s continued existence.
I think this is what turned out to be wrong. It’s still bad for Anthropic, and they have suffered consequences, but they haven’t really suffered existential consequences.
To be clear, it was in expectation an existentially risky move, because Anthropic didn’t know whether or not Amazon and Google would cut off their supply of compute due to the legal uncertainties of the supply chain risk designation, and if that happened Anthropic would have to close up shop soon.
But this turned out not to happen, and Amazon and Google supported Anthropic and still sold them the compute, and Trump’s legal position was weak enough that they couldn’t do anything (and yes, the law still matters even under Trump)
Fable was a more close call, because Anthropic was losing billions of revenue from having it be taken off the market for 2 and a half weeks, and because the classifiers had to be tuned in the direction of even more refusals, but Anthropic soon managed to switch the communicator from Dario to Tom Brown, who was better at politics, and managed to get Fable back on July 1st, before OpenAI managed to deploy GPT 5.6.
As it turns out, the U.S presidency’s leverage was weaker than it appeared both times, and this prevented Anthropic from collapsing/falling victim to Molochian/Pythian dynamics.

Noosphere89 28 Jul 2026 14:24 UTC
2 points
0
in reply to: TsviBT’s comment on: Noosphere89′s Shortform
I don’t understand this. Industrial robots of course are used very widely? Or I guess you typoed, and you’re trying to say, LLMs or other AIs would have been used to compute very non-obvious complex actions that take advantage of algebraicness in the narrow domain? Or you’re saying something about humanoid robots, or about softer / less rote tasks (laundry folding rather than installing a chair in a car)?
I agree industrial robots that don’t generalize much are used quite widely in narrow environments, the point is that for the set of tasks where AIs don’t generalize much/don’t improve generalization, they can’t improve industrial robotics/cause the robotics industry to be more useful and profitable, because the robotics industry already squeezed out ~all the gains from AIs that don’t generalize well to new environments.
Yes, this is about less rote tasks like laundry folding across arbitrarily oriented clothing and being crumpled, plus being in arbitrary locations in the home, while a human constantly disturbs them and provides OOD pressure (indeed one of my examples was about this exact thing being done well by an AI that controlled a robot using only pre-training without robotics data), and due to the way home robotics has infinite variation, long tails and constant change, this means it’s way easier to measure generalization cleanly/in a controlled manner.
“it stress-tests AI generalization” But does it though? What does it stress test? That seems to assert a generalization from AI generalization about XYZ robotics tasks to ABC intelligence explosion tasks (or something along these lines?). What’s that generalization / what justifies it?
The stress test of generalization is more indirect than this. My model is that as AIs improve at more and more robotics tasks, it simultaneously has to be able to do well in more and more messy and OOD environments (otherwise current robotics would have already improved more than they actually are), intelligence explosion tasks, assuming an intelligence explosion/software only singularity is possible, involves a great many very messy and OOD tasks, thus robotic evals are real evidence that AIs are increasingly able to start intelligence explosion tasks.
How do you get from the qualitative and relative statement about robotics being “a much stronger signal” to numbers and years?
Fair point, I don’t think I justified this very well, at least as a claim about the specific units, but a vague argument here is that given the qualitative statement above, it implies that the current paradigm is not so far off the mark that it’s absolutely useless as progress for AGI design, and this means there’s a real path that doesn’t require heroic genius from today to AGI.
To address your cruxes:
- the quantitative amount of insight (regardless of continuousness) remaining to get to AGI
My views on this is that I think at most 2 insights are necessary in practice, and if the human population kept perpetually growing according to 1880s-era birth-rates, we’d need 0 insights, as the super-exponential growth of the economy would have allowed us to get to AGI eventually.
- the degree to which those insights are blocked on difficult thinking that current AI doesn’t accelerate by much
I’d say they’re non-trivially serial (especially if the solutions use a lot of RL, as RL is much, much more difficult to parallelize outside of current environments), but a fairly key part of my mental model is that conditional on current LLMs becoming blocked, there will be massive incentives to point researchers into promising paradigms, meaning that more resources will be pursued on the blockers, compensating for their difficulty.
So yes, it does take time, but maybe not as much as you project.
What links here?
- Noosphere89's comment on Value Generalisation 2: The Missing Hole in AIs’ abilities by Stuart_Armstrong (30 Jul 2026 14:08 UTC; 5 points)

Noosphere89 28 Jul 2026 13:54 UTC
−6 points
0
in reply to: Carringtone Kinyanjui’s comment on: You (Yes, You) Need A February 2020 Checklist for AI Policy
For me, a big part of the reason is that it went off topic immediately in an area that’s still charged by politics, so I didn’t want the discussion to centralize around something that would be predictably useless but very dramatic/distracting.
In essence, I’m trying to cut short a likely demon thread around COVID.

Noosphere89 28 Jul 2026 2:24 UTC
6 points
0
in reply to: TsviBT’s comment on: Noosphere89′s Shortform
Alright, here’s my simple syllogism/list:
1. The use-cases of robotics beyond current robots, especially home robotics require handling lots of environments and have varied challenges, so it stress-tests AI generalization.
2. Unlike other domains, there’s no unexploited algebraicness/overhang that would allow AIs to be useful for these cases without generalizing, because if there were such a thing, very limited current industrial robots would already have been used. They haven’t, so there’s no room for algebraicness/non-generalization to be an issue.
3. It’s easier to control/exclude training data when the domain is physical, rather than digital due to a multitude of reasons.
4. For these reasons, if current AIs improve on robotic tasks, it is a much stronger signal that they’re generalizing, and therefore is a signal that AGI could come in 10-20 years, and depending on how fast they improve, this might shift to 2-4 years.
In particular, in this case you’re using some concepts here:
they have a positive but subhuman amount the property of effectively and continuously acquire deep knowledge and exploit this knowledge to construct and execute goal-directed plans over long lifetimes and consequence horizons/generalization.
And you’re doing an induction about it. And I’m just going to object that the way you’re using the concept doesn’t support the inference you’re trying to make; or at least, AFAICT you haven’t explained the concept and how you’re making the inference well enough to justify the inference.
So, the way I see Vanessa’s concept here is that it’s pointing to the concept of time horizons as defined by METR, which got introduced to the world by METR in 2025 called METR: Measuring AI Ability to Complete Long Tasks, and there’s a lot to be said here, including it’s problems, but the single most important thing to keep in mind here is that this is a continuous metric, so it’s meaningful to talk about how much the AI can acquire long-term knowledge to exploit it.
Now by itself, I would have viewed it as weak evidence, but combined with other evidence showing that exponential trends do exist in other domains, the fact that there are good general reasons to assume exponential growth in AI capabilities, plus the new robotics data shift me towards a view where the current LLM algebraicness/generalization issues are real, but probably fixable, and do not require throwing out the entire paradigm, and thus the induction/inference I’m making is valid here, or to put it another way the observations I have seen are more likely in worlds where LLMs can continuously acquire deep knowledge to exploit it than in worlds where they aren’t.
*Reads your model on progress to AGI, taking the definition of AGI in the post as given/assumed*
I very strongly disagree with this model of progress:
The first way humanity makes AGI is by combining some set of significant ideas about intelligence. Significant ideas are things like (the ideas of) gradient descent, recombination, probability distributions, universal computation, search, world-optimization. Significant ideas are to a significant extent bottlenecked on great natural philosophers doing great natural philosophy about intelligence, with sequential bottlenecks between many insights.
And instead believe in this model:
Algorithmic progress/good ideas/research taste do genuinely matter, but most people are prone to overestimate how much they matter without compute or data, and the main way they currently matter is by being more-compute/data efficient at larger compute scales, and importantly the current paradigm isn’t irrelevant to building the next one, because you can reuse the compute and data from the old paradigm, and the new paradigms will not reduce the demand for compute.
This means progress to AGI isn’t as serial as you state (though I do expect serialness in a different way through RL, but you can still reuse compute/training data you already bought once you have a better paradigm).
More generally, what has made AI go from being ~completely useless and generalization was fake to current AIs where generalization exists, but is worse than humans and can do real economic work is basically just compute and data (and note that in a RL/synthetic data heavy world, data bottlenecks are folded into compute bottlenecks, so compute is the main input)
Or to put it another way, you seem to treat AGI as though there’s a discrete set of challenges to get there, one after the other to be solved, while I instead treat AGI as a set of challenges which all have continuous metrics, and in particular there’s ways to reuse previous progress, because the load-bearing elements of AGI depend less on a particular paradigm/theory of intelligence and they depend more so on compute.
Put another way, you expect sharp/discontinuous progress towards AGI and a singularity, whereas I think we will smoothly go towards AGI and a singularity (note this isn’t talking about how fast or slow the takeoff is.)
A key reason for this is I think the history of AI progress for the last 10-15 years contributes way more evidence for this question than you seem to think, and more generally based on other evidence like papers on human brain scaling laws, where human brains turned out to have a more efficient scaling law than other animals with larger brains, the fact that our brains need a disproportinately large amount of energy to run unlike many animals, which suggests that the human brain is using much more compute, and finally, the most interesting post by gwern on how the insight generator called the Default Mode Network in our brains is largely powered by lots of compute time at inference/test-time compute, and it imposes an effective tax of 1-2 OOMs of compute in order to produce insights, and is a major differentiator of humans (though primates and rats likely do have it as well, but is much less effective due to worse scaling laws and smaller brains in the first place)
This also solves disagreements with the inputs section of the post.
To simplify the disagreement in one sentence, I notice that the central crux of whether or not the load-bearing AI progress is in compute/dat or conceptual breakthroughs on theories of intelligence keeps explaining a lot (but not all) of our disagreements, and disagreements expectations of how smooth vs discrete progress is in AI explain why I expect very different things around AI progress.

Noosphere89 27 Jul 2026 21:19 UTC
3 points
0
in reply to: robo’s comment on: robo’s Shortform
This is a major reason I think waking up the government/Congress in particular is way more important than trying to wake up the public, beyond the other reasons (namely that it will result in less partisan and stupid policies.)

Noosphere89 27 Jul 2026 17:49 UTC
22 points
−2
on: Noosphere89′s Shortform
A take I’ve come to believe around generalization/sample efficiency is that if we do start to see AIs improving a lot at generalization/sample efficiency, we will see it first in robotics/robotics evals, and we have early signs of robotics/robotics evals starting to improve without robotics-specific data, which is bullish for generalization.
There are a couple of reasons for this:
1. A lot of the proposed use cases for robotics, like industry robots that handle varied environments or home robots require more generalization, especially from other data, unlike what has appeared to be the situation for tests and coding/easily grindable mathematics (and potentially white collar work.) Home robotics are an especially hard test case, because homes are spaces of infinite variation, long tails and constant change. As such, the scope of a home robot cannot be narrow. To create real value, it must perform reliably across open-ended variation with no special setup, and that is a practical test of embodied general intelligence.)
2. I’d say the major reason for this comes down to the fact that in areas where we don’t need much generalization, we already have good enough robots for various tasks, and there’s no overhang for AIs to take and become suddenly good without generalizing to other skills (unlike tests where IQ tests do measure human performance at lots of things somewhat well, or coding/easily grindable mathematics.)
3. One other thing that helps is that it’s much easier to restrain LLM training data around the physical world, partially from the world itself being only partial information, but also the fact that it’s much easier to restrain poorly generalizing AIs from just cheating it with a lot of training data, because it’s easier to avoid training data leakage/easier to bound how much AIs need to generalize. It’s also easier to give them hard problems, unlike benchmarks on the internet, because the model might just not be trained on robotics data (and this is feasible), or if it is done, it’s much easier to only give them training data on specific tasks and have a genuinely secure holdout test set, unlike pre-training where you have to give AIs more or less the entire internet, so you can’t train them on specific tasks and have a holdout test set, because the training data might leak information on how to solve the test set, and you have no way of controlling this in the digital world, but you can control data much more effectively in the physical world.
4. Given that generalization is a current bottleneck for robotics, this means that foundation models have to increase their generality of intelligence in order to improve on the tasks.
Now for the part about robotics improving, the current best sources are ACT-2 preview, where pre-training data (that isn’t robotics specific) improved out of domain performance dramatically (from ~15% to 99.1%) on folding laundry, and they have strict standards to ensure that generalization is the explanation for their improvements. The other one is Opus 5, which had also improved a lot on controlling robots at different tasks, namely stacking bowls and holding a fork (which is difficult for robots to grasp due to being thin and flat), purely through the power of it’s mind with zero robotics data.
If I’m right that robotics evals are the main place to look for when assessing LLM generalization/sample efficiency abilities, this looks a lot to me like pre-training is genuinely giving LLMs generalization, and Vanessa Kosoy asked a question about about whether LLMs have a positive amount but are worse than humans at the ability to effectively and continuously acquire deep knowledge and exploit this knowledge to construct and execute goal-directed plans over long lifetimes and consequence horizons, or have zero of the ability and are only doing it because pre-training is simply just getting human algorithms from the internet and once the internet is exhausted, AIs will stop getting better, and now the answer for 2026 LLMs is yes, they have a positive but subhuman amount the property of effectively and continuously acquire deep knowledge and exploit this knowledge to construct and execute goal-directed plans over long lifetimes and consequence horizons/generalization.
Some other implications of “robotics/robotics evals are where we will see generalization ability/sample efficiency, if it appears in AIs”:
1. It means the LLM paradigm is not completely off the mark, even if it requires major changes like continual learning/neuralese recurrence and memory, and there’s a real sense in which current LLMs generalize (Contra Tsvi or Thane Ruthenis’s hypotheses on LLMs)
2. It means that 10-20 year timelines are quite likely, even if the very fast scale up of the 2020s fails to produce AGI for whatever reason, and this means we need to be prepared for AGI/ASI to come in our lifetimes.
3. Relatedly, this also means there’s a plausible path for LLM researchers to work on to improve generalization.
4. Depending on how much RL was used for Opus 5, this might be a real “GPT-3 moment for RL”, where it generalized from lots of RL environments to new problems, demonstrating that RL can actually generalize beyond the trained tasks.
5. Finally, this might be able to replace the METR plots, assuming a time horizon equivalent can be made for physical tasks/robotics, and with much less worry about external validity/measurement problems.
Are there any takes from LLM skeptics like @TsviBT or @Thane Ruthenis about the idea of using robotics evals/robotics progress to predict generalization rather than using current benchmarks?
What links here?
- Noosphere89's comment on Value Generalisation 2: The Missing Hole in AIs’ abilities by Stuart_Armstrong (30 Jul 2026 14:08 UTC; 5 points)

Noosphere89 26 Jul 2026 14:24 UTC
4 points
0
in reply to: Oliver Sourbut’s comment on: LLMs are (still) mostly powered by imitative learning, not RL
Right, so you think the weightings won’t move sufficiently toward interactive/RL curricula for another few years at least? I agree that would preserve some incentive (not quite a constraint) toward keeping externalised CoT in token space rather than embeddings, at which point there’s at least a reasonable chance of that remaining broadly legible.
Yes, absent an intelligence explosion by 2028/2030 ala AI 2027/2040 assumes (and here I’m not going to debate how likely that happens here)
The other point I want to make though is that neuralese recurrence and memory is I think inevitable in the longer run, and by 2030-2032 at the latest, incentives start pushing ever more towards neuralese recurrence and memory, so I think plans that rely on CoT legibility are still basically doomed in the medium to long-term.
I’m not sure compute vs data is the way I’d frame this; rather I’d say that the ‘fossil fuel’ of crudely curated scraped and compiled data still has some runway(which weakly constrains toward unsupervised pretraining and externalised CoT architectures as we’ve discussed). If data pipelines start getting more synthetic/RL-heavy, creation/processing/learning of that is still compute-constrained.
Fair point, especially since sample-efficient ML models are likely to require way more compute, at least at inference, and quite plausibly training compute too than current methods, so yeah compute is likely still the bottleneck after a short transition from ‘fossil fuel’ pre-training data to RL/synthetic-data heavy (and emphasis intentionally placed on the RL part), so data being the primary bottleneck is only true for a short time.
Which incidentally answers your question here about why the AI Futures model takes the key input to be compute/effective compute, because under the assumption of RL dominating post-training eventually, the creation of effective datasets for various tasks becomes bottlenecked on compute, rather than data itself.
I honestly don’t know what to forecast in terms of reliance on one or other source of data. There’s probably still a lot of room for dedicated curation of non-interactive curricula, or a kind of partially-interactive (but not live RL) approach like DAgger. But interactive environments and RL are easier to scale and turn the crank on once set up.
My current take is that the labs are in fact doing dedicated curation/partially interactive but not live RL approaches for at least some of RLVR, but I’m of the opinion that outside of a few fields where you can trivially parallelize RL environments like easily-grindable math or programming/SWE jobs, this will largely not work because feedback loops are long and the data is essentially non-stationary/always changing, so you cannot forgo online RL.
So it does work in some case, but it doesn’t really work nearly as well as the labs need, and this is why I expect them to go for live RL approaches.
Also as you say, interactive environments/RL are easier to scale and turn the crank on, so incentives favor online RL even if it isn’t necessary.

Noosphere89 25 Jul 2026 12:22 UTC
2 points
0
in reply to: Oliver Sourbut’s comment on: LLMs are (still) mostly powered by imitative learning, not RL
Pretraining-cost-dominated pipelines strongly incentivise recurrence-limited (e.g. transformer) architectures, which almost by (accidental) design enforce a certain amount of externalised reasoning and therefore (fallible, but real) CoT legibility. As that pretraining dominance in cost dwindles, that architectural incentive diminishes. This is independent of the relative contribution of each stage to the actual competences of the resulting model.
This is a reason I remain reasonably optimistic about AI architectures over the next 2-4 years (absent an intelligence explosion) not going towards neuralese recurrence and memory, as I do think that compute is still a limiting factor, rather than data, and pre-training compute still mostly dominates, but over the next 2-4 years, I expect data to be a bottleneck rather than compute (absent an intelligence explosion), as we finally run out of pre-training data and have to start going more towards RL environments/synthetic data, meaning sample efficiency becomes much more of a relevant question than now, and the fact that you have to use way, way more RL environments biases you more towards neuralese recurrence and memory because you can’t parallelize RL nearly as much as pre-training (except in a very limited number of environments, which just so happens to be the environments that currently make labs an immense amount of money like coding and easily-grindable mathematics), unlike pre-training.

Noosphere89 24 Jul 2026 14:00 UTC
3 points
1
on: [Linkpost] Thoughts on the Recent OpenAI Hack
Beyond that, I encourage all readers here to read more on the dangers of AI, and talk about them publicly. Some people might be drawn to switching careers, or otherwise starting projects that are good for the world, advocacy and otherwise. But even beyond the immediate benefits, I’m a strong believer in the power of public discourse to help with sensemaking and common understanding. Talk is far from sufficient for the challenges ahead of us, but is probably necessary. It’d be frankly embarrassing if we all sleepwalk to disaster without even bothering to make public our concerns.
I agree that we do need to talk about the concerns, but I have a very different model of how to talk about this, and a big reason for this is I basically don’t believe in the power of public discourse to help with sense-making/common understanding, and a large portion of why this happens is that the public has a conspiratorial mindset in the way Rob Bensinger explains, and genuinely believe that AI is pure hype/fakery, no matter what real-world achievements exist (this has been unfortunately well shown by the warning shot of Huggingface being hacked by OpenAI.)
The main update that we should make is that it’s possible to make Congress/the elites sane on this topic, and we don’t need our cause to be popular to get good laws (which is good here.)
So we absolutely do need to get political will, but we generally should target Congress/The Presidency first, rather than the general public.

Noosphere89 23 Jul 2026 19:23 UTC
2 points
0
in reply to: hillz’s comment on: We should push for no-fault liability for actions taken by AI
Fun, off-topic fact is that the corollary isn’t actually right.
Right now, AIs are deployed in a paradigm where the neurons have been frozen once they get externally deployed, and AI weights stop updating after a very short time compared to humans (and the reason this works is mostly downstream of amortization being much easier and less costly to do digitally than biologically.)
This could absolutely happen in the future, and frontier labs are seeing it as the next big research frontier, but lets not get ahead of ourselves.

Noosphere89

Par­alleliza­tion con­straints could de­lay a tech­nolog­i­cal sin­gu­lar­ity [Linkpost]

Parallelization constraints could delay a technological singularity [Linkpost]