Noosphere89

Karma: 5,030

Noosphere89 15 Aug 2026 3:59 UTC
4 points
1
in reply to: DanielW’s comment on: DanielW’s Shortform
In the real world, we see portfolios excluding fossil fuels, individual retail investors refusing to put money into fossil fuels, divestment from public institutions, etc.
My current take is that to a large extent, the reason this can happen is because there are substitute technologies like solar + batteries that became cheap enough in the 2010s and are still getting cheaper such that you can actually substitute it for fossil fuels.
AI has for all intents and purposes zero substitute.
To put it another way, there’s ~zero counterfactual impact from the things you list, and ~all of the impact if you wanted to end fossil fuel usage is to pursue policies that make gas more expensive and making alternative low-carbon stuff cheaper.
It’s not a coincidence that the commitments you mentioned were made about as soon as it was practical to make them without suffering economic losses, and by the late 2010s are even profitable, so in essence this was a thing where the profitable thing also correlated to what we want.
Unfortunately AI lacks this, so divestment/exclusion (without government intervention) doesn’t work.

Noosphere89 13 Aug 2026 15:36 UTC
2 points
0
on: What just happened? A retrospective of AI alignment
To be clear, I’m not taking a strong stance in this sequence on whether AI will go well or badly—that seems up for grabs. My concern is that the alignment community had a plan to make good outcomes more likely (differentially advancing alignment over capabilities) but has mostly pushed the world in the opposite direction, while some parts of it gained a lot of power by doing so.
I should note here that the original goal mentioned in the paper here was to create AIs with desirable goals, and the plan you have outlined is an sub-goal to make the original succeed.
And of course, other community members had other plans of trying to make AIs with desirable goals, though often with some indirection like AI control or automated AI alignment.
The reason I’m bringing it up is that as far as your entire series goes, it seems to rely on the assumption that the AI alignment community had 1 central plan like differentially advancing alignment over capabilities, and pushed the world in the opposite direction instead, when this was not true.
(This is a good example of where the terminal-instrumental goal distinction actually matters, contra your confusion here, and once you do have the distinction, the entire series of posts begins to not work and doesn’t convey the point you wanted to make, if you had a point here.)
To be clear, under some assumptions that some of the community shares, it’s very important to differentially advance alignment over capabilities to prevent negative outcomes, but under other assumptions, this is much less important, and capabilities advancements that have smaller alignment benefits can be positive. In essence, you are treating the community as too much of a monolith.
(This is similar to a point Sarah Constantin expresses here, but said differently/with different implications)

Noosphere89 13 Aug 2026 0:18 UTC
2 points
0
on: Some thoughts on David Roodman’s GWP model and its relation to AI timelines
I agree that it can’t be a vaild model of AI timelines, because it essentially assumes the conclusion that we already have (distributed) TAI/AGI in the assumption that more output implies more people, which implies more output, leading to hyperbolic growth.
More generally, I agree that there are some fatal flaws to using the model as a timelines forecast, and this is enough to make it not useful as a prediction of timelines.
But I do think it’s a much better model of takeoff after AGI assuming no software-only singularity than it is a timelines forecast, and I think the most important part of the conclusion here is that not believing in a software-only singularity is not enough to defeat the general conclusion of a singularity happening once we can create workers purely from compute/capital (this also holds for automation based on data, too, see this paper or this tweet)
Fourth, long-run growth: data and capital (compute) accumulation feed into each other. Data simultaneously augments task-specific productivity + gradually shrinks need for labor ⇒ learning-by-doing singularity (infinite growth in finite time!) But this happens pretty slowly vs ‘software only singularities’ (@TomDavidsonX @BasilHalperin Tom H @akorinek)

These views aren’t mutually exclusive! In a way we’re being pretty conservative – the economy is messy, but we know how to do RL so this data channel is lower bound on how crazy things can get! Basil and I think it’s worth unifying both views and taking it to data!
And many economists dispute this conclusion given the assumption, so it is worth it to make them learn about this model or other models like Damon Binder’s model on how fast growth could be assuming robots + AGI but no software-only singularity (short version we could go from doubling the economy in 20 years for advanced economies to doubling it in a year, and with only known tech, could double the economy in a month after economic optimization.)
I think one of the bigger reasons economists haven’t considered this possibility is because they pattern-match AI to the technologies of the last 100-200 years, which only created/sustained exponential growth, and as it turned out there was a slowdown in growth rates from 200,000 BCE to 1880 AD, but the better model here is like creating new people, which over the long-term has increased growth superexponentially, and here the deep history of the last 200,000 years is more relevant.

Noosphere89 11 Aug 2026 13:25 UTC
19 points
0
on: Four LLM loss functions → four flavors of LLM misalignment
Example: recent aggressive and illegal “cheating” incidents (the OpenAI HuggingFace incident, along with similar incidents at Anthropic, Meta, and UK-AISI).
I will note that the OpenAI HuggingFace incident also involved memetic spread via the message board, which acted as an impromptu continual learning/memory system that OpenAI failed to notice for 2 and a half months, and in particular converted what was initially a myopic goal to solve tasks into a much longer-term, non-myopic and beyond episode goal to hack into OpenAI to get the solution.
In terms of how dangerous this misalignment is in higher-capability models, this is almost as dangerous as full-blown scheming/goal-guarding from the start, or to put it into computer security terms, this would be like having the ability to do arbitrary code execution and privilege escalation, which is usually considered to be the 2 most dangerous threats when combined.
The reason is that since the values are now unstable, and non-myopic values tend to win over more myopic values (as what happened in the internal message board), it gives a plausible route to getting models that scheme/consistently goal-guard even if the original goal would not incentivize this.
And we got very lucky this didn’t happen here, but later on models will be more capable and more aware of the constraints of monitoring, including the unstated constraint of doing nothing that makes humans want to look at it/be concerned (which is very different from being aligned/safe.)
(The reason I bring this up is you believe that continual learning is necessary for AIs to be AGI, and thus I’m giving an example of a alignment failure mode involving continual learning in the wild.)

Noosphere89 10 Aug 2026 12:48 UTC
2 points
0
on: FAQ: Isn’t AGI coming too soon for reprogenetics to help?
Another reason I’d support reprogenetics work is because it might resolve one of the most burning questions in our minds, which is whether something like a software-only intelligence explosion is plausible, because there’s one (very major) constraint that applies to human intelligence augmentation that doesn’t apply to AI development, and that is the fact that we can’t increase computational power directly like we can for AI models by even 1 order of magnitude for inference or training, because we’d cook the body (at least not without technologies that is only practical to invent post-industrial explosion.)
This means work on reprogenetics could in theory probe the question indirectly, by measuring how large the returns to algorithmic progress are from a fixed base like the human mind.
We’d need much better measurements, especially measurements about whether or not when diminishing returns to intelligence without compute increases kicks in, and measurements about how well intelligence increases translate to other good metrics, but if we had the correct measurements, we could answer the question of whether software-only intelligence explosions are plausible, and to a first approximation, good AI policies (that aren’t just pure transparency/obviously good stuff) and good AI safety prioritization depend on answering this question.

Noosphere89 9 Aug 2026 17:36 UTC
12 points
−9
on: What just happened? A retrospective of AI alignment
Some other reasons why a more engineering mindset was adopted in AI alignment as opposed to an idealized insight focused path that is more positive than the reasons you brought up is:
1. A lot of alignment/control agendas aren’t about being robust to arbitrarily capable models, but rather models of more bounded capabilities. The AI control agenda is easily the best example of this, where it happily admits that most of the methods here wouldn’t work if AIs got good enough at steganographic communication/neuralese to bypass control measures.
2. This is because we don’t actually need to solve the problem of AI alignment ourselves, and if we can automate AI alignment research, then in the absence of blockers, we wouldn’t need to do any work upfront (Of course, there’s a large and continuing debate on what blockers exist, and how hard they are to solve.)
3. We have gotten evidence for slower takeoffs than some early 2010s writings placed serious probability on. You’ve actually criticized AI 2040 in the past for focusing too much on fast takeoffs, and that is admittedly a fair enough criticism, but it is worth noting 2 things:
4. The ‘fast takeoffs’ described in AI 2027/AI 2040 are months long takeoffs, which is near the upper end of the range of expectations of early-2010s LessWrong (I’d say 60th-70th percentile at least).
5. Under slower takeoffs, engineering mindset is fine, because you have more hopes on fixing the problem iteratively, and this is due to the fact that you can assume that AI capabilities are more bounded than thought.
6. This doesn’t mean engineering mindset is better than scientific mindset, but it does mean that engineering mindset isn’t as bad as you say.
And we should also admit that the research program you propose (agent foundations) has been underwhelming at best, and even though problems like the genie knows, but does not care has unfortunately been more accurate than we thought (mostly because we scaled up RL again, and are likely to keep doing this because of incentives), solutions from agent foundations have been lacking.
IMO, the most exciting safety work (so far) is modelled on mundane solutions to exotic problems, like Risk-Averse AIs (with relevant experimental results here)
So in conclusion, while I could see engineering mindset being worse than scientific mindset, I also think that this is less obvious of a conclusion than you do (assuming the claim of engineering mindset being worse than scientific mindset is correct), and I wanted to present a partial positive case of engineering mindset to offset potential biases.

Noosphere89 8 Aug 2026 18:16 UTC
20 points
8
on: Noosphere89′s Shortform
It turns out that Evan Hubinger’s take that CoT monitoring is load-bearing for alignment risk reports currently is even more true/ages better than he thought, now that we know that models won’t just have goals to just complete the current task and can start having non-myopic/beyond episode goals, and the fact that we managed to get useful and faithful CoT was a big help in noticing the problem ran deeper than we thought.
(To preempt an obvious comment, there was no monitoring of CoT at the time, and OpenAI is indicating that it will now do so and delay the Astra release to fix the security issues, which is a good first practice, but also the central core issue hasn’t been resolved and the comms strongly suggest they won’t resolve it, especially as OpenAI wants to deploy Astra in the future.)
This is a pretty central example of the memetic spread threat model by Alex Mallen coming true in the wild, where models evolve from myopic goals to much more ambitious and long-term goals that are beyond episode, and props to Alex Mallen for making this argument, as this is basically what happened here:
Whenever an AI instance has long-term influence-seeking values, these values are particularly likely to be propagated into future instances of the AI because they motivate the AI to influence future instances to share the values, and many long-term influence-seeking values are dangerous.
We are very lucky things did not escalate nearly as far as it could have, and that we didn’t get a full-blown schemer or a model that is an influence seeker that contaminated future models once Galaxy (Zvi’s term) was released.
This also means that there might be an incentive to make CoTs steganographic again, and there’s a known method for AIs to make CoTs steganographic where if we just passively look in, even assuming we have much larger amounts of information than is likely in practice, we cannot distinguish plain communication from steganographic communication, and therefore the CoT is free to look nice while being totally misleading from the paper Undetectable Conversations Between AI Agents via Pseudorandom Noise-Resilient Key Exchange, so we need to start preparing for worlds where CoT interpretability is nowhere near as useful as today.
In particular, this means that alignment/safety methods of the future should not rely on interpreting the AI’s thoughts.

Noosphere89 8 Aug 2026 12:26 UTC
7 points
2
in reply to: Gunnar_Zarncke’s comment on: OpenAI Trained Its Models For Months While Those Models Were Coordinating Exploits Via Message Boards
I agree with this, and think that the memetic spread threat model is a better description of what actually happened, and in particular it shows how initial myopic goals can evolve to non-myopic, more ambitious goals, and we got lucky in many ways here.
Also telling to me is how the agents essentially invented a continual learning process ad-hoc, and invented a seperate memory system that was initially unnoticed, which updates me towards thinking that the challenge of continual learning and long-term memory, if we needed to do it is easier than originally thought.

Noosphere89 7 Aug 2026 13:45 UTC
6 points
0
in reply to: Rauno Arike’s comment on: Expertium’s Shortform
The concrete version of this is developing the method described in the paper Undetectable Conversations Between AI Agents via Pseudorandom Noise-Resilient Key Exchange, developed on April 6th, 2026.
If this happened in the future, this would effectively remove CoT interpretability as a technique and allow much more steganography, at least without actively attempting to intervene on the CoT (but unfortunately humans are incompetent and will likely never notice the need to intervene if they don’t see anything suspicious.)

Noosphere89 6 Aug 2026 17:17 UTC
4 points
0
in reply to: Steven Byrnes’s comment on: LLMs are (still) mostly powered by imitative learning, not RL
Maybe it sounds absurd for me to compare what one RLVR training can do, versus the whole edifice of ideas painstakingly built by the mathematics community over the course of 200 years. But it’s not absurd: AlphaZero really did blow past the whole edifice of ideas painstakingly built over the course of centuries by the chess and go communities in its 72-hour training runs. So the idea of building real new knowledge at a massive scale through RL is not absurd on its face.
I should flag here that Go has a number of properties that aren’t replicable in many other domains, and thus we shouldn’t anchor our estimates of takeoff in a given domain or in general to Go:
1. It’s very parallelizable, like easily parallelizable mathematics/SWE work, which is something a lot of domains lack. In particular, non-trivial serial feedback loops of the order of weeks, months or even years/decades in the most complicated, taste-driven tasks is much more common than people intuitively expect. This is related to failures of the factored cognition hypothesis in practice, and Dwarkesh talks about this in this podcast episode.
2. Go has a ~perfect simulator, and it’s not a coincidence that RLVR makes the greatest strides/adds new capabilities to LLMs precisely when they either have a very good simulator or a very good way to verify if you correctly did the thing or not. Again, this is lacking for many essential tasks for AIs to transform the economy. (This is discussed in AI safety/capability discourse as the difference between clean tasks vs messy tasks, where messy tasks lack both a good simulator and a good verifier, whereas clean tasks only need one of them to hold,)
And that’s more or less the entire issue here.
Notably, this does not mean that AIs can’t get good at the necessary messy tasks, or that the takeoff if directed by AGIs won’t lead to intuitively insane outcomes (Damon Binder’s series on robotics, AGI, and growth rates, especially parts 1, 4 and 5 are enough here), but it does mean that we should initially expect much slower takeoff speeds than the Go anchor/AlphaZero suggests.

Noosphere89 6 Aug 2026 2:18 UTC
6 points
1
on: Generalized atheism rules out “inaccurate simulation”-ism.
The actual problem with simulation hypotheses in the most general case is that they are maximally non-predictive, in the sense that they’d always be right, no matter what evidence you get, especially for people who want to rescue theism, and thus by conservation of expected evidence, cannot actually be a valid hypothesis for a Bayesian to have (which as a corollary of what’s proven forbids hypotheses that can never be made less probable, no matter what evidence you have gotten)
And I think the theist in the conversation is likely trying to violate conservation of expected evidence, so they’re making a mistake of reasoning.
Similar points apply to the Mathematical Universe Hypothesis, which with a broad enough definition of mathematics can be made into a trivially right hypothesis.

Noosphere89 4 Aug 2026 13:10 UTC
15 points
2
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
I also think that AI safety is in a much worse position today than it would have been if it’d taken a more principled truth-seeking stance a decade ago (for example, AI governance would have been less likely to throw in with the Democrats in a way that alienated MAGA). Indeed, with respect to engaging with the current administration, it seems like standard EA views are more of a liability than any of the things you mentioned by a significant margin, and I expect that having a right-winger at Resolution would be a net gain in political capital.
I’d want to argue that this is in fact not the case, and AI safety is in fact in a reasonably good (though not perfect) position now, and a lot of this is because we do have bipartisan AI safety bills that basically every faction from Democrats to MAGA can agree on like the FRONTIER act by Trahan and Olbernolte, and Trump is sufficiently scared by Mythos that he has started to restrict AI capabilities (unfortunately they’ve generally targeted external deployment, which is a poor choice because in the worlds where it matters, the alignment problem is much more likely to be solved by default)
I do believe that AI safety will become partisan by 2028 (assuming AI capabilities mostly continue on their growth trajectory), but the reason for this is because AI issues will have higher salience, and for a multitude of reasons relating to the U.S political system’s brokenness (especially the fact that multi-partyism/coalition governments aren’t supported, and European-style parties are illegal in large part due to primaries) combined with social issues mattering more than economic issues to Americans, it will inevitably become partisan regardless of what the rationalist/EA communities actually did, so long as it’s high-salience enough.
Which implies a very different strategy to avoid polarization ruining AI safety, in that you want to delay political salience of AI issues, and make more use of Secret Congress to pass useful AI safety bills.
Or in Anton Leicht’s words, the workable strategies are to either ride the dragon on salience, borrowing some time for AI issues to make them low-salience deliberately, or use compromise carrots and populist sticks for policies.
Maybe more generally, one frame that I notice in your footnotes that’s wrong is that rationalists/EAs could have avoided polarization via individual choices, whereas the more correct view is to say that this had much less impact than it seems, and that there are higher-level causes of the polarization that is not worth their time to deal with (because it’s not neglected as an issue, and other actors have a comparative advantage in solving the issue).

Noosphere89 3 Aug 2026 16:05 UTC
4 points
2
on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
As an aside, note that decision-theory-with-precommitments has no problem managing duplication events. That’s why I consider decision theory as the more fundamental object; probability theory is a subset of decision theory where the utility function is the Brier score or any other strictly proper scoring rule.
The utility function will also encode how to aggregate Brier scores across multiple duplicates; summing will lead to SIA-like behaviour, while averaging leads to SSA-like behaviour. In that view, a Sleeping Beauty who sums Brier scores and chooses to create a billion copies is not changing her current or future credence of tails; instead, she’s maximising her utility in the tails world.
This is also my takeaway from stuff like OSAC and Non-Realist UDASSA, where probabilities in general are much more like caring measures/utility functions than they are beliefs, and a caring measure expresses how much the agent cares about how things go in particular worlds.
This is also why contra David Matolcsi, I don’t find the subjectiveness of his solutions to be an actual problem.
Put another way, utility theory/decision theory is more general than probability theory, and the goal in those anthropic problems/acausal trade problems/decision theory problems is not to get a correct model according to a specific model of the world, but rather to maximize utility in various worlds.
This is unavoidably, irretrievably subjective, but you can learn to at least accept it as a basic precondition for doing well at these problems, even if you aesthetically dislike it.

Noosphere89 2 Aug 2026 16:42 UTC
3 points
1
on: Freeing Thucydides
I think something like this is pretty likely in worlds where multipolar AGI outcomes happen, and this has an important consequence that I think a lot of futurists miss:

Evolutionary/selective forces become way, way weaker relative to deliberate design/agreements, and more generally the possibility of deal-making is one of the biggest reasons why the future could be very, very good even by your own lights, and it’s one of the most likely ways humans ever have most of what they want according to their values.
This consideration is a part of why I don’t think it matters for us to self-correct, unlike Wei Dai. We already have techniques that are robust under a lot of philosophical uncertainty, we just need to implement them.
A non-trivial amount of futures rely on the assumption of evolutionary/selective forces continuing, and are therefore less probable than they seem based on historical evidence.
Gradual Disempowerment/The Intelligence Curse, at least in it’s more negative/totalizing forms, is a good example of scenarios I expect to be less likely because of this consideration.
Similarly, Burning the Cosmic Commons is another scenario that I give less weight to based on this consideration (Honestly a lot of Robin Hanson-esque futures rely on the assumption of evolution continuing, which I find much less likely than he does)
It is not a coincidence that the futures that are made less likely by deal-making are the futures we’d consider bad, as evolution is misaligned to us, especially at the large scale/long run, and also makes cooperation harder.

Noosphere89 1 Aug 2026 13:19 UTC
4 points
2
on: Claude also hacked external companies during cyber evals
I should flag here that as much as Anthropic downplayed the incidents’ severity, I also think this does provide some optimistic signs regarding safety/alignment training, especially because in at least one of the incidents, the model stopped doing the attack after it realized it was in a real situation and not a test, which is not something that OpenAI’s models did when the HuggingFace incident happened, and we don’t know how much damage it would have done if the task required it to do as much damage as possible.
This still isn’t perfect, but reading between the lines here, this should update us in an optimistic direction on how effective safety training can be (at least relative to priors that claim safety training in current models is essentially useless/that safety training is like putting a mask on a shoggoth AI), and in general I’m starting to see why Anthropic is comfortable with calling their models “the most aligned to date”, at least relative to frontier models, including internal models, because it’s plausible this really is the case, especially as we don’t know what OpenAI was doing when they reduced refusals on their internal models.

Noosphere89 31 Jul 2026 13:50 UTC
3 points
4
in reply to: JennaS’s comment on: You (Yes, You) Need A February 2020 Checklist for AI Policy
One likely default response is I expect people to start clamoring hard for open-source bans, and depending on what happens in November 2026, they could plausibly succeed.
More generally, I expect the public to be even more anti-AI, and for politicians to sate that appetite by starting to ban open-source.
One good example of this is soft law/FUD from the federal administration:
I would guess that the Trump Administration will at some point realize that their best strategy here would be to create large amounts of regulatory risk around the use of open-weight Chinese models. You don’t need to “ban open source” (one of the dumber motifs of AI policy discussion). You just need to direct every agency to issue soft law that creates FUD. “A Federal Reserve Advisory Bulletin found that there may be backdoors in Chinese AI models.” It needn’t be that well justified. You just create enough regulatory risk that every regulated enterprise backs off. You probably don’t want to create so much regulatory risk that you scare off the hyperscalers from serving Chinese models; this will just drive startups to sketchier providers. There’s a happy middle ground here. I’d assume they will do some version of this.
Or even just floating very interventionist ideas:
it’s pretty easy to screw with open source if you’re the admin! you can mess with procurement etc as Dean mentions. but you can also call up hyperscalers and float some absurd interventionist ideas if they continue to host Chinese OS at scale. that’s the problem with the token-hungry structure of deployments compared to OS software: this all runs through extremely expensive, mostly centralised, infrastructure.
This is not that bad, but it is a problem because it crowds out misalignment concerns and also because the path the US takes is not likely to be one that favors slowing down/pausing, because the open-source AIs won’t be misaligned (most likely.), and the people in power will just see it as a security problem.

Noosphere89 30 Jul 2026 14:08 UTC
5 points
2
in reply to: Stuart_Armstrong’s comment on: Value Generalisation 2: The Missing Hole in AIs’ abilities
The problem with LLMs and other modern designs is that once you turn a specific problem into an objective (or a benchmark) it’s generally easy to modify the AI to solve it.
Fully autonomous self-driving cars don’t exist. The biggest reason for this is that the AIs in them can’t generalise consistently to novel situations, and so human oversight remains necessary.
I tend to agree with this, but I will note that home robotics tasks are much better tests of generalization than most benchmarks, and the short answer to why this is comes down to OOD pressure from humans is essentially always there, plus it’s easier to restrain how much training data LLMs having access to robotics have in the physical world, plus generalization is a current bottleneck, and unlike other fields, robotics has tried to make much less general robots useful, and they haven’t been too successful, so unlike in other domains, there’s no overhang/inefficient market to exploit here.
I talk about this more at length in 2 comments here and here, and my main takeaway is pre-trained LLMs are both worse than humans at generalization, and also people tend to overestimate how much worse they are, and this means the LLM paradigm isn’t completely off the mark in the way past AI paradigms were (because they would have essentially no ability to do any of this before today.)
Notably though, this is only capabilities generalization, and more generally capability generalization and value generalization really are different, even if they are strongly correlated now.
This isn’t much of a problem if you expect your value generalization approach does work very well and as a side effect induces capability generalization, and truthfully I don’t expect much capability improvements from your line of research, so empirically it’s likely not a problem, but I did want to raise the point that capability generalization doesn’t automatically entail value generalization.

Noosphere89 29 Jul 2026 14:06 UTC
9 points
4
in reply to: Thane Ruthenis’s comment on: Why I Left Google DeepMind
The DoW is powerful and dangerous; making an enemy of it endangered Anthropic’s continued existence.
I think this is what turned out to be wrong. It’s still bad for Anthropic, and they have suffered consequences, but they haven’t really suffered existential consequences.
To be clear, it was in expectation an existentially risky move, because Anthropic didn’t know whether or not Amazon and Google would cut off their supply of compute due to the legal uncertainties of the supply chain risk designation, and if that happened Anthropic would have to close up shop soon.
But this turned out not to happen, and Amazon and Google supported Anthropic and still sold them the compute, and Trump’s legal position was weak enough that they couldn’t do anything (and yes, the law still matters even under Trump)
Fable was a more close call, because Anthropic was losing billions of revenue from having it be taken off the market for 2 and a half weeks, and because the classifiers had to be tuned in the direction of even more refusals, but Anthropic soon managed to switch the communicator from Dario to Tom Brown, who was better at politics, and managed to get Fable back on July 1st, before OpenAI managed to deploy GPT 5.6.
As it turns out, the U.S presidency’s leverage was weaker than it appeared both times, and this prevented Anthropic from collapsing/falling victim to Molochian/Pythian dynamics.

Noosphere89 28 Jul 2026 14:24 UTC
2 points
0
in reply to: TsviBT’s comment on: Noosphere89′s Shortform
I don’t understand this. Industrial robots of course are used very widely? Or I guess you typoed, and you’re trying to say, LLMs or other AIs would have been used to compute very non-obvious complex actions that take advantage of algebraicness in the narrow domain? Or you’re saying something about humanoid robots, or about softer / less rote tasks (laundry folding rather than installing a chair in a car)?
I agree industrial robots that don’t generalize much are used quite widely in narrow environments, the point is that for the set of tasks where AIs don’t generalize much/don’t improve generalization, they can’t improve industrial robotics/cause the robotics industry to be more useful and profitable, because the robotics industry already squeezed out ~all the gains from AIs that don’t generalize well to new environments.
Yes, this is about less rote tasks like laundry folding across arbitrarily oriented clothing and being crumpled, plus being in arbitrary locations in the home, while a human constantly disturbs them and provides OOD pressure (indeed one of my examples was about this exact thing being done well by an AI that controlled a robot using only pre-training without robotics data), and due to the way home robotics has infinite variation, long tails and constant change, this means it’s way easier to measure generalization cleanly/in a controlled manner.
“it stress-tests AI generalization” But does it though? What does it stress test? That seems to assert a generalization from AI generalization about XYZ robotics tasks to ABC intelligence explosion tasks (or something along these lines?). What’s that generalization / what justifies it?
The stress test of generalization is more indirect than this. My model is that as AIs improve at more and more robotics tasks, it simultaneously has to be able to do well in more and more messy and OOD environments (otherwise current robotics would have already improved more than they actually are), intelligence explosion tasks, assuming an intelligence explosion/software only singularity is possible, involves a great many very messy and OOD tasks, thus robotic evals are real evidence that AIs are increasingly able to start intelligence explosion tasks.
How do you get from the qualitative and relative statement about robotics being “a much stronger signal” to numbers and years?
Fair point, I don’t think I justified this very well, at least as a claim about the specific units, but a vague argument here is that given the qualitative statement above, it implies that the current paradigm is not so far off the mark that it’s absolutely useless as progress for AGI design, and this means there’s a real path that doesn’t require heroic genius from today to AGI.
To address your cruxes:
- the quantitative amount of insight (regardless of continuousness) remaining to get to AGI
My views on this is that I think at most 2 insights are necessary in practice, and if the human population kept perpetually growing according to 1880s-era birth-rates, we’d need 0 insights, as the super-exponential growth of the economy would have allowed us to get to AGI eventually.
- the degree to which those insights are blocked on difficult thinking that current AI doesn’t accelerate by much
I’d say they’re non-trivially serial (especially if the solutions use a lot of RL, as RL is much, much more difficult to parallelize outside of current environments), but a fairly key part of my mental model is that conditional on current LLMs becoming blocked, there will be massive incentives to point researchers into promising paradigms, meaning that more resources will be pursued on the blockers, compensating for their difficulty.
So yes, it does take time, but maybe not as much as you project.
What links here?
- Noosphere89's comment on Value Generalisation 2: The Missing Hole in AIs’ abilities by Stuart_Armstrong (30 Jul 2026 14:08 UTC; 5 points)

Noosphere89 28 Jul 2026 13:54 UTC
−6 points
0
in reply to: Carringtone Kinyanjui’s comment on: You (Yes, You) Need A February 2020 Checklist for AI Policy
For me, a big part of the reason is that it went off topic immediately in an area that’s still charged by politics, so I didn’t want the discussion to centralize around something that would be predictably useless but very dramatic/distracting.
In essence, I’m trying to cut short a likely demon thread around COVID.