Noosphere89

Karma: 4,721

Noosphere89 11 Jun 2026 17:23 UTC
3 points
0
in reply to: Petropolitan’s comment on: Dissolving the Deep Learning Sample Efficiency Gap
Another useful section for people to notice, which says that even if the prior is doing most of the work, the marginal sample efficiency for AIs is also very bad, because compared to humans they need 100x more marginal data:
- - Even if it were the case that we can explain away the trillions of tokens required to pretrain a base model as catching up to evolution, it doesn’t explain why the marginal capabilities take so much data—once you have been educated, you don’t need 100 different professors to learn a new programming language, but the AIs (even once pretrained) do.

Noosphere89 10 Jun 2026 2:47 UTC
7 points
6
in reply to: Raymond Douglas’s comment on: The Machines Lack Honour
That’s not quite how I’d put it. What I actually think is more like: “A lot of optimisation went into making human society, and there were structural forces pushing that towards finding good solutions. We shouldn’t be surprised if our attempts to figure out how to handle AI end up converging on things we ‘already discovered’, and we might be able to save a lot of bother by skipping ahead.” This is kind of what I was trying to get at with the ‘utilitarians rediscovering honour’ point.
A potential crux is that I think the structural reasons why current society is good by our values fundamentally disappears with AI, and critically whether or not it’s corrigible don’t matter here.
Specifically, the reasons are a combination of humans being necessary to run the economy + better economic performance requires giving citizens most of their selfish wants + democracy turning out to be better for economies and wars than autocracies.
Ai threatens all three, due to humans becoming less necessary and thus you no longer have to give humans what they want in order for the economy to do well, and democracy in the AI era will start to worsen in performance compared to autocracy because UBI + almost everyone being out of a job means that perpetual unrest similar to 2020 America will worsen efficiency/economic growth dramatically compared to machine autocracies which don’t have to deal with unrest.
And critically, it doesn’t matter whether or not the AI is corrigible or value-aligned, it only matters that due to AI, incentives to give humans what they want is fundamentally weaker, meaning that some amount of intrinisic value-alignment is necessary if humans are to survive with anything like a decent life.

Noosphere89 7 Jun 2026 19:05 UTC
5 points
−6
in reply to: Mo Putera’s comment on: Mo Putera’s Shortform
I agree alignment isn’t nearly as important as people thought, but for a different reason.
In general, one update I’ve made is that the vulnerable world hypothesis is mostly just wrong, except in narrow scenarios, and that’s largely due to me believing that defense against even the worst pathogens possible is actually super cheap, with Andrew Snyder-Beattie talking about it here.
More generally, I’ve updated towards a world model where easy to acquire stuff that is an existential risk either do not exist, or is easy to defend against, meaning the vulnerable world hypothesis isn’t correct.
The actual general reason alignment/corrigibility is less important is because contra some hypotheses, the value of a future controlled by AIs vs a future controlled by AI enhanced humans is not large, and realistically the difference is negligible, meaning moral trade (including acausal trade) and space governance becomes way more important than alignment/corrigibility.

Noosphere89 6 Jun 2026 15:32 UTC
2 points
0
in reply to: nostalgebraist’s comment on: nostalgebraist’s Shortform
Yeah, this has been probably the single biggest update I’ve made over the past couple of years, and yeah the trendline of training compute increasing is basically my first-pass guess on why METR’s trend has been as stable as it is.
And yeah, the plot on here basically validates the story where training compute is by far the most important determinant of AI progress, with everything else like algorithms/data contributing a massively smaller share of improvements.

Noosphere89 2 Jun 2026 17:55 UTC
5 points
0
in reply to: AnthonyC’s comment on: Tech I’m skeptical of and why
My two picks for the category are reversible computers and space industries (in the long run).
For space industries like data centers, the logic of the post mostly checks out assuming you don’t have AIs that can fully automate the industries needed for space industries to be big, and human employment is optional, but if they do exist, then space becomes profitable because it’s way, way cheaper to send robotics/AIs up into space when you don’t need to support anything like livable conditions for standard humans, and the rewards of creating megastructures are much bigger (the main effect of megastructures in my mind is that they force a jump of 10-20 OOMs or more of compute power, meaning far, far more fields are amenable to full simulation without requiring empirical evidence, dramatically improving data efficiency of AI, for one example).
Harsimony’s post on the end of semiconductors and what comes next does sort of imply that he’d be convinced that reversible computers have a use case if space compute was more useful (to oversimplify things), so the space case on it’s own is probably enough, but I do have two points to make on computing tech, one general and one specialized to reversible computing specifically.
On reversible computers, I agree that you can’t make energy dissipation as heat go to 0, and that there’s a minimum energy calculated in this paper for all computing methods, which is likely much lower than classical, but not enough to make reversible computers have arbitrarily low energy at constant speed of computation.
That said, even just providing multiple OOMs worth of energy efficiency is enough, which I think is still likely inside the realm of possibility.
More general point is that I interpret the evidence for unconventional paradigms not mattering is happening in large part because we don’t need to do it, since chips already can get multiple OOMs of energy efficiency and we don’t need to invest in new approaches that soon, rather than fundamental limitations on unconventional computing.
Put another way, the demand for unconventional computing is low largely because there is no need to pick new paradigms to increase efficiency, not because they wouldn’t be valuable if successful.

Noosphere89 28 May 2026 16:37 UTC
1 point
−1
in reply to: Anthony DiGiovanni’s comment on: Probabilities are not the right concept
I should have qualified this, because the reason why probabilities are (fully) caring measures in the limit of optimal Bayesian reasoning is because at that point, you have enough control over the probabilities/models that you can shift the probabilities of certain events happening arbitrarily, or equivalently editing the model is akin to editing reality/moving to a different one, and thus there is no non-completely arbitrary way to set the probability.
For your example, the reason why we can have subjective beliefs that aren’t completely arbitrary is that no one can yet control the Sun’s rise, but assuming AI progress continues, this will likely happen and turn the proposition’s probability into a completely arbitrary number, where values determine the entire outcome.
Indeed, it’s not unfair to say that the end result of getting smarter/having more advanced tech is to make more and more of the multiverse your sandbox, where probabilities of outcomes are entirely arbitrary and dependent on values, and this sort of intuition being made into a workable formalism in the limit cases of optimal Bayesian reasoning is basically the genesis of UDT.
Of course, the big question is whether we can get UDT or a modified alternative to work in the non-limiting cases of the far future.

Noosphere89 26 May 2026 18:19 UTC
9 points
4
in reply to: Vladimir_Nesov’s comment on: Vladimir_Nesov’s Shortform
I’d call this sort of scenario “limping to the Singularity”, and this has been my median scenario for a while, with the caveat that my median takeoff speed is 3-5 years, with doubling times (initially) on the order of a year for the economy.

Noosphere89 25 May 2026 14:47 UTC
2 points
0
in reply to: Oliver Sourbut’s comment on: A (Slightly) Mechanistic Theory for Exponentially Increasing AI Time Horizons?
I’d also maybe disagree with even 1, but this is probably going to depend on how much RL is required. If it’s say 5-20% of compute costs, I’d believe it, but if it’s closer to 50%+, I absolutely think Anthropic has not done this yet.
However I want to flag that most compute costs being RL costs is at least reasonably plausible as a future, especially if continual learning/neuralese brings huge capabilities gains (because in this case you’d constantly want to add more RL tasks to update the weights constantly).
When I’m discussing the upshot of RL and proactive data collection on task success, I’m talking about ‘relevance-weighted’ data size (3) - the fact that self-supervised pretraining didn’t get ‘all the way’ suggests there’s not enough signal to cover enough relevant subtasks, and this is where RL can be ‘heavy’ in that sense.
Directionally agree, but I’d say that we should just wait until 2030, because I do expect pre-trained models to get more capable as well, seperate from the RL part, but I also have updated in the direction that self-supervised pre-training will not go all the way either, at least without megastructure levels of compute and data investments.

Noosphere89 24 May 2026 19:32 UTC
7 points
0
in reply to: Oliver Sourbut’s comment on: A (Slightly) Mechanistic Theory for Exponentially Increasing AI Time Horizons?
Keep in mind here that since RL is less efficient information wise than pre-training (1,000x-1,000,000x less efficient) and RL/inference becomes more expensive the larger the model is, I’d say that basically every AI model, including maybe Mythos is RL light in the sense that RL contributes a lot less of the capabilities compared to pre-training.
But I think you have identified a plausible hole, and this is the case where a larger model makes RL more sample efficient, and this sample efficiency advantage leads to a Jevons scenario where more RL is used. I currently don’t think this is likely for cost reasons, but this is absolutely a very plausible future to keep in mind.

Noosphere89 24 May 2026 18:24 UTC
3 points
−1
on: A (Slightly) Mechanistic Theory for Exponentially Increasing AI Time Horizons?
I should probably say that the Weibull distribution is almost as good of a fit, with the hazard rate parameter called K being less than 1 near-uniformly for AIs, meaning their hazard rates decline over time (but don’t decline as sharply as humans) in Hazard Rates for AI Agents Decline as a Task Goes On, which I think provides evidence that AIs are both worse than humans at generalizing and also evidence that the LLM paradigm isn’t completely off the mark.
On this:
For some types of activity, developers are probably ‘running out’ of raw example data to scrape from the internet. The era of mostly-pretraining is over.
I’d argue that this is probably wrong, based on AI models like Mythos which apparently chose to mostly just scale up parameters, but I agree with something close to this, in that high-quality pre-training data will probably run out by 2028, or in optimistic timelines 2030, around the same time that the immense compute scale up has to slow down to the trend of hardware (at least without new fab construction, which could very well happen).

Some commentaries project that, once AI can autonomously do software and machine learning work reliably, it will thereafter enter a ‘recursive self-improvement’ phase and rapidly colonise all capabilities.
Probably the best public case for this (for now) is the paper called When Does Automating AI Research Produce Explosive Growth? Feedback Loops in Innovation Networks, but I do directionally agree that this probably doesn’t happen and it’s worth preparing for worlds where we don’t get a software intelligence explosion, as quite a few things change in this scenario.

Noosphere89 24 May 2026 2:44 UTC
3 points
−8
on: Probabilities are not the right concept
In the general case, I ultimately think probabilities are caring measures, or to put it another way this is just another thing UDT got totally right (and the mainstream decision theories got this point very wrong).
The main reason for this is that when we attempt to try to focus on arbitrary worlds/thought experiments, we forget that any prior is just as good as any other based on only objective measures, and priors/probabilities become as arbitrary as values.
I’d say one of the main insights of UDT (and possibly FDT/EDT) is that probabilities are caring measures, not about the states of the worlds in and of themselves.

Noosphere89 22 May 2026 21:02 UTC
8 points
0
on: AI #169: New Knowledge
No, it is not fine, and if you ask an LLM they figure this one out pretty easily. The underemployment rate for recent college graduates (22-27 with a BA) is over 40% on top of that (not even seasonally adjusted) 5.3%, a huge percentage of college graduates can’t find jobs that would justify having gone to college or has a good career path, and the job matching and hiring markets have broken down.
The actual reason for this is almost certainly more mundane, and the basic answer, as I’m sure you know is that the signal of recent college graduates being relatively good basically completely broke down, due to intangibles being weighted more and more compared to stuff like the SAT and ACT tests, and grading basically becoming worthless at most colleges as an indicator of quality due to it becoming more and more difficult to not receive A grades, no matter the actual quality of a student (I’m less sold on AI killing the value proposition of colleges, contra this post mostly because another big reason for schooling/college is that not only do you learn from professors, but also the fact that professors (at least used to be pre-2020) much less sycophantic than modern AIs and college had some level of difficulty, and one of the takeaways of education research is that the most effective ways for people to learn involve the stuff that is difficult for them to do, and can’t be simplified without losing the learning benefits, though of course this use-case is now difficult to incentivize as teacher jobs now got easier).
You’ve covered this back in the Childhood and Education Series #17 and #18, but the reason I’m bringing it up is that it’s almost certainly much more causative of large underemployment rates than AI, at least in it’s current state (To be clear, labour-replacing AI is probably coming at timelines that 10 years ago mainstream society would have scoffed at), but currently it’s way too jagged in it’s capabilities/way too incapable to cause large scale, underemployment/unemployment.

Noosphere89 21 May 2026 17:03 UTC
5 points
0
in reply to: jacquesthibs’s comment on: jacquesthibs’s Shortform
I’d say this less strongly, but agreed on the general trend.
I will say 2 things here:
1. AI training faces very different tradeoffs from human training, but a big one here is that AIs don’t need to be nearly as sample efficient to get good results, and this is so far due to them not currently focusing on robotics, where sample efficiency is for now paramount, and this combined with low latency is probably the single biggest constraint on human evolution. While humans are slower to learn on physical movements than many animals, we are still shockingly sample efficient. Especially in timelines where a software intelligence explosion is in the cards, sample efficiency will matter a lot less. There’s also a more general explanation from Carl Shulman that roughly goes where AI training is massively more compute limited, whereas we can teach models lots of data, while the reverse is true for evolution, which had enough compute to brute-force biology if appropriately directed, but had very limited data to work with.
2. One of my updates on AI progress is that even if this current paradigm stalls out, people will still innovate and compute stocks will grow larger, and that this is enough to make median timelines be in the 2040s. To be clear, I’d be really happy if AGI and then ASI was developed in the 2040s, instead of today, because I’d update towards slower takeoffs and more alignment success/more sanity in general, but by and large one of the updates I’ve made is that the CCF/Bioanchors models were basically tracking the right things, but got the numbers very wrong.

Noosphere89 20 May 2026 20:38 UTC
4 points
0
in reply to: Linch’s comment on: leogao’s Shortform
For the first example, I do provisionally agree that LW was probably not responsible, though we’d need the weights and training data, and these are likely inaccessible now, so will edit.
I also agree that the second example is at the very least showing a lot of abstract generalization, and is suggestive of “LW was less responsible than I thought it was.” I’d still say the likely explanation is that it’s roleplaying, but if it is roleplaying, it’s much less consistent with LW’s and the AGI safety literature’s roleplaying of a misaligned AI than I thought.
Ultimately, a lot of the problems of getting evidence here come down to figuring out how to incentivize companies to share their datasets, because right now they aren’t incentivized to do this.

Noosphere89 19 May 2026 15:34 UTC
4 points
−8
in reply to: anaguma’s comment on: leogao’s Shortform
I would probably have accepted these examples earlier on, but nowadays I am a lot more skeptical, an~~d a lot of that reason is I now think LW is more to blame for the misalignment examples than I used to,~~ due to the Influence Functions paper by Anthropic.
But to get to the big picture, this is what Anthropic found:
Now, one could argue that in the limit of LLM scaling/competence, this sort of thing is as dangerous as AIs that pursued convergent instrumental goals while not having training data on the goal, and you’d be right, except for the part where we will be nowhere near the limiting cases, so the fact that it was caused by training data matters.
Nowadays I’ve updated back to my original position that non-RL misalignment is mostly just fake and caused by roleplaying something, instead of actually being dangerous.

Noosphere89 16 May 2026 16:00 UTC
2 points
0
on: Programmers Should Plan For Lower Pay
I think with the rise of AI, we are getting a more clear answer, and it is that programming, like (some parts of) mathematics was only hard for humans due to human specific limitations, and wasn’t all that hard to become good at in a more absolute sense, because programming is probably the 2nd easiest task to verify that you got a correct output in a lot more domains than expected, behind mathematics (at least some parts of mathematics).
And yeah, this post is becoming really, really relevant. By 2029-2030 (or even earlier if the new RL trend of 102 day doubling times keeps going), programmers should assume that they are at high risk of no longer being able to earn even subsistence wages from programming, and plan to move to a new job/save up right now, and should start treating lower pay as something that is likely coming in the near term.

Noosphere89 13 May 2026 0:07 UTC
2 points
0
on: How useful is the information you get from working inside an AI company?
Notably, open weight models aren’t that far behind closed weight models, despite the disadvantage of being trained with less compute and worse data.
I think this sentence is basically false, despite the preceding sentence being true, and the reasons for that is as Lisan al Gelb explains, closed models being better at inference compute usage + open source focusing much harder on coding/computer use and agentic tasks than closed source, meaning outside those areas open models have a much bigger gap to traverse + open source having much less capability and incentive to not directly train on the benchmark data, essentially having larger data contamination problems (this is more so of a cached belief based on a paper 3-4 years ago + Gavin Leech’s paper on benchmarks being inflated because AI companies and open source fail to remove semantic duplicates of test data, and my suspicion is that frontier labs have much more incentive to address these sorts of problems than open source).
I agree with the view that AI companies don’t have much secret sauce right now, and most of the reason why the frontier AI labs is outperforming open source so hard is mostly down to compute + having stronger incentives to actually not train on the test set.

Noosphere89 11 May 2026 18:56 UTC
4 points
3
on: Anthropic’s strange fixation on hyperstition
A potential crux I believe is driving a lot of arguments around the persona selection model/hyperstitioning, is how much you expect goals to be based on context/data compared to the weights, and in particular if I steelman Anthropic enough, I do get the idea that a lot of their alignment efforts assume that we can put arbitrary data/context even after the weights are frozen and have the AI generalize more or less as we want from that data/context.
More generally, a looming crux/divide that really shows up everywhere in AI debates, especially x-risk debates is whether or not data/context or model weights matters more, and an example of this is Herbie Bradley talking about how data is the cause of most compute efficiency gains, meaning takeoff is slower than people think.
In a way, this is kind of analogous to nature vs nurture/gene vs culture debates over which has more impact in humans.

Noosphere89 6 May 2026 14:16 UTC
5 points
0
in reply to: Petropolitan’s comment on: Irretrievability; or, Murphy’s Curse of Oneshotness upon ASI
This is close to correct, and is the reason why the control agenda is focused around interventions before you catch the AI, because after you catch the AI, the situation becomes easier in hard-to-predict ways.
1 caveat to this is that the AI likely has more tries than just 1 try, but it’s not unlimited, and is plausibly on the order of 10-1000 (though we probably don’t need this many real tries because of proliferation).
But yes, especially in the regime where we need to automate AI safety research, we probably get multiple tries if we can play our cards well, and the AI doesn’t have nearly as many iteration attempts to take over as is often assumed.

Noosphere89 2 May 2026 22:34 UTC
3 points
3
on: Sanity-checking “Incompressible Knowledge Probes”
I would absolutely like a post on the densing law from you, so consider me interested.