Noosphere89

Karma: 4,766

Noosphere89 17 Jun 2026 13:59 UTC
2 points
0
in reply to: gwern’s comment on: Scaling Hypothesis #2: Are Humans Just More Over-Parameterized?
I generally agree with this, though I will make some comments:
because Chinchilla only claims to be compute-optimal
That to be fair is probably targeted because back in the day (and to a lesser extent even now), the amount of data was clearly much larger than the amount of compute, so sample inefficiency was not really a problem, and there’s still a reasonable chance that it doesn’t actually matter for LLMs being transformative in the way we want.
And it could very well be that at least part of the answer to the puzzle is that you cannot train both a compute-optimal and a data-optimal model, and you have to choose one or the other to target.
old Transformers trained with that specific arch
To be fair, companies are very conservative with architecture changes, and again that’s because right now they don’t need to and trying to do it would have serious downside risk for their profitability. That said, it’s definitely a lot less true for research.
It’s a bit like claiming that Fable is impossible because the Kaplan et al 2020 scaling curves on LSTM RNNs show that RNNs scale poorly—like many impossibility proofs, there is less there than meets the eye.
Note Dwarkesh does not claim that it’s impossible, only that it requires (at least) a non-trivial amount of research to solve the issue. If there is an error, it’s that he didn’t realize that there was already research that made progress on the issue.
That said, one reason for this:
I don’t know why Dwarkesh thinks that any extrapolation from Chinchilla tells us anything but a loose lower bound on NN sample-efficiency in alternative scaling regimes, especially unknown ones.
Is partially ignorance of research and partially because companies are understandably conservative about trying new things (for pretty good reason here, because AI is now in the era where you can actually make real profits since AIs are now good enough that they can do real economic work, and this means your products have to be reliable, and new tech is often unreliable).

Noosphere89 17 Jun 2026 3:38 UTC
19 points
0
on: Scaling Hypothesis #2: Are Humans Just More Over-Parameterized?
At least for sample efficiency, using the Chinchilla paper for LLMs, even trying to maximize the number of parameters towards infinity only gets you about 1 OOM less data to reach the same loss, when there’s a 3-6 OOM gap to be explained, and also even if we do believe that human sample efficiency is mostly just the prior, the marginal sample efficiency of models is also a lot worse, and prior differences don’t help to explain that one.
All quotes taken from the Dwarkesh article on sample efficiency here.
The quote about the Chinchilla scaling law meaning we only get 1 OOM of sample efficiency even if we scaled neural nets to an infinite number of neurons.
- The way the scaling law equations work is that parameter and data terms are added to the loss independently. If you have a model that is trained compute optimally, and suppose you ask, well what if I just wanna maximize sample efficiency and use less data—and I’ll throw in as many parameters as it takes to make that happen. With the constants from the Chinchilla scaling laws paper (and the nature of the result wouldn’t change even with different constants), even if you increased the number of parameters by infinity, that would only decrease by a factor of ~10 the amount of data you need in order to keep the same loss. Humans are somewhere between thousands to millions of times more sample efficient than these models. Scaling of current models simply can’t make up for that discrepancy. This really does suggest that humans are on a different scaling curve altogether.
And the quote about sample efficiency for marginal capabilities also being worse in a way that we can’t explain via prior differences:
- Even if it were the case that we can explain away the trillions of tokens required to pretrain a base model as catching up to evolution, it doesn’t explain why the marginal capabilities take so much data—once you have been educated, you don’t need 100 different professors to learn a new programming language, but the AIs (even once pretrained) do.
So your proposal, if it worked would have to have a much more favorable sample efficiency curve with increasing parameters than Chinchilla’s scaling laws.
I’m just noting how much of a big deal it would be if the catapulted NN idea actually worked, because right now the scaling curves of LLM sample efficiency, even if we added more parameters to the NNs, are very terrible, because even an infinite number of parameters has 1 OOM less data required to get the same loss, compared to 3-6 OOMs of sample efficency difference between LLMs and humans.
Also, another useful puzzle is why even after pre-training ends, where the priors should have been baked in, do AIs still require 100 different professors to learn a new programming language, compared to humans who often need 1-3 professors at most, implying a 2 OOM sample efficiency advantage even when the priors baked in by pre-training are taken into account.
One final point about prior differences, or lack thereof:
Many billions of years of evolution is our pre-training, so it’s unfair to compare how little data we see simply within our lifetime to what these cold-started LLMs have to learn from.
- Our genome is 3GB, about 1-2% protein coding. That is just not enough space to store the model parameters that are supposedly pretrained (frontier models are terabytes sized). The closer analogy is probably that evolution has found the right hyperparameters and loss functions (Sidenote: I had an interesting podcast with Adam Marblestone where he argued that the loss functions were the more significant find from evolution), but that the equivalent of parameter training is still happening within lifetime, and is encoded in the map of neural connections in the brain built up over a lifetime.

Noosphere89 14 Jun 2026 0:30 UTC
3 points
0
in reply to: Petropolitan’s comment on: Sympathy for both sides of the egregious misalignment debate
Added the paragraph asked for here.

Noosphere89 13 Jun 2026 16:29 UTC
3 points
1
in reply to: Petropolitan’s comment on: Sympathy for both sides of the egregious misalignment debate
I just wrote a linkpost advocating for the use of more specific terms here, and I wholeheartedly agree with you on the point you are making here, and think the AGI and ASI buzzwords are useless at this point.

Noosphere89 11 Jun 2026 17:23 UTC
3 points
0
in reply to: Petropolitan’s comment on: Dissolving the Deep Learning Sample Efficiency Gap
Another useful section for people to notice, which says that even if the prior is doing most of the work, the marginal sample efficiency for AIs is also very bad, because compared to humans they need 100x more marginal data:
- - Even if it were the case that we can explain away the trillions of tokens required to pretrain a base model as catching up to evolution, it doesn’t explain why the marginal capabilities take so much data—once you have been educated, you don’t need 100 different professors to learn a new programming language, but the AIs (even once pretrained) do.

Noosphere89 10 Jun 2026 2:47 UTC
8 points
7
in reply to: Raymond Douglas’s comment on: The Machines Lack Honour
That’s not quite how I’d put it. What I actually think is more like: “A lot of optimisation went into making human society, and there were structural forces pushing that towards finding good solutions. We shouldn’t be surprised if our attempts to figure out how to handle AI end up converging on things we ‘already discovered’, and we might be able to save a lot of bother by skipping ahead.” This is kind of what I was trying to get at with the ‘utilitarians rediscovering honour’ point.
A potential crux is that I think the structural reasons why current society is good by our values fundamentally disappears with AI, and critically whether or not it’s corrigible don’t matter here.
Specifically, the reasons are a combination of humans being necessary to run the economy + better economic performance requires giving citizens most of their selfish wants + democracy turning out to be better for economies and wars than autocracies.
Ai threatens all three, due to humans becoming less necessary and thus you no longer have to give humans what they want in order for the economy to do well, and democracy in the AI era will start to worsen in performance compared to autocracy because UBI + almost everyone being out of a job means that perpetual unrest similar to 2020 America will worsen efficiency/economic growth dramatically compared to machine autocracies which don’t have to deal with unrest.
And critically, it doesn’t matter whether or not the AI is corrigible or value-aligned, it only matters that due to AI, incentives to give humans what they want is fundamentally weaker, meaning that some amount of intrinisic value-alignment is necessary if humans are to survive with anything like a decent life.

Noosphere89 7 Jun 2026 19:05 UTC
5 points
−6
in reply to: Mo Putera’s comment on: Mo Putera’s Shortform
I agree alignment isn’t nearly as important as people thought, but for a different reason.
In general, one update I’ve made is that the vulnerable world hypothesis is mostly just wrong, except in narrow scenarios, and that’s largely due to me believing that defense against even the worst pathogens possible is actually super cheap, with Andrew Snyder-Beattie talking about it here.
More generally, I’ve updated towards a world model where easy to acquire stuff that is an existential risk either do not exist, or is easy to defend against, meaning the vulnerable world hypothesis isn’t correct.
The actual general reason alignment/corrigibility is less important is because contra some hypotheses, the value of a future controlled by AIs vs a future controlled by AI enhanced humans is not large, and realistically the difference is negligible, meaning moral trade (including acausal trade) and space governance becomes way more important than alignment/corrigibility.

Noosphere89 6 Jun 2026 15:32 UTC
2 points
0
in reply to: nostalgebraist’s comment on: nostalgebraist’s Shortform
Yeah, this has been probably the single biggest update I’ve made over the past couple of years, and yeah the trendline of training compute increasing is basically my first-pass guess on why METR’s trend has been as stable as it is.
And yeah, the plot on here basically validates the story where training compute is by far the most important determinant of AI progress, with everything else like algorithms/data contributing a massively smaller share of improvements.

Noosphere89 2 Jun 2026 17:55 UTC
5 points
0
in reply to: AnthonyC’s comment on: Tech I’m skeptical of and why
My two picks for the category are reversible computers and space industries (in the long run).
For space industries like data centers, the logic of the post mostly checks out assuming you don’t have AIs that can fully automate the industries needed for space industries to be big, and human employment is optional, but if they do exist, then space becomes profitable because it’s way, way cheaper to send robotics/AIs up into space when you don’t need to support anything like livable conditions for standard humans, and the rewards of creating megastructures are much bigger (the main effect of megastructures in my mind is that they force a jump of 10-20 OOMs or more of compute power, meaning far, far more fields are amenable to full simulation without requiring empirical evidence, dramatically improving data efficiency of AI, for one example).
Harsimony’s post on the end of semiconductors and what comes next does sort of imply that he’d be convinced that reversible computers have a use case if space compute was more useful (to oversimplify things), so the space case on it’s own is probably enough, but I do have two points to make on computing tech, one general and one specialized to reversible computing specifically.
On reversible computers, I agree that you can’t make energy dissipation as heat go to 0, and that there’s a minimum energy calculated in this paper for all computing methods, which is likely much lower than classical, but not enough to make reversible computers have arbitrarily low energy at constant speed of computation.
That said, even just providing multiple OOMs worth of energy efficiency is enough, which I think is still likely inside the realm of possibility.
More general point is that I interpret the evidence for unconventional paradigms not mattering is happening in large part because we don’t need to do it, since chips already can get multiple OOMs of energy efficiency and we don’t need to invest in new approaches that soon, rather than fundamental limitations on unconventional computing.
Put another way, the demand for unconventional computing is low largely because there is no need to pick new paradigms to increase efficiency, not because they wouldn’t be valuable if successful.

Noosphere89 28 May 2026 16:37 UTC
1 point
−1
in reply to: Anthony DiGiovanni’s comment on: Probabilities are not the right concept
I should have qualified this, because the reason why probabilities are (fully) caring measures in the limit of optimal Bayesian reasoning is because at that point, you have enough control over the probabilities/models that you can shift the probabilities of certain events happening arbitrarily, or equivalently editing the model is akin to editing reality/moving to a different one, and thus there is no non-completely arbitrary way to set the probability.
For your example, the reason why we can have subjective beliefs that aren’t completely arbitrary is that no one can yet control the Sun’s rise, but assuming AI progress continues, this will likely happen and turn the proposition’s probability into a completely arbitrary number, where values determine the entire outcome.
Indeed, it’s not unfair to say that the end result of getting smarter/having more advanced tech is to make more and more of the multiverse your sandbox, where probabilities of outcomes are entirely arbitrary and dependent on values, and this sort of intuition being made into a workable formalism in the limit cases of optimal Bayesian reasoning is basically the genesis of UDT.
Of course, the big question is whether we can get UDT or a modified alternative to work in the non-limiting cases of the far future.

Noosphere89 26 May 2026 18:19 UTC
9 points
4
in reply to: Vladimir_Nesov’s comment on: Vladimir_Nesov’s Shortform
I’d call this sort of scenario “limping to the Singularity”, and this has been my median scenario for a while, with the caveat that my median takeoff speed is 3-5 years, with doubling times (initially) on the order of a year for the economy.

Noosphere89 25 May 2026 14:47 UTC
2 points
0
in reply to: Oliver Sourbut’s comment on: A (Slightly) Mechanistic Theory for Exponentially Increasing AI Time Horizons?
I’d also maybe disagree with even 1, but this is probably going to depend on how much RL is required. If it’s say 5-20% of compute costs, I’d believe it, but if it’s closer to 50%+, I absolutely think Anthropic has not done this yet.
However I want to flag that most compute costs being RL costs is at least reasonably plausible as a future, especially if continual learning/neuralese brings huge capabilities gains (because in this case you’d constantly want to add more RL tasks to update the weights constantly).
When I’m discussing the upshot of RL and proactive data collection on task success, I’m talking about ‘relevance-weighted’ data size (3) - the fact that self-supervised pretraining didn’t get ‘all the way’ suggests there’s not enough signal to cover enough relevant subtasks, and this is where RL can be ‘heavy’ in that sense.
Directionally agree, but I’d say that we should just wait until 2030, because I do expect pre-trained models to get more capable as well, seperate from the RL part, but I also have updated in the direction that self-supervised pre-training will not go all the way either, at least without megastructure levels of compute and data investments.

Noosphere89 24 May 2026 19:32 UTC
7 points
0
in reply to: Oliver Sourbut’s comment on: A (Slightly) Mechanistic Theory for Exponentially Increasing AI Time Horizons?
Keep in mind here that since RL is less efficient information wise than pre-training (1,000x-1,000,000x less efficient) and RL/inference becomes more expensive the larger the model is, I’d say that basically every AI model, including maybe Mythos is RL light in the sense that RL contributes a lot less of the capabilities compared to pre-training.
But I think you have identified a plausible hole, and this is the case where a larger model makes RL more sample efficient, and this sample efficiency advantage leads to a Jevons scenario where more RL is used. I currently don’t think this is likely for cost reasons, but this is absolutely a very plausible future to keep in mind.

Noosphere89 24 May 2026 18:24 UTC
3 points
−1
on: A (Slightly) Mechanistic Theory for Exponentially Increasing AI Time Horizons?
I should probably say that the Weibull distribution is almost as good of a fit, with the hazard rate parameter called K being less than 1 near-uniformly for AIs, meaning their hazard rates decline over time (but don’t decline as sharply as humans) in Hazard Rates for AI Agents Decline as a Task Goes On, which I think provides evidence that AIs are both worse than humans at generalizing and also evidence that the LLM paradigm isn’t completely off the mark.
On this:
For some types of activity, developers are probably ‘running out’ of raw example data to scrape from the internet. The era of mostly-pretraining is over.
I’d argue that this is probably wrong, based on AI models like Mythos which apparently chose to mostly just scale up parameters, but I agree with something close to this, in that high-quality pre-training data will probably run out by 2028, or in optimistic timelines 2030, around the same time that the immense compute scale up has to slow down to the trend of hardware (at least without new fab construction, which could very well happen).

Some commentaries project that, once AI can autonomously do software and machine learning work reliably, it will thereafter enter a ‘recursive self-improvement’ phase and rapidly colonise all capabilities.
Probably the best public case for this (for now) is the paper called When Does Automating AI Research Produce Explosive Growth? Feedback Loops in Innovation Networks, but I do directionally agree that this probably doesn’t happen and it’s worth preparing for worlds where we don’t get a software intelligence explosion, as quite a few things change in this scenario.

Noosphere89 24 May 2026 2:44 UTC
3 points
−8
on: Probabilities are not the right concept
In the general case, I ultimately think probabilities are caring measures, or to put it another way this is just another thing UDT got totally right (and the mainstream decision theories got this point very wrong).
The main reason for this is that when we attempt to try to focus on arbitrary worlds/thought experiments, we forget that any prior is just as good as any other based on only objective measures, and priors/probabilities become as arbitrary as values.
I’d say one of the main insights of UDT (and possibly FDT/EDT) is that probabilities are caring measures, not about the states of the worlds in and of themselves.

Noosphere89 22 May 2026 21:02 UTC
8 points
0
on: AI #169: New Knowledge
No, it is not fine, and if you ask an LLM they figure this one out pretty easily. The underemployment rate for recent college graduates (22-27 with a BA) is over 40% on top of that (not even seasonally adjusted) 5.3%, a huge percentage of college graduates can’t find jobs that would justify having gone to college or has a good career path, and the job matching and hiring markets have broken down.
The actual reason for this is almost certainly more mundane, and the basic answer, as I’m sure you know is that the signal of recent college graduates being relatively good basically completely broke down, due to intangibles being weighted more and more compared to stuff like the SAT and ACT tests, and grading basically becoming worthless at most colleges as an indicator of quality due to it becoming more and more difficult to not receive A grades, no matter the actual quality of a student (I’m less sold on AI killing the value proposition of colleges, contra this post mostly because another big reason for schooling/college is that not only do you learn from professors, but also the fact that professors (at least used to be pre-2020) much less sycophantic than modern AIs and college had some level of difficulty, and one of the takeaways of education research is that the most effective ways for people to learn involve the stuff that is difficult for them to do, and can’t be simplified without losing the learning benefits, though of course this use-case is now difficult to incentivize as teacher jobs now got easier).
You’ve covered this back in the Childhood and Education Series #17 and #18, but the reason I’m bringing it up is that it’s almost certainly much more causative of large underemployment rates than AI, at least in it’s current state (To be clear, labour-replacing AI is probably coming at timelines that 10 years ago mainstream society would have scoffed at), but currently it’s way too jagged in it’s capabilities/way too incapable to cause large scale, underemployment/unemployment.

Noosphere89 21 May 2026 17:03 UTC
5 points
0
in reply to: jacquesthibs’s comment on: jacquesthibs’s Shortform
I’d say this less strongly, but agreed on the general trend.
I will say 2 things here:
1. AI training faces very different tradeoffs from human training, but a big one here is that AIs don’t need to be nearly as sample efficient to get good results, and this is so far due to them not currently focusing on robotics, where sample efficiency is for now paramount, and this combined with low latency is probably the single biggest constraint on human evolution. While humans are slower to learn on physical movements than many animals, we are still shockingly sample efficient. Especially in timelines where a software intelligence explosion is in the cards, sample efficiency will matter a lot less. There’s also a more general explanation from Carl Shulman that roughly goes where AI training is massively more compute limited, whereas we can teach models lots of data, while the reverse is true for evolution, which had enough compute to brute-force biology if appropriately directed, but had very limited data to work with.
2. One of my updates on AI progress is that even if this current paradigm stalls out, people will still innovate and compute stocks will grow larger, and that this is enough to make median timelines be in the 2040s. To be clear, I’d be really happy if AGI and then ASI was developed in the 2040s, instead of today, because I’d update towards slower takeoffs and more alignment success/more sanity in general, but by and large one of the updates I’ve made is that the CCF/Bioanchors models were basically tracking the right things, but got the numbers very wrong.

Noosphere89 20 May 2026 20:38 UTC
4 points
0
in reply to: Linch’s comment on: leogao’s Shortform
For the first example, I do provisionally agree that LW was probably not responsible, though we’d need the weights and training data, and these are likely inaccessible now, so will edit.
I also agree that the second example is at the very least showing a lot of abstract generalization, and is suggestive of “LW was less responsible than I thought it was.” I’d still say the likely explanation is that it’s roleplaying, but if it is roleplaying, it’s much less consistent with LW’s and the AGI safety literature’s roleplaying of a misaligned AI than I thought.
Ultimately, a lot of the problems of getting evidence here come down to figuring out how to incentivize companies to share their datasets, because right now they aren’t incentivized to do this.

Noosphere89 19 May 2026 15:34 UTC
4 points
−8
in reply to: anaguma’s comment on: leogao’s Shortform
I would probably have accepted these examples earlier on, but nowadays I am a lot more skeptical, an~~d a lot of that reason is I now think LW is more to blame for the misalignment examples than I used to,~~ due to the Influence Functions paper by Anthropic.
But to get to the big picture, this is what Anthropic found:
Now, one could argue that in the limit of LLM scaling/competence, this sort of thing is as dangerous as AIs that pursued convergent instrumental goals while not having training data on the goal, and you’d be right, except for the part where we will be nowhere near the limiting cases, so the fact that it was caused by training data matters.
Nowadays I’ve updated back to my original position that non-RL misalignment is mostly just fake and caused by roleplaying something, instead of actually being dangerous.

Noosphere89 16 May 2026 16:00 UTC
2 points
0
on: Programmers Should Plan For Lower Pay
I think with the rise of AI, we are getting a more clear answer, and it is that programming, like (some parts of) mathematics was only hard for humans due to human specific limitations, and wasn’t all that hard to become good at in a more absolute sense, because programming is probably the 2nd easiest task to verify that you got a correct output in a lot more domains than expected, behind mathematics (at least some parts of mathematics).
And yeah, this post is becoming really, really relevant. By 2029-2030 (or even earlier if the new RL trend of 102 day doubling times keeps going), programmers should assume that they are at high risk of no longer being able to earn even subsistence wages from programming, and plan to move to a new job/save up right now, and should start treating lower pay as something that is likely coming in the near term.