To be clear, I don’t think there is any trivial way to prove the hash released in 2024 corresponds to this potentially 2-year-later revised and modified version, unless @gwern himself confirms of course. My line of reasoning is purely speculation, since he released this with seemingly very little fanfare on gwernnet/twitter/substack before posting on LW and its fun to try and guess the hidden motivations.
Relatedly, since this is in fact, at its core, a very good sounding idea, and gwern has been somewhat prescient previously about pointing out the early signs of both pretrain-scaling->alexnet and test-time-compute-scaling->4chan, it does seem strange the grandfather comment was initially downvoted so heavily for considering the capability acceleration implications (which again ties into my motivational curiosity). Personally, I think a compute overhang is to be avoided, so I’m all for the open dissemination of ideas - but then again gwern does seem to like to play some fun 5d games from time to time.
Paragox
I find this a little strange to drop without context, given how 2024-centric it is. I fully claim fallibility in my interpretation, but I feel this was largely written post-gpt4 in 2024, with small updates circa 2025 and even smaller c. 2026. I follow your posts a bit more zealously than even the average LW reader, and still I’m left a bit confused about what this is, exactly. My best guess is along the lines of: publicly releasing something you had previously hashed (eg. this is in fact that 3rd hash) originally with the intent of proving your novel, prescient authorship? But now with the potential bragging rights dwindling away, your utility preference has swapped to at least getting the ideas / article out into the public LLM training set (perhaps on the eve of scaled RSI being seriously attempted), vs continuing to bank on when the frontier labs of the ensuing two years would surely get around to figuring it out themselves?
On its own merits, this an incredible collection of insights, so regardless I thank you for finally sharing. But it seems rather strange to not acknowledge the apparent shortcomings the 2024 framing brings, or the continual awkwardness in not referencing or building core arguments off newer papers, models, knowledge—the vastly expanded frontier since 2024! And this possibly leaves a cynical conclusion, that this is not your actual present, frontier-informed thesis—perhaps that is only to be revealed 2 years from now?
Apologetically, I concede this could be the real deal, and the lack of updates is largely mundane, ex. due to the buttoning up of public frontier research, perhaps lightly salted with a pinch of sino-lab disdain? The core ideas still stand: commercialization obsessed US labs have done nothing but shallowly scale the same old tired transformer, Chinchilla-esque-thinking reigns supreme and drowns inference-shaped nets in tokens, no one has gotten anywhere nearer (ideologically) to densely training 10T on 100M tokens, etc. etc.
I don’t know if you are referencing it, but your point about 5.2 midtraining failing and being fixed in 5.3/5.4 lines up pretty much exactly with what the paywalled information article “OpenAI Developing ‘Garlic’ Model to Counter Google’s Recent Gains” said back in December, discussing “shallotpeat” being a new pretrain method (5.2), that had some issues and so was being currently refined into a model “garlic” (5.3/5.4) that could properly take on google, with the lessons learned being used as they started creation of a newer, bigger model (5.5).
5.2 being functionally complete, with 5.3 in post training, and pretraining in progress for 5.5 seems to match up very nice with the Dec 2 information article post date and subsequent release dates of 5.2 − 5.5. It is a bit muddied since OAI twitter “vagueposted” many garlic-themed memes around 5.2 release, but the timeline of 5.2 being garlic (still in post training in Dec 2nd, released Dec 11th) doesn’t make any sense, while shallotpeat being complete before Dec 2 and undergoing safety/deployment testing through Dec 10th seems much more reasonable.
If you don’t have have access to the article you can still get all the relevant snippets reposted on twitter, or resummarized by other outlets. There is also an even earlier information article that has the first mentions of shallotpeat (which was itself supposedly a refinement of the first, possibly completely failed internal pretrain attempt).
Relatedly, on Nov 11th Dwarkesh podcast, Illya made a reference to the rumors of a new pretraining method used for gemini 3.0, which is likely what was similarly used in the shallotpeat+ base models.
Catching up on your posts, just wanted to pop in to say that, at the risk of not realizing you already understand this, I think you are generally missing that the major point of the upcoming model scale-ups IS solely about scaling up the active parameters, which only became feasible once enough NVL72/TPUv7 etc existed internally to RL that many active params at scale. (Big total parameters with small active have existed for a while, since large base training + small active RL is not nearly as dependent on scale-up/world size, as we affectionally borrow Dylan’s terminology). So we can expect Mythos to actually have the ~5x increase in active count that corresponds to the ~5x cost, with the delayed availability of hardware to actually enable public availability matching up nicely with the masterful Glasswing dog & pony show.
Then, I think I’ve commented this to you before, but really I think you just somewhat continually underestimate the volume of newer hardware labs possess and deploy, and then underappreciate the insane gap in usage between the mass-offerings/free models vs paid/flagship variants, and how correspondingly little the ratio of, say, NVL72 Blackwell:NVL8 Hopper has to be in the fleet to enable economic deployment of say, 5T+ (with small active) GPT5-thinking or Opus. (which also ties into your fixation on academically-derived opinions on optimal sparsity, vs the competitive burning rubber on road pragmatism that determines what labs actually push)
These are some solid contributions to the seemingly rare critiques of Vladmir’s interpretations of otherwise fantastically aggregated public data, but I must say there is still some interesting jaggedness to it—like I’m surprised you are not aware of Pro being primarily consensus-based (ex. sampling 10x instances per token into consensus via similar mechanism as speculative decoding). Plenty of public reporting on it across the usual suspects. Plus you can more or less implement it yourself with local models and see the exact same performance-improvement and throughput-reduction curves!
To whack-a-mole another, you correctly call out how early both Blackwell was deployed in volume and targeted for deployment, but still manage to miss the forest for the trees on how critical native Q4 is for everything inference: inference-time scaling (ITS) is king, across both RL and deployed-model performance. In other words, inferencing more tokens, faster, annihilates any improvement you get from running Q8+ on Q4 native hardware, and all modern lab deployments + mid/post trainings are running Q4 (apart from the usual caveats) wherever possible. (With total/active size delicately balanced between the ITS, parameter-scaling, and cost trifecta)
Hmm, well yes I suppose my main point is that I personally don’t believe it represents a year of work, but worry the pro-marketing slant of the Anthropic blog post pushes this perspective moreso than a cynic’s breakdown, which I wanted to make sure you were aware of. (There are ofc other analyses than the HN discussed one, see 1, 2)
But I certainly believe it replaced a month+, so as long as it’s now confirmed we are operating under similar information it becomes just a minor quibble over a singular OOM in the interpretation, which seems fine to me.
I don’t disagree overall (Victories continue to come closer to home), but to whackamole a specific datapoint, how much critique of the Claude C Compiler did you read? https://harshanu.space/en/tech/ccc-vs-gcc/ was on top of HN for a day and I believe Zvi reported on it so I’d imagine you have.
But to quickly summarize, CCC only implemented the “easy” parts, directly copied/transpiled the hard parts of that easier part, and most importantly is over 100,000 times slower than an actual compiler! I feel this generally smells of the types of distorted truth that you should expect to omnipresently manifest when literal trillions of capital are at stake.
Might this not just simply due to:
1. Base model training environment is mostly comprised of masked documents ⇐ training context windowand likewise,
2. RL environments contain tasks with CoT rollouts necessarily ⇐ context windowin conjunction with:
3. A likely pyramidal distribution of short:long training material?
LLMs are well documented to struggle outside of the training distribution and the training distribution here doesn’t seem very favorable. (yet)
The system card also contains some juicey regression hidden within the worst-graph-of-all-time in the SWE-Lancer section:
If you can knobble together the will to work through the inane color scheme, it is very interesting to note that while the expected RL-able IC tasks show improvements, the Manager improvements are far less uniform, and particularly o1 (and 4o!) remains the stronger performer vs o3 when weighted by the (ahem, controversial) $$$ based benchmark. And this is all within the technical field of (essentially) system design, with verifiable answers, that the swe-lancer manager benchmark represents.
So the finite set of activated weights is likely getting canabalized further from pre-training generality, towards the increasingly evident fragility of RLed tasks. However, I feel it is also decent evidence towards the perennial question of activated weight size re. o1 vs o3, and that o3 is not yet the model designed to consume the extensive (yet expensive) shared world size capacity of OAI’s shiny new NV72 racks.
As a separate aside, it was amusing setting o3 off to work on re-graphing this data into a sane format, and observing the RL-ed tool-use fragility: it took 50+ tries over 15 minutes of repeatedly failed panning, cropping, and zooming operations for it to diligently work out an accurate data extraction, but work out an extraction it did. Inference scaling in action!
Well yes, but that is just because they are whitelisting it to work with NVLink-72 switches. There is no reason a Hoppper GPU could not interface with NVLink-72 if Nvidia didn’t artificially limit it.
Additionally, by saying
>can’t be repeated even with Rubin Ultra NVL576
I think they are indicating there is something else improving besides world size increases, as this improvement would not exist even in 2 more gpu generations when we get 576 (194 gpus) worth of mono-addressable pooled vram, and the giant world / model-head sizes it will enable.
>Blackwell is a one-time thing that essentially fixes a bug in Ampere/Hopper design (in efficiency for LLM inference)
Wait, I feel I have my ear pretty close to the ground as far as hardware is concerned, and I don’t know what you mean by this?
Supporting 4-bit datatypes within tensor units seems unlikely to be the end of the road, as exponentiation seems most efficient at factor of 3 for many things, and presumably nets will find their eventual optimal equilibrium somewhere around 2 bits/parameter (explicit trinary seems too messy to retrofit on existing gpu paradigms).
Was there some “bug” with the hardware scheduler or some low level memory system, or perhaps an issue with the sparsity implementation that I was unaware off? There were of course general refinements across the board for memory architecture, but nothing I’d consider groundbreaking enough to call it “fixing a bug”.
I reskimmed through hopper/blackwell whitepapers and LLM/DR queried and really not sure what you are referring to. If anything, there appear to be some rough edges introduced with NV-HBI and relying on a virtualized monolithic gpu in code vs the 2x die underlying. Or are you perhaps arguing that going MCM and beating the reticle limit was itself the one time thing?
This is the most moving piece I’ve read since Situational Awareness. Bravo! Emotionally, I was particularly moved by the final two sentences in the “race” ending—hats off to that bittersweet piece of prose. Materially, this is the my favorite holistic amalgamation of competently weighted data sources and arguments woven into a cohesive narrative, and personally has me meaningfully reconsidering some of the more bearish points in my model (like the tractability of RL on non-verifiability: Gwern etc have made individual points, but something about this context-rich presentation really helped me grok the power of iterated-distillation-amplification at scale)
Mimetically, I did find the prose of the “slowdown” ending leans a little too eagerly into the presentation that it will be the smart, wise, heroic, and sexy alignment researchers who come in and save the day, and likely smells a touch too much of self-serving/propagandizing to convince some of those sophisticated yet undecided, but perhaps I cannot disagree with the central point that at the end of it all, how else are we going to survive?
My most bearish argument remains, however, that the real bitterness from Sutton is not that scalable algorithms dominate the non-scalable, but that hardware itself dominates the algorithms. Apropos of Nesov’s models and the chronically underappreciated S-curve reshaping of moore-esque laws: if by 2030 only 100-1000x present compute cannot get you a “datacenter full of geniuses” capable of radically redefining lithography and/or robotics, you are bitterly stuck waiting the decade for standard industrial cycles on post-silicon substrates etc. to bridge the gap. I find the https://ai-2027.com/research/compute-forecast blitheringly optimistic on the capabilities of 100-1000x compute, in both seemingly ignoring the bleak reality of a logarithmic loss curves on Goodhart polluted objectives (10-100x gpt4 compute most likely gave gpt4.5 the predicted loss reduction, but “predict sequences of data from previous data” objective further diverged from “make AGI” objective), whilst not appreciating how much the continually escalating sucking of compute brought by new inference(TTC), RL (IDA?), and training(continual learning?) paradigms will compound the problem.
How to square these claims with the GLP-1 drug consumption you’ve stated in a previous post? I’d wager this powers the pareto majority of your perceived ease of leanness vs average population, and that you are somewhat pointlessly sacrificing the virtues of variety in reverse-pareto fashion.
For funding timelines, I think the main question increasingly becomes: how much of the economical pie could be eaten by narrowly superhuman AI tooling? It doesn’t take hitting an infinity/singularity/fast takeoff for plausible scenarios under this bearish reality to nevertheless squirm through the economy at Cowen-approved diffusion rates and gradually eat insane $$$ worth of value, and therefore, prop up 100b+ buildouts. OAI’s latest
sponsored pysopleak today seems right in line with bullet point numero uno under real world predictions, that they are going to try and push 100 billion market eaters on us whether we, ahem, high taste commentators like it or not.Perhaps I am biased by years of seeing big-numbers-detached-from-reality in FAANG, but I see the centaurized Senior SWE Thane alluded too easily eating up a 100 billion chunk[1] worldwide (at current demand, not even adjusting for the marginal cost of software → size of software market relation!) Did anyone pay attention to the sharp RLable improvements in the O3-in-disguise Deep Research model card, vs O1? We aren’t getting the singularity, yes, but scaling RL on every verifiable code PR in existence (plus 10^? of synthetic copies) seems increasingly likely to get us the junior/mid level API (I hesitate to call it agent), that will write superhuman commits for the ~90% of PRs that have well-defined and/or explicitly testable objectives. Perhaps then we will finally start seeing some of that productivity 10xing that Thane is presently and correctly skeptical off; only Senior+ need apply of course.
(Side note: in the vein of documenting predictions, I currently predict that in the big tech market, at-scale Junior hiring is on its waning and perhaps penultimate cycle, with senior and especially staff compensation likewise soon skyrocketing as every ~1 mil/year USD quartet of supporting Juniors is replaced with a 300k/year Claude Pioneer subscription straight into an L6′s hands.)
I think the main danger is race-to-bottom dynamics and commoditization self-cannibalizing sufficient funding before it could plausibly take off to a N+1 paradigm, with all the requisite scaling in tow.- ^
Amazon alone has in the tens of thousands of US-based L4/Junior engineers, which with TC averaging ~160k * ~1.4 all-in cost of 225k a head, gives a solid 2 billion+ just from this one company, in one country, for one level of one job category.
- ^
My other comment was bearish, but in the bullish direction, I’m surprised Zvi didn’t include any of Gwern’s threads, like this or this, which apropos of Karpathy’s blind test I think have been the best clear examples of superior “taste” or quality from 4.5, and actually swapped my preferences on 4.5 vs 4o when I looked closer.
As text prediction becomes ever-more superhuman, I would actually expect improvements in many domains to become increasingly non-salient, as it takes ever increasing thoughtfulness / language nuance to appreciate the gains.
But back to bearishness, it is unclear to me how much this mode-collapse improvement could just be dominated by postraining improvements instead of the pretraining scaleup. And of course, to wonder how superhuman text prediction improvement will ever pragmatically alleviate the regime’s weaknesses in the many known economical and benchmarked domains, especially if Q-Star fails to generalize much at scale, just like multimodality failed to generalize much at scale before it.We are currently scaling super human predictors of textual, visual, and audio datasets. The datasets themselves, primarily composed of the internet plus increasingly synthetically varied copies, is so generalized and varied that this prediction ability, by default, cannot escape including human-like problem solving and other agentic behaviors, as Janus helped model with Simulacrums some time ago. But as they engorge themselves with increasingly opaque and superhuman heuristics towards that sole goal of predicting the next token, to expect that the intrinsically discovered methods will continue trending towards classically desired agentic and AGI-like behaviors seems naïve. The current convenient lack of a substantial gap between being good at predicting the internet and being good at figuring out a generalized problem will probably dissipate, and Goodhart will rear it’s nasty head as the ever-optimized-for objective diverges ever-further from the actual AGI goal.
Is this actually the case? Not explicitly disagreeing, but just want to point out there is still a niche community who prefers using the oldest available 0314 gpt-4 checkpoint via API, which by the way is still almost the same price as 4.5, hardware improvements notwithstanding, and pretty much the only way to still get access to a model that presumably makes use of the full ~1.8 trillion parameters 4th-gen gpt was trained with.
Speaking of conflation, you see it everywhere in papers: somehow most people now entirely conflate gpt-4 with gpt-4 turbo, which replaced the full gpt-4 on chatgpt very quickly, and forget that there were many complaints back then that the faster (shrinking) model iterations were losing the “big model smell”, despite climbing the benchmarks.
And so when lots of people seem to describe 4.5′s advantages vs 4o as coming down to a “big model smell”, I think it is important to remember 4turbo and later 4o are clearly optimized for speed, price and benchmarks far more than original release gpt-4 was, and comparisons on taste/aesthetics/intangibles may be more fitting when using the original, non-goodharted, full scale gpt-4 model. At the very least, it should fully and properly represent what it looks like to have a clean ~10x less training compute vs 4.5.
Great observation, but I will note that OAI indicates the (hidden) CoT tokens are discarded in-between each new prompt on the o1 APIs, and it is my impression from hours of interacting with the ChatGPT version vs API that it likely retains this API behavior. In other words, the “depth” of the search appears to be reset each prompt, if we assume the model hasn’t learned meaningfully improved CoT from from the standard non-RLed + non-hidden tokens.
So I think it might be inaccurate to consider it as “investing 140s of search”, or rather the implication that extensive or extreme search is the key to guiding the model outside RLHFed rails, but instead that the presence of search at all (i.e. 14s) suffices as the new vector for discovering undesired optima (jailbreaking).
To make my claim more concrete, I believe that you could simply “prompt engineer” your initial prompt with a few close-but-no-cigar examples like the initial search rounds results, and then the model would have a similar probability to emit the copyrighted/undesired text on your first submission/search attempt; that final search round is merely operating on the constraints evident from the failed examples, not any previously “discovered” constraints from previous search rounds.
Thanks to @Petropolitan for alerting me to the discussion spawned here, I had written that deleted comment before bed but then removed it shortly before sleep as I felt it was overly rude/triggered for cunningham-law-esque reasons. I appreciate you replying to it anyways @Vladimir_Nesov! I don’t seem to have any copy / draft anywhere on my profile, but feel free to repost it yourself if you somehow still have a record of it.
Frontier US labs not actually maxxing out the (economically) technically viable deployed size has had strong evidential support lately, with both recent open source releases like GLM-5.2 holding out surprisingly well against benchmaxxing on (relatively) tiny parameter counts when extensively RLVRed even with Z.ai’s comparatively hobbled access to distillation, and then the “leaked” financials giving concrete confirmation of the absurd inference margins labs are taking, which seem hard to napkin math out if considering extensive usage of ex. 4x NVL72 systems per gpt 5.5 instance. The main holdout is still OpenAI’s apparent lack of 5.5 scale vs Mythos/Fable, which still seems to me most obviously occam’s razored away by Anthropic having TPUs at scale and OpenAI not, implying Mythos/Fable is much larger that what fits in NVL72, implying much larger than 10T total..…
I still maintain that the labs are obviously pretraining and even RLVRing far larger model sizes internally, to then distill from, where supposed pretraining dealbreakers like naïve “bubbling” are way overblown on modern asynchronous training clusters, and where they will happily eat the suboptimal but non-ruinous economic penalties of extensive inference pipelining for RL rollouts. And even OpenAI has access to enough TPUs for some respectable internal use!
And to tie it all back to puny, jailbreak-cope-distilled 750B glm 5.2, remembering that every single scaling “shaped” curve has been stubbornly log-linear, which means they can be abused in inverse, implying the labs can quite happily scale as large as technically permitted by hardware internally, and then distill down much smaller in deployment as desired while preserving an unintuitively large share of the gains, maintaining a research:deploy spend that’s shrewd in an inference-dominated revenue regime. I think an underappreciated share of frontier lab taste comes down to the fine art of charting the pareto optimal intersection of all the log-linear scaling lines, from parameters to sparsity to data to RLVR TTC to inference TTC to distillation ratio to consensus/speculation ratio.