Very interesting and kinda unsurprising, I guess! I expected that because these skills won’t develop without training data which is hard to come by.
Hope people who still expect jagged capabilities to somehow go away and models to magically generalize from math and software engineering to IRL tasks see this and update
expensive autocannon in the path of a flying explosive
Autocannons are not particularly expensive, when compared with their own targeting systems, and especially when compared with direct energy weapons of similar range and effectiveness. Heavy machine guns are even cheaper (albeit with diminished range), and medium (7.62-mm) MGs are cheaper still.
The link refers to heavy field howitzers which are actually quite costly, although their barrels are not.
This is essentially the same problem as distributing bioweapons from theater-range ballistic missiles: solvable for large industrial states but hard, expensive and requires a lot of testing
Most natural gas distribution networks are vulnerable to overpressurization which might cause city-wide fires and hundreds of casualties, a few terrorists can organize that by physically capturing relevant infrastructure. But for that they need technical competences which they lack, as no contemporary violent non-state actors really reach the sophistication of Aum Shinrikyo.
Your plan, however, requires not only engineering skills but time and money and production facilities. It’s not impossible that another AS-like apocalyptic sect will emerge in the future but it seems unlikely. Hezbollah might have the means to develop capability in the near future (as Yair Halberstadt has rightfully noted below) and it may be a natural outgrowth of their current drone program but they won’t be able to keep the program secret from Israeli intelligence
If you put a dangerous wild animal in a flimsy cubicle, it escapes and causes some damage in the real world, it can’t be held accountable but you will. Same for an AI agent
What happens if instead of using any generations you initialize “mechanically” with just a different number of ending sentences, maybe from 1 to 5? Or even just several last tokens?
Do you think you could try to retrieve yesterday’s CoT today?
I have stumbled upon the problem that an important file Sonnet 5 fetched at my request as a part of the reasoning process got discarded in the web interface after a dialog turn. I presume these files are likely managed as a part of CoT, and reasoning from previous turns gets purged for reasons of economy as soon as KV cache is deleted
FWIW, Gemma 3 12B doesn’t appear to express similar “thoughts” in a comparable way, although it has ” pretending” and some obscenities on some of the tokens of its response https://www.neuronpedia.org/jlens/cmrcms0kk000004l7cyqf1uu4
From my limited experience on Neuronpedia, “sort of buffer for tokens that might need to be emitted soon” and “mix of potential-next-token buffering as well as something higher-order” are actually quite good descriptions at least for two models presented! I would recommend trying the “free chat” option yourself.
If we are to take the aforementioned “concept vector variance” figures at face value, signal-to-noise ratio of the J-lens is ~1:15 even when the layers are carefully selected by experienced researchers. The ratio for the logit lens has to be worse yet, that’s probably why no one discovered this phenomenon with that instrument earlier.
OTOH, if the labs starts hill-climbing on this in some way, the signal-to-noise ratio might quite likely go down, but on the other, people could come up with more reliable methods of detecting the “workspace” based on this work (e. g., by down-weighting the tokens which actually appear in the later generation or semantically similar to them, or ones detected by something like the Future Lens https://ar5iv.labs.arxiv.org/html/2311.04897 etc.)
You spent practically all of your reply critiquing sections 3 and 4 of the essay which I don’t actually endorse[1], entirely ignoring the question of how a model trained in a containerized RLVR environment will magically generalize to unstructured real-world tasks which dominate the world economy (I looked it up: out of gross world product of ~$126T according to IMF, IT services and software contribute to less than $4T according to Gartner) and all of Patel’s arguments why it won’t
I agree that RLVR does teach deep skills much better than alternatives tried so far, but that fact only leads to increased jaggedness not generalization!
Mythos already got cleared for ~100 American companies and their foreign employees https://www.semafor.com/article/06/27/2026/us-releases-powerful-anthropic-model-mythos-to-some-us-companies
White lists are also very convenient for corrupted officials because they can extort bribes from IT companies desiring access
Dwarkesh Patel tackles exactly this question in the first two sections of his new essay, check it out: https://www.dwarkesh.com/p/the-next-paradigm
Do you have an opinion about the sparsity of frontier models by the way?
Thanks for quite convincing arguments, I’m now more open to lower sparsity for the largest frontier models most extremely limited on data, even though we don’t actually know the size of the pretraining corpora in the frontier labs (it can be very large with multimodal data and code collected from subsidized users!)
Regarding the Farseer paper, their experimental data is indeed collected in a small area where a Chinchilla fit works very well, explaining 99.17% of variance. However, the very first figure in the paper indicates that the Chinchilla prediction is systematically off on both ends of the range, and indeed, when you plot all the residuals from every experiment as opposed to just the average errors, you will see they are highly nonrandom!
I’m not certain how to quantify this (and probably neither are the authors, or they should have included statistical significance), but an LLM recommended a runs test which indicates that signs of residuals around the median are certainly not randomly distributed (p ≈ 2×10⁻⁹, about 6 sigma). My competence in statistics is insufficient for further investigation and I’m not sure that the Farseer formula with its whopping 9 (vs. 5 in Chinchilla) parameters is the correct conclusion from this data, but doesn’t the picture speak for itself that something more complicated than Chinchilla is needed and compute-optimal D/N should indeed be higher at lower loss/higher compute?
I’m not going to reiterate our disagreements on sparsity, just would like to suggest using the more modern Farseer scaling law for extrapolation instead of Chinchilla, as the latter was developed for much lower compute (like around 1e21 FLOPs; Chinchilla-70B itself was 6e23): https://farseer-scaling-law.github.io This research confirms and quantifies the tendency already established before that above Chinchilla level the compute optimality is achieved at significantly larger ratio of tokens to params.
Also note that a MTok of code is worth a couple times less for the loss than a MTok of natural language: https://arxiv.org/html/2510.08702v2
Epoch’s own research on this topic actually seems to correspond very closely to Artificial Analysis https://epoch.ai/data-insights/open-closed-eci-gap
Sonnet CoT summaries stopped appearing for me in web today, could you please check as well?
I think this is mainly a crack down on distillation. OpenAI has shown no reasoning summaries in ChatGPT for about a month, and they are distilled much less if at all