Try the same in the mobile app, perhaps due to lower distillation risks Anthropic still displays summarized reasoning there
Petropolitan
Karma: 475
Could you please explain why this query in particular?
a draw or a loss scores 0
This is quite unusual, does anything change if you write about conventional 0.5 for a draw here instead?
Benchmarks directly distinguish things Fable 5 doesn’t seem to be on track to beat from those Astra 6 is saturating. You need to look for the factors that act differently in these cases.
These things are basically CAD, robotics, computer use and math, but those are likely (in the case of math, certainly!) prioritized by OpenAI when developing RL environments and not prioritized by Anthropic. So the null hypothesis should be that data and not param count determines this differences.
In coding, however, Fable 5.1 and Astra are roughly on par without accounting for the token efficiency (Astra is cheaper) AFAIK
I doubt it’s correct to compare a looped transformer with an ordinary one: Astra could be more “shallow” than it’s conventional, compensating that with “double” or “triple” forward passes. Fable, Opus and Sol, however, are likely to have the same depth-to-active-param ratio as open-weight models (for 60B-110B active, formula 50+0.44*active predicts the number of layers quite well)
To influence the public opinion on AI safety, one has to be considered an expert on ML/AI not just “get attention”.
Shulman is apparently unknown outside of the AI safety community, in fact I had to google him to find out who was that (is it Carl Shulman, or did you mean John Schulman?). Gwern and Greenblatt are only moderately known (I tested on some ML practitioner friends), each frontier lab has over a dozen of people known better than them, and there are many dozens of people in academia and startups who are much more likely to be considered AI experts. Yudkowsky is known but not really taken seriously by the public, his expertise is not well-respected. That only leaves Soares, who in fact has a Wikipedia page and satisfies all the criteria.
So I think your list rather self-defeats your argument
Yeah, kind of “We have built the environment for humans, robots have to be produced in large quantities in order to become cheap due to Wright’s law, humanoid form factor together with AGI provides very large total addressable market because of direct human substitution”.
Nesov’s conservative timeline provides almost a decade to develop the strain-wave gear industry, which is at least plausible, even though that requires very large investments over the years
Technologies can pass this spot of “rapid surge of investment and technical progress” and then flop commercially, think of all the nuclear-powered (civilian) transport theoretically made available by the development of compact PWR in the 1950s and 1960s, supersonic air travel in the 1970s or GOFAI in the 1980s. For more contemporary examples, consider high-temperature semiconductors, fuel-cell cars, VR or thermonuclear fusion, which honestly hasn’t flopped yet but IMHO is going to.
I don’t argue that humanoid robots are necessarily going to flop, it’s more common that a technology can find its limited niche (think of segways or blockchain), but there’s a fundamental hard tradeoff between expensive strain wave reduction gears allowing high torque needed for carrying and moving heavy objects and cheaper planetary gears. Unlike AI, humanoids are essentially a mature technology
I presume you meant humanoid robots not industrial ones and not vacuum cleaners, right?
~85% of the humanoid robots are sold in China, and about 3⁄4 of all the demand in China is from the education and research sectors—including, notably, government-backed training centers which sell the data back to the manufacturers: https://www.ft.com/content/26735a23-315f-47ef-8cf2-6c6ea9713998
>100k workers, most of whom did not know what they were building
JFYI, about 2⁄3 of the people were construction workers so it’s entirely unsurprising they didn’t know they were building the emerging nuclear industry. See https://www.osti.gov/opennet/manhattan-project-history/People/NonTechnicalPersonnel/non-technical-personnel.html and https://blog.nuclearsecrecy.com/2013/11/01/many-people-worked-manhattan-project
And you didn’t even mention all the global healthcare aspects, perhaps because it’s too obvious
The Bitter Lesson-pilled way to look at Nos. 1 and 3 is to realize that the bottleneck is compute not research:
1) In the medium term, the Chinese AI labs can’t win the race because they lack the chips to do so because of sanctions, therefore the question is moot. Theoretically and in the long term, AI alignment is viewed in China as alignment to the party agenda not to some “universal human values” (in a sense how it’s used in the West) which according to the Chinese state ideology do not exist.
3) As long as the top-2 frontier labs have similar capitalization and similar amount of funding available, they will have similar amount of compute and their RSI programs will be bottlenecked by that roughly equally. So as long as that holds, they will be in similar position, and note there’s no concrete “finish line” because the AI capabilities are jagged and will remain so.
I have not thought through how long will it hold though, presumably if one the AI labs folds financially (AI bubble hypothesis etc.) they might decidedly lose the race. But even then their alignment research might get published and used in the leading lab so it won’t be useless
Russia, Belarus, Iran, Venezuela and Myanmar are pariah countries under sanctions, their regimes control very limited amount of compute which they have to prioritize and manage strategically. Romania and Bulgaria are literal EU countries, EU will manage that. The rest are integrated into global economy and will be pressured by the rich countries which are priority targets for cybercrime to shut down illicit inference.
The current list is not informative for the near-future where cybercriminals use GLM-5.3 (possibly slightly finetuned, or perhaps another model with better performance) with a specialized harness at scale as Vast.ai or any other large service will have to crack down on cybercrime as soon as the latter scales up
Which country in particular? In civilized countries local authorities will likely shut such operations down as soon as there is real damage from cybercriminals using it, failed states generally lack infrastructure to sustain them, and pariah states will control such GPUs as a national asset
Hosted where?
Such activity requires feeding lots of cyber-related prompts (including offensive) to powerful models (at least Kimi K3-level). Such a possibility would be very useful to human cybercriminals, why would API providers allow this without KYC?
The reason consumer GPUs and even a single A100 might be scattered around without use is that one can’t inference any useful coding agents on them, making the scenario you suggest impossible.
If they were to appear (very improbable by the end of this year and unlikely even next year), this hardware would become much more valuable both for its legitimate owners and for cybercriminals.
It’s quite obvious that professional (human) cybercriminals with advanced agents inferenced on large clusters (say, 8xH200s) will exploit such hardware much earlier and more effectively than AIs, making such compute basically unavailable to the latter
Mind you, the AI capabilities are very jagged and will remain so. This is BTW entirely missing from the Zvi post, but I don’t have time to argue that, especially since the author doesn’t respond in comments anyway.
For me it looks more like LLMs around 4o cracked the theory of mind and figured out how to persuade people to some extent, likely better than humans writing text—even though humans actually achieve their (our) best level of persuasion in person. But the progress basically stagnated on that, and from the apparent lack of impact of the current AI persuasion we can infer that the current level of capabilities is not world-changing
Which kinds of misalignment might one get from the on-policy distillation with no direct RL on release candidates as practiced by DeepSeek on v4 (e. g., see https://youtu.be/AIRfT41A89s?t=1213 )? How likely would undesirable characteristics of the third and fourth kind be “smuggled” from the RL’d checkpoints via mechanisms similar to subliminal learning? Could the filtering mechanisms prevent that?
Looks like a rich and interesting empirical research direction
Let me clarify what I had in mind by “smart engineering”: sure, 8-high HBF is slower than 8-high HBM but you can supplement part of 12-high HBM with, say, 6-high HBF and get same or better bandwidth for a lower chip price but higher electricity consumption.[1]
As for the prefill, if you can allocate a certain share of hardware to prefill-only, you can minimize HBM on that hardware. However, there are usually ~3x less prefill than decode nodes and in practice even less than a quarter of hardware is “locked” into fixed pools for flexibility, so the economic effect will be limited (maybe that’s why research into heterogeneous hardware generally is so slow).
- ^
There’s also a latency aspect, but experts in the mid-to-late layers can be prefetched early in the forward pass, while those in the early layers can be predicted by the MTP heads on the previous token. Although both approaches come with some bandwidth penalty as the predictions will never be perfect
- ^
I presume you have tried to test Hacker Opus on real user prompts and haven’t found anything unusual, is this correct? What happens if you finetune the checkpoint from before the RL on these seemingly good generations/reasoning traces from Hacker Opus? In particular, is there anything similar to subliminal learning?