Hyperion

Karma: 76

Hyperion 14 Jul 2026 17:46 UTC
11 points
7
in reply to: Thomas Larsen’s comment on: Our response to Séb Krier on Plan A
I don’t think belief in ASI necessarily implies belief in all of:

> (a) extremely fast diffusion and societal transformation, (b) a view that all profits accrue maximally to the labs, (c) that the prescriptions advanced in the essay (like expropriations and forced IP diffusion) have minimal impacts on said profits; (d) the claim that you get explosive GDP growth very soon, (e) that ‘de facto’ nationalisation and profit redistribution through UBI is an optimal response.
Or did you mean some other part of his fourth response?

I also don’t think it requires you believe that no comparative advantage for humans post-ASI exists: positional goods are still a thing, a price premium for human-produced goods is very plausible (and positing that it won’t happen is not a question of capabilities).

I think it’s a common move to claim that people with a different conception of a post-ASI trajectory “don’t believe in” ASI, but when digging into it usually they don’t disagree on raw capabilities, just on what those raw capabilities imply the ASI would be able to do in the world.

Hyperion 8 Jul 2026 12:12 UTC
1 point
0
in reply to: Vladimir_Nesov’s comment on: Vladimir_Nesov’s Shortform
This is good! My main counterarguments:
- On the RLVR side I think you neglect the data bottlenecks:
  - Many tasks in knowledge work, especially those done by senior employees, are deep skills that are in principle learnable via RLVR but have long real-time feedback loops. You can simulate a proxy, but in many cases I think the proxy is quite decorrelated from ground truth. This is one reason why I think we need a continual learning breakthrough, so that models can be deployed into the real-world to “wait out” these long feedback loop tasks and eventually learn from them.
  - Many tasks in knowledge work are separately RLVR-able in theory but have ambiguous resolution, preventing the construction of effective RL envs right now. These are the “hard to verify” or “fuzzy” tasks described in papers like these: https://arxiv.org/abs/2605.06390 . In this case, models likely need to learn representations over time and experience which match human intuition.
  - These two types of tasks are why I expect highly jagged economic effects from AI: as you say engineering-type or highly verifiable tasks don’t have these problems so much, but huge swathes of the economy does, which means the TAM is quite lopsided (as we see today with ~60%+ of new revenue added to AI labs coming from coding in 2026).
  - So overall I prefer a conception of “economy-wide RSI”; we get to a level of capabilities with some breakthroughs in continual learning and forgetting which enable broad deployment of continual learning models into the economy. At this point the speed of further capabilities is governed by data bottlenecks like the types I describe above, as well as compute scale-ups which you’ve analyzed.
    From this point, AGI improves over time almost like a collective intelligence (eg a company) improves over time: increasingly bottlenecked by how fast it can act in the world and integrate the outcomes of those actions.
- Then on ASI breakthroughs: the model you propose is possible, but there should also be some weight on “hardware lock-in”; it’s plausible to me that an ASI breakthrough is of a form which is an inconvenient fit for the increasingly specialized hardware form factor of GPUs in 2030, thus blunting the compute overhang.

Hyperion 8 Jul 2026 11:47 UTC
2 points
1
in reply to: Vladimir_Nesov’s comment on: Vladimir_Nesov’s Shortform
I think this is a key crux in these discussions: my understanding of grader quality is that it’s systemically worse than human judge quality on tasks that current models struggle with, either because the grader is itself often OOD or because tasks often have ambiguous resolution. See CAIS section 3 here: https://safe.ai/blog/significant-increase-in-digital-labor-automation
Do you have a reason for why this might change? Is it just hoping that pretraining scale solves it? In physics, an LLM can call a physics sim or world model; in engineering as you note in another comment we can modularize, but the sim2real gap surely is much worse for, for example, HealthBench Professional tasks: https://cdn.openai.com/dd128428-0184-4e25-b155-3a7686c7d744/HealthBench-Professional.pdf (what would the sim even be?). A very large fraction of the economy is like this, and in theory that becomes a larger fraction as highly legible & verifiable tasks get automated in the coming few years.

Hyperion 25 Feb 2026 4:22 UTC
3 points
0
in reply to: habryka’s comment on: Responsible Scaling Policy v3
That’s fair! I am mostly thinking of the AI safety community and the parts of it interested in cybersecurity. I had a lot of discussions with people around the funding ecosystem/government AI safety-interested people/AI policy thinktanks around a year ago about the merits of attempting SL5 and never heard a mention of Anthropic’s RSP specifically, although it seems plausible it was a contributing factor for decisions to pursue that direction.

Hyperion 25 Feb 2026 4:16 UTC
2 points
0
in reply to: habryka’s comment on: Responsible Scaling Policy v3
If there had been common knowledge (which I think there probably would have in the absence of Anthropic’s RSP and associated recruitment/marketing/comms efforts) that achieving nation-state robust cybersecurity was not achievable unless very drastic actions were taken, I do think this would have caused some people to change strategies substantially.
My sense is most efforts which seem to be aimed at nation-state robust cybersecurity for AI (outside labs) have been driven by things like the RAND report and some theory of change downstream of pieces like Situational Awareness which argue that governments may push for TS/SCI classified AI development at some point (for some applications)—not necessarily downstream of Anthropic’s RSP (I’ve never heard anyone mention it directly in, for example, any discussion around why SL5 security is important).
That said (a) I agree with you that this goal is impractial for general AI development and it was foolish of Anthropic to commit to something close to security against state-backed attacks (b) I have heard critique of Jason Clinton’s PoV from parts of the AI cybersecurity community that, having never worked in an Intelligence Community cybersecurity role, he lacks information that would update him on the difficulties (c) I remain confused why some still consider SL5 security for AI model weights a tractable or important goal in the field.

Hyperion 21 Jan 2026 22:12 UTC
1 point
0
on: The case for AGI safety products
I’m curious for your reasons for choosing PBC. Having looked into it a little, it seems like:
- There are plenty of historical records of normal for-profits doing things which nominally seem to be hurting profits but which get justified by the CEO because “it’s better for us long-term” effectively. Ben & Jerry’s is a well known example. So if you go the normal for-profit route it seems like you have freedom of action to do this. If the concern is the board, you can choose your board members wisely and/or set up an additional “advisory board” with various powers under a normal for-profit too.
- PBCs are materially worse for investors, as I understand it, even if the difference is not large, and so you slightly hurt your ability to raise.

Hyperion 21 Jan 2026 22:06 UTC
14 points
1
on: The case for AGI safety products
If these are close enough, then AGI safety products are a reasonable idea. If not, they’re an actively bad idea because the new incentives pull you in a less impactful direction.
I’ve heard this argument a lot, but I think it has a key failure mode: it assumes incentives are more or less invariant with scale (of org size, staff, cash on hand, etc). As they say in the Valley “most problems are solved with scale”.
Another heuristic common in the startup world but which seems to not occur to people on this forum too much: many successful startups do not start intending to build in their actually successful direction—they pivot at least once.
Together, these facts to me imply a dynamic where building a successful company is both very compatible with AGI safety in the long run, but going about it by trying to backchain from AGI safety seems unlikely to work. Instead, good founders are likely to pivot around a bunch to find PMF. But even if the idea with PMF is not the maximally AGI safe one to build, that doesn’t mean it’s a bad idea! If you build that, then come back a year later with much more capital and a large team to build a second product, then you are in a significantly better position than if you had tried to build for AGI safety from the start.
I think a common failure mode amongst the AI safety startup market is “trying to build the perfect product from the start”; trying to find the narrow path that has both safety and 0 to 1 profitability for a company with no revenue or existing customers.

Hyperion 20 Apr 2025 0:01 UTC
13 points
2
on: Why Should I Assume CCP AGI is Worse Than USG AGI?
There’s also the possibility that a CCP AGI can only happen through being trained on Western data to some extent (i.e., the English language internet) because otherwise they can’t scale data enough. This implies that it would probably be a “Marxism with Chinese characteristics [with American characteristics]” AI since it seems like that just raises the “alignment to CCP values” technical challenge difficulty a lot.

Hyperion 15 Apr 2025 14:58 UTC
7 points
1
in reply to: elifland’s comment on: Thoughts on AI 2027
If so much effort is being focused into AI research capability, I’d actually expect modally Agent-3 to be better than typical OpenBrain employee but completely incapable of replacing almost all employees in other fields. “capabilities are spiky” is a clear current fact about frontier AI, but your scenario seems to underestimate it.

Hyperion 2 Apr 2025 4:40 UTC
1 point
0
in reply to: Neel Nanda’s comment on: Good Research Takes are Not Sufficient for Good Strategic Takes
I suppose I mean influence over politics, policy, or governance (this is very high level since these are all distinct and separable), rather than actually being political necessarily. I do think there are some common skills, but actually being a politician weighs so many other factors more heavily that the strategic skill is not selected on very strongly at all. Being a politician’s advisor, on the other hand...

Yes, it’s a special case, but importantly one that is not evaluated by Brier score or Manifold bucks.

Hyperion 31 Mar 2025 16:38 UTC
1 point
0
in reply to: Neel Nanda’s comment on: Good Research Takes are Not Sufficient for Good Strategic Takes
I guess that’s the main element I didn’t mention: many people on this forum would suggest judging via predictive skill/forecasting success. I think this is an ok heuristic, but of course the long time horizons involved in many strategic questions makes it hard to judge (and Tetlock has documented the problems with forecasting over long time horizons where these questions matter most).
Mostly, the people I think of as having strong strategic skill are closely linked to some political influence (which implicitly requires this skill to effect change) such as attaining a senior govt position, being influential over the Biden EO/export controls, UK govt AI efforts, etc. Alternatively, they are linked to some big major idea in governance or technical safety, often by spotting something missing years before it became relevant.
Often by interacting regularly with good thinkers you can get a sense that they have stronger mental models for trends and the levers controlling trends than others, but concrete judgement is sometimes extremely difficult until a key event has passed and we can judge in hindsight (especially about very high level trends such as Mearsheimer’s disputed take on the causes of the Ukraine invasion, Fukuyama’s infamous “end of history” prediction, or even Pinker’s “Better Angels of Our Nature” predictions about continually declining global conflict).

Hyperion 31 Mar 2025 3:21 UTC
12 points
0
on: Good Research Takes are Not Sufficient for Good Strategic Takes
Nice post! As someone who spends a lot of time in AI policy on strategic thought and talking to people who I think are amongst the best strategic thinkers on AI, I appreciated this piece and think you generally describe the skills pretty well.
However, you say “research” skill by default does not lead to strategic skill, which is very true, but this varies drastically depending on the type of research! Mechanistic interpretability, in fact, appears to me to be an example of a field which is so in the weeds empirical with good feedback loops, that it makes it much harder for researchers in this field to learn better strategic thinking. Other research fields with slower feedback loops are different—for example, societal impacts of AI research. More broadly, I think many fields of social science train strategic skill well, and some of the best political science thinkers clearly have significant strategic skill: Fukuyama, James C. Scott, Dominic Cummings, etc.
I made an attempt to brainstorm ways to evaluate strategic skill based on the abilities of the best thinkers I know, and came up with a list of characteristics I think it is correlated with:
- awareness of both the state of and fundamental principles behind politics and power. A recent example I was discussing with friends: if you read the transcript of the leaked Yemen signal group chat and could immediately write a 2 page doc with likely guesses of the power dynamics between the participants implied by their communication and roles, you probably have some of this awareness.
- scenario-based thinking: the ability to cleanly split the future into distinct scenarios, reason about each independently, effectively simulating the different actors involved (which makes me wonder how much one could accelerate this with LLMs, c.f. simulator theory), before comparing different scenarios based on a small number of key drivers.
- zooming in and zooming out: an ability to not get lost in the details, and put their focus where it is useful, whilst still being aware of the details or having the skill to dig in if necessary.
- generalist knowledge: their skills are usually T shaped, rather than I shaped. They may have read widely across law/econ/political theory/philosophy/sociology/international relations/evo psych or may have background in different worlds (finance, startup ecosystem, defense contracting/intelligence community, etc). This generalism helps in many ways, but one of the biggest is simply that because much of the thinking in any close intellectual community is highly correlated, since everyone reads the same things, you need some source of fresh signal to come up with significant new insights.
- an aptitude for the skill of translation, meaning translation of information and beliefs between frames/perspectives/ways of thinking—something that heavily overlaps with an ability to shape and tell believable narratives. This is the core skill you need to explain, for example, why compute is a useful lever for governing AI to politicians who don’t know what a GPU is, but is also very useful to consider and discard lenses & framings on strategic problems until you can find the one which most cleanly carves up the situation.
- an ability, similar to startup founders, to tolerate everyone thinking they are wrong (correlated with disagreeableness). Often, an exceptional strategic insight backchains to actions which look like they have a weak theory of change to everyone else, which means that good thinkers are often in the position of having to explain themselves.
- context: having both the ability (through reading widely) and position (i.e. access to the thinking of significant figures in the field) to be high context on the big strategic problems. Context is that which is scarce (Cowen). I say context instead of seniority, because they are not quite the same, although strongly correlated—I occasionally meet a number of junior people who clearly are unusually high context for their experience level and have good strategic takes, which is one of the ways I spot promising talent.
Finally, I do notice a lot of those I think have the best strategic thought often use lenses and framings inspired by systems thinking, social evolution/selection processes, memetics, biology, and other similar ways of viewing society and human behavior.

Hyperion 30 Aug 2023 20:49 UTC
2 points
0
on: Report on Frontier Model Training
This is very impressive work, well done! Improving compute/training literacy of the community is very valuable IMO, since I have often thought that not knowing much of this leads to poorer conclusions.

Hyperion 11 Aug 2023 10:12 UTC
2 points
1
on: Inflection.ai is a major AGI lab
Note that the MLPerf benchmark for GPT-3 is not on the full C4 dataset, it’s on 0.4% of the C4 dataset.
See: https://twitter.com/abhi_venigalla/status/1673813863186452480?s=20

Hyperion 7 Dec 2022 3:45 UTC
3 points
0
in reply to: Ronny Fernandez’s comment on: A challenge for AGI organizations, and a challenge for readers
This is an intuition only based on speaking with researchers working on LLMs, but I think that OAI thinks that a model can simultaneously be good enough at next token prediction to assist with research but also be very very far away from being a powerful enough optimizer to realise that it is being optimized for a goal or that deception is an optimal strategy, since the latter two capabilities require much more optimization power. And that the default state of cutting edge LLMs for the next few years is to have GPT-3 levels of deception (essentially none) and graduate student levels of research assistant ability.

Hyperion 2 Oct 2022 22:04 UTC
1 point
0
in reply to: porby’s comment on: Why I think strong general AI is coming soon
I don’t think it’s odd at all—even a terrible chess bot can outplay almost all humans. Because most humans haven’t studied chess. MATH is a dataset of problems from high school competitions, which are well known to require a very limited set of math knowledge and be solveable by applying simple algorithms.
I know chain of thought prompting well—it’s not a way to lift a fundamental constraint, it just is a more efficient targeting of the weights which represent what you want in the model.
It really isn’t hard. No new paradigms are required. The proof of concepts are already implemented and work. It’s more of a question of when one of the big companies decides it’s worth poking with scale.
You don’t provide any proof of this, just speculation, much of it based on massive oversimplifications (if I have time I’ll write up a full rebuttal). For example, RWKV is more of a nice idea that is better for some benchmarks, worse for others, than some kind of new architecture that unlocks greater overall capabilities.

Hyperion 2 Oct 2022 17:53 UTC
0 points
−1
in reply to: porby’s comment on: Why I think strong general AI is coming soon
I mean, to me all this indicates is that our conception of “difficult reasoning problems” is wrong and incorrectly linked to our conception of “intelligence”. Like, it shouldn’t be surprising that the LM can solve problems in text which are notoriously based around applying a short step by step algorithm, when it has many examples in the training set.
To me, this says that “just slightly improving our AI architectures to be less dumb” is incredibly hard, because the models that we would have previously expected to be able to solve trivial arithmetic problems if they could do other “harder” problems are unable to do that.

Hyperion 1 Aug 2022 19:28 UTC
12 points
5
in reply to: Lone Pine’s comment on: chinchilla’s wild implications
Mostly Discord servers in my experience: EleutherAI is a big well known one but there are others with high concentrations of top ML researchers.

Hyperion 24 Jul 2022 19:15 UTC
2 points
0
in reply to: AβMale’s comment on: Everything I Need To Know About Takeoff Speeds I Learned From Air Conditioner Ratings On Amazon
I happened to be reading this post today, as Science has just published a story on a fabrication scandal regarding an influential paper on amyloid-β: https://www.science.org/content/article/potential-fabrication-research-images-threatens-key-theory-alzheimers-disease
I was wondering if this scandal changes the picture you described at all?