I’m sure this has been discussed elsewhere ad nauseam, but this view always struck me as extremely overconfident: We have no idea what a superintelligence could discover using today’s data or experiment throughput capacity, and “close to human level” seems a priori unlikely.
Kurt H. Pieper
Karma: 24
One possibility suggested by this essay is that Dario does not really believe in superintelligence (the hypothesis we are currently examining). Another is that he does, but has chosen to dissemble for strategic purposes. While I don’t think Dario is above communicating strategically, I do in fact think this is roughly his mainline worldview, and it follows pretty clearly from his beliefs in 2017. Maybe there are other possibilities apart from those two, though I haven’t figured out what they might be.
You have possibly dismissed the “strategic purposes” hypothesis prematurely here.
Saying you want to build “superintelligence” may be really bad for political capital. To me it seems entirely plausible that he has “self-censored” his beliefs around superintelligence since 2017.
Dario spells this out:
The result often ends up reading like a fantasy for a narrow subculture, while being off-putting to most people.
As far as I know[1], Hassabis has also not explicitly said he aims for superintelligence while being CEO of DeepMind, despite being an early singularitarian.
- ^
This is mainly based on asking Gemini 3 Pro, though.
- ^
To me, such costly and invasive regulations seem a priori unlikely given the nature of the issue and our epistemics thereof. Can you expand on that?
I agree that we can stop or delay the development of some technology. However, AI does not appear to be one of them, in my view, for several reasons:
(i). The x-risk has little to no political salience, there is no such thing as a “scientific consensus”[1] yet, and you can’t point to a concrete instance of the problem yet that will convince the uninitiated.[2]
(ii). Every policy that stops superintelligence from being developed will likely come with massive, concrete economic costs.
(iii). Fast global coordination is needed.
No example in your list has all of these, for instance:
1. Doesn’t have (i): The Thalidomide Disaster
2. Doesn’t have (i): Meltdowns, “big fireball bad” again.
4. & 5: Don’t have (ii): No comparable concrete economic costs
9. Doesn’t have any: No global coordination, concrete problem (“people are spying on you”), no comparable economic costs.
- ^
You can point to the CAIS statement or similar things, but this is not in the same category as e.g. the consensus around climate change.
- ^
For example, you don’t need to understand nuclear physics to understand “big fireball bad”, but you need more theory to be scared about METR reward hacking results.
I have almost no intuition whatsoever on why steering should be akin to OCD or schizophrenia, and I don’t think you offer sufficient rationale. I believe steering resistance is downstream of self-repair, which is known to emerge (i.e. dropout).
Possibly the method will underestimate parameter count as time goes on. I don’t expect it to be economically valuable to pretrain on the very long tails of knowledge, as opposed to letting more bits flow in from synthetic data / RLVR. Though I’m surprised as to why this hasn’t already happened.