To clarify, the conditional probability in the parent comment is not conditioned on no policy effort or intervention, it’s conditional on whatever policy / governance / voluntary measures are tried being insufficient or ineffective, given whatever the actual risk turns out to be.
If a small team hacking in secret for a few months can bootstrap to superintelligence using a few GPUs, the necessary level of policy and governance intervention is massive. If the technical problem has a somewhat different nature, then less radical interventions are plausibly sufficient.
I personally feel pretty confident that:
Eventually, and maybe pretty soon (within a few years), the nature of the problem will indeed be that it is plausible a small team can bootstrap to superintelligence in secret, without massive resources.
Such an intelligence will be dramatically harder to align than it is to build, and this difficulty will be non-obvious to many would-be builders.
And believe somewhat less confidently that:
The governance and policy interventions necessary to robustly avert doom given these technical assumptions are massive and draconian.
We are not on track to see such interventions put in place.
Given different views on the nature of the technical problem (the first two bullets), you can get a different level of intervention which you think is required for robust safety (the third bullet), and different estimate that such an intervention is put in place successfully (the fourth bullet).
I think it’s also useful to think about cases where policy interventions were (in hindsight) obviously not sufficient to prevent doom robustly, but by luck or miracle (or weird anthropics) we make it through anyway. My estimate of this probability is that it’s really low—on my model, we need a really big miracle, given actually-insufficient intervention. What “sufficient intervention” looks like, and how likely we are to get it, I find much harder to estimate.
To clarify, the conditional probability in the parent comment is not conditioned on no policy effort or intervention, it’s conditional on whatever policy / governance / voluntary measures are tried being insufficient or ineffective, given whatever the actual risk turns out to be.
If a small team hacking in secret for a few months can bootstrap to superintelligence using a few GPUs, the necessary level of policy and governance intervention is massive. If the technical problem has a somewhat different nature, then less radical interventions are plausibly sufficient.
I personally feel pretty confident that:
Eventually, and maybe pretty soon (within a few years), the nature of the problem will indeed be that it is plausible a small team can bootstrap to superintelligence in secret, without massive resources.
Such an intelligence will be dramatically harder to align than it is to build, and this difficulty will be non-obvious to many would-be builders.
And believe somewhat less confidently that:
The governance and policy interventions necessary to robustly avert doom given these technical assumptions are massive and draconian.
We are not on track to see such interventions put in place.
Given different views on the nature of the technical problem (the first two bullets), you can get a different level of intervention which you think is required for robust safety (the third bullet), and different estimate that such an intervention is put in place successfully (the fourth bullet).
I think it’s also useful to think about cases where policy interventions were (in hindsight) obviously not sufficient to prevent doom robustly, but by luck or miracle (or weird anthropics) we make it through anyway. My estimate of this probability is that it’s really low—on my model, we need a really big miracle, given actually-insufficient intervention. What “sufficient intervention” looks like, and how likely we are to get it, I find much harder to estimate.