While the singularity doesn’t have a reference class, benchmarks do have a reference class—we have enough of them that we can fit reasonable distributions on when benchmarks will reach 50%, be saturated, etc., especially if we know the domain. The harder part is measuring superintelligence with benchmarks.
Yup, agree this holds for a while, and agree that benchmarking superintelligence is tricky. However, I think there’s reason to expect the dynamics driving the benchmarks to change in ways which don’t have clear precedent at some point, once new parts of the feedback loops seriously take off.
I operationalize as: I expect >2 and at least twice as many trend-breaking events in the reference of the move from 7 to 4 month doubling times as downwards trend breaking ones.
I expect that at some point, possibly after loss of control, a large number of things unblock in quick succession and you get a supercritical chain reaction of improvements to capabilities which bows naive extrapolation out of the water.
While the singularity doesn’t have a reference class, benchmarks do have a reference class—we have enough of them that we can fit reasonable distributions on when benchmarks will reach 50%, be saturated, etc., especially if we know the domain. The harder part is measuring superintelligence with benchmarks.
Yup, agree this holds for a while, and agree that benchmarking superintelligence is tricky. However, I think there’s reason to expect the dynamics driving the benchmarks to change in ways which don’t have clear precedent at some point, once new parts of the feedback loops seriously take off.
I operationalize as: I expect >2 and at least twice as many trend-breaking events in the reference of the move from 7 to 4 month doubling times as downwards trend breaking ones.
I expect that at some point, possibly after loss of control, a large number of things unblock in quick succession and you get a supercritical chain reaction of improvements to capabilities which bows naive extrapolation out of the water.