I think that “we should only slow down AI development if for each year of slowing down we would be reducing risk of human extinction by more than 1%” is not a sufficient crux for the (expensive) actions which I most want at current margins, at least if you have my empirical views. I think it is very unlikely (~7%?) that in practice we reach near the level of response (in terms of spending/delaying for misalignment risk reduction) that would be rational given this “1% / year” view and my empirical views, so my empirical views suffice to imply very different actions.
For instance, delaying for ~10 years prior to building wildly superhuman AI (while using controlled AIs at or somewhat below the level of top human experts) seems like it probably makes sense on my views but this moral perspective, especially if you can use the controlled AIs to substantially reduce/delay ongoing deaths which seems plausible. Things like massively investing in safety/alignment work also easily makes sense. There are policies that substantially reduce the risk which merely require massive effort (and which don’t particularly delay powerful AI) that we could be applying.
I do think that this policy wouldn’t be on board with the sort of long pause that (e.g.) MIRI often discusses and it does materially alter what look like the best policies (though ultimately I don’t expect to get close to these best policies anyway).
I think that “we should only slow down AI development if for each year of slowing down we would be reducing risk of human extinction by more than 1%” is not a sufficient crux for the (expensive) actions which I most want at current margins, at least if you have my empirical views. I think it is very unlikely (~7%?) that in practice we reach near the level of response (in terms of spending/delaying for misalignment risk reduction) that would be rational given this “1% / year” view and my empirical views, so my empirical views suffice to imply very different actions.
For instance, delaying for ~10 years prior to building wildly superhuman AI (while using controlled AIs at or somewhat below the level of top human experts) seems like it probably makes sense on my views but this moral perspective, especially if you can use the controlled AIs to substantially reduce/delay ongoing deaths which seems plausible. Things like massively investing in safety/alignment work also easily makes sense. There are policies that substantially reduce the risk which merely require massive effort (and which don’t particularly delay powerful AI) that we could be applying.
I do think that this policy wouldn’t be on board with the sort of long pause that (e.g.) MIRI often discusses and it does materially alter what look like the best policies (though ultimately I don’t expect to get close to these best policies anyway).