Redefining Fast Takeoff

This post is a re­sult of nu­mer­ous dis­cus­sions with other par­ti­ci­pants and or­ga­niz­ers of the MIRI Sum­mer Fel­lows Pro­gram 2019. It de­scribes ideas that are likely already known by many re­searchers. How­ever, given how of­ten dis­agree­ments about slow/​fast take­offs come up, I be­lieve there is sig­nifi­cant value in mak­ing them com­mon knowl­edge.


Take­off speed & why it matters

In Su­per­in­tel­li­gence, Nick Bostrom dis­t­in­guishes be­tween slow, medium, and fast AI take­off sce­nar­ios (where the take­off speed is mea­sured by how much real-world time passes be­tween the mile­stones of hu­man-level AI (HLAI) and su­per­in­tel­li­gent AI (SAI)). He ar­gues that slow take­off should be rea­son­ably safe since the hu­man­ity would have suffi­cient time co­or­di­nate and solve the AI al­ign­ment prob­lem, while fast take­off would be par­tic­u­larly dan­ger­ous since we wouldn’t be able to re­act to what the AI does.

Real-world time is not what matters

In many sce­nar­ios, the real-time take­off speed in­deed strongly cor­re­lates with our abil­ity to in­fluence the out­come. How­ever, we can also imag­ine many sce­nar­ios where this is not the case. As an ex­am­ple, sup­pose we ob­tain HLAI by simu­lat­ing hu­mans in vir­tual en­vi­ron­ments, and that this pro­ce­dure ad­di­tion­ally fully pre­serves the simu­lated hu­mans’ al­ign­ment with hu­man­ity. Since this effec­tively in­creases the speed at which hu­man­ity op­er­ates, we might get a “fully con­trol­led take­off” even if the tran­si­tion from HLAI to SAI[1] only takes a few days of real-world time. More gen­er­ally, if our path to HLAI also in­creases the effec­tivity of hu­man­ity’s efforts, the “effec­tive time” we get be­tween HLAI and SAI will scale ac­cord­ingly. For ex­am­ple, this might be the case if we go the way of Iter­ated Distil­la­tion and Am­plifi­ca­tion or Com­pre­hen­sive AI Ser­vices. Less con­tro­ver­sially, sup­pose we au­to­mate most of the cur­rent pro­gram­ming tasks and in­crease the re-us­abil­ity of code, such that ev­ery com­puter sci­en­tist be­comes 100-times as effec­tive as they are now.

Con­clu­sion: mea­sure use­ful work

Given these ex­am­ples, I think we should mea­sure take­off speeds not in real-world time, but rather in (some op­er­a­tional­iza­tion of) the work-to­wards-AI-al­ign­ment that hu­man­ity will be able to do be­tween HLAI and SAI. Anec­do­tal ex­am­ples of such mea­sures might in­clude “in­te­gral of the hu­man-origi­nat­ing GDP be­tween HLAI and SAI” or “num­ber of AI safety pa­pers pub­lished be­tween HLAI and SAI”. I be­lieve that find­ing a non-anec­do­tal op­er­a­tional­iza­tion would benefit many AI policy/​strat­egy dis­cus­sions.


  1. Re­call that Bostrom dis­t­in­guishes be­tween speed, col­lec­tive, and qual­ity su­per­in­tel­li­gence. Ar­guably, be­ing able to simu­late hu­mans (with enough com­pute) already con­sti­tutes a speed su­per­in­tel­li­gence. How­ever, I don’t think this diminishes the over­all point of the post. ↩︎

No comments.