Bostrom’s results seem very sensitive to deviations from a wholly person-affecting perspective. To investigate, I coded up the model from Appendix A with one modification: I supposed that, instead of being wholly self interested, people are willing to sacrifice 10% of life expectancy for the sake of all future generations.
My method was to calculate the launch time that is later than the optimal time-point according to a selfish view, but only so much that life expectancy is reduced 10% from the selfish optimum.[^1] This method is crude, but illustrates how rush-to-launch loses support if one walks mildly away from a person-affecting view.
For example, with 20% $P_{doom}$ and 10%/yr safety progress, the selfishly optimal launch time is 8 months (Bostrom’s Table 3), which offers you 1,120 years of life expectancy. If you are willing to sacrifice 10% of that life expectancy (leaving you with 1,008 expected years of life!) for future generations, you would wait 11 years before launch to help safety become established. More generally, all superintelligence launch times from Table 3 were delayed by at least 4 years (none were ASAP anymore) and many were delayed 10-20 years. The rush to superintelligence was ameliorated.
Last, I estimated delays under more sacrifice. If people are willing to lose half of life expectancy to help ensure the existence of future generations, then superintelligence launch times would be delayed by at least 28 years for all scenarios covered in Bostrom’s Table 3. Results are below. Further, for all cases with $P_{doom}$ of 80% or less, the life expectancies of those making the sacrifice would remain generous, exceeding 140 years for $P_{doom}$ of 80%, exceeding 349 years for $P_{doom}$ of 50%, and exceeding 550 years for $P_{doom}$ of 20% or less.
Table A. ASI launch delay by P(doom) and safety progress, offering 50% of life expectancy. The table includes the scenarios from Bostrom’s Table 3.
Safety progress
P(doom)
1%
5%
20%
50%
80%
95%
99%
No progress
(0%/yr)*
29 y
29 y
29 y
30 y
35 y
Never
launch
Never
launch
Glacial
(0.1%/yr)
29 y
29 y
30 y
32 y
42 y
Never
launch
Never
launch
Very slow
(1%/yr)
29 y
30 y
32 y
43 y
80 y
102 y
108 y
Moderate
(10%/yr)
29 y
31 y
38 y
47 y
52 y
54 y
54 y
Brisk
(50%/yr)
29 y
31 y
33 y
34 y
35 y
35 y
35 y
Very fast
(90%/yr)
29 y
30 y
31 y
31 y
31 y
31 y
31 y
Ultra fast
(99%/yr)
29 y
30 y
30 y
30 y
30 y
30 y
30 y
* Note that the sacrifice is fruitless in this case, because there is no safety progress during the delay. Similarly, the sacrifice may not have reasonable justification in the ultra-fast case.
[^1]: To confirm my code’s correctness, I also recreated Bostrom’s Table 3. This revealed a typo in Table 3: For $P_{doom} = 0.95$ and safety progress of 1%, the launch time is listed as 14.3 years but should be about 31.4.
Bostrom’s results seem very sensitive to deviations from a wholly person-affecting perspective. To investigate, I coded up the model from Appendix A with one modification: I supposed that, instead of being wholly self interested, people are willing to sacrifice 10% of life expectancy for the sake of all future generations.
My method was to calculate the launch time that is later than the optimal time-point according to a selfish view, but only so much that life expectancy is reduced 10% from the selfish optimum.[^1] This method is crude, but illustrates how rush-to-launch loses support if one walks mildly away from a person-affecting view.
For example, with 20% $P_{doom}$ and 10%/yr safety progress, the selfishly optimal launch time is 8 months (Bostrom’s Table 3), which offers you 1,120 years of life expectancy. If you are willing to sacrifice 10% of that life expectancy (leaving you with 1,008 expected years of life!) for future generations, you would wait 11 years before launch to help safety become established. More generally, all superintelligence launch times from Table 3 were delayed by at least 4 years (none were ASAP anymore) and many were delayed 10-20 years. The rush to superintelligence was ameliorated.
Last, I estimated delays under more sacrifice. If people are willing to lose half of life expectancy to help ensure the existence of future generations, then superintelligence launch times would be delayed by at least 28 years for all scenarios covered in Bostrom’s Table 3. Results are below. Further, for all cases with $P_{doom}$ of 80% or less, the life expectancies of those making the sacrifice would remain generous, exceeding 140 years for $P_{doom}$ of 80%, exceeding 349 years for $P_{doom}$ of 50%, and exceeding 550 years for $P_{doom}$ of 20% or less.
Table A. ASI launch delay by P(doom) and safety progress, offering 50% of life expectancy. The table includes the scenarios from Bostrom’s Table 3.
No progress
(0%/yr)*
Never
launch
Never
launch
Glacial
(0.1%/yr)
Never
launch
Never
launch
Very slow
(1%/yr)
Moderate
(10%/yr)
Brisk
(50%/yr)
Very fast
(90%/yr)
Ultra fast
(99%/yr)
* Note that the sacrifice is fruitless in this case, because there is no safety progress during the delay. Similarly, the sacrifice may not have reasonable justification in the ultra-fast case.
[^1]: To confirm my code’s correctness, I also recreated Bostrom’s Table 3. This revealed a typo in Table 3: For $P_{doom} = 0.95$ and safety progress of 1%, the launch time is listed as 14.3 years but should be about 31.4.