My intuition is that a simulation such as the one being proposed would take far longer to develop than the timeline outlined in this post. I’d posit that the timeline would be closer to 60 years than 6.
Also, a suggestion for tl;dr: The Truman Show for AI.
Listening to Eliezer walk through a hypothetical fast takeoff scenario left me with the following question: Why is it assumed that humans will almost surely fail the first time at their scheme for aligning a superintelligent AI, but the superintelligent AI will almost surely succeed the first time at its scheme for achieving its nonaligned outcome?
Speaking from experience, it’s hard to manufacture things in the real world. Doubly so for anything significantly advanced. What is the rationale for assuming that a nonaligned superintelligence won’t trip up at some stage in the hypothetical “manufacture nanobots” stage of its plan?
If I apply the same assumption of initial competence extended to humanity’s attempt to align an AGI to that AGI’s competence in successfully manufacturing some agentic-increasing tool, then the most likely scenario I get is that we’ll see the first unaligned AGI’s attempt at takeoff long before it actually succeeds in destroying humanity.