I feel that both this and EYs complex nanotechnology are far too fairy tale like.
Any competent virologist could make a vaccine resistant, contagious, highly lethal to humans virus. We know how to do it—this is the entire field of gain of function research. It doesn’t need any global infrastructure—just a local lab, resources at the few million dollar level, and intention. An AGI could certainly do this. No new technology (beyond the AGI itself) required. I feel that if any scenario is going to convince the “no problem here” skeptics it would be that one. Especially since COVID is a highly contagious, new virus that by dumb luck is not all that lethal.
Many of these arguments seem pathological when applied to an individual.
I have a friend, let’s call her B, she has a 6 year old daughter A. She of course adores her daughter. If I walked up to B and said “I’m going to inject this syringe into your daughter. There’s a 10% chance it’ll kill her, and a 50% chance it’ll extend her natural lifetime to 200.” Then I jab A.
EV on A’s life expectancy is strongly positive. B (and almost everybody) would be very upset if I did this. I’m upset with accelerationists for the same reasons.