Thanks, your numbered list was very helpful in encouraging to go through the claims. Just two things that stood out to me:
39 Nothing we can do with a safe-by-default AI like GPT-3 would be powerful enough to save the world (to ‘commit a pivotal act’), although it might be fun. In order to use an AI to save the world it needs to be powerful enough that you need to trust its alignment, which doesn’t solve your problem.
What exactly makes people sure that something like GPT would be safe/unsafe?
If what is needed is some form of insight/break through: Some smarter version of GPT-3 seems really useful? The idea that GPT-3 produces better poetry than me while GPT-5 could help to come up with better alignment ideas, doesn’t strongly conflict with my current view of the world?
Worth noting that the more precise #12 is substantially more optimistic than 12 as stated explicitly here.
#12:
“An aligned advanced AI created by a responsible project that is hurrying where it can, but still being careful enough to maintain a success probability greater than 25%, will take the lesser of (50% longer, 2 years longer) than would an unaligned unlimited superintelligence produced by cutting all possible corners.”
This might come across as optimistic if this was your median alignment difficulty estimate, but instead Elizier is putting 95% on this, which on the flip side suggests a 5% chance that things turn out to be easier. This seems rather in line with “Carefully aligning an AGI would at best be slow and difficult, requiring years of work, even if we did know how.”
Running upstairs is also a good example (the trick is getting to the point where you can run up the stairs fast enough that you only feel you are exhausted when you are already up). Or jumping downstairs. Generally, optimizing for fun is probably the most sustainable way of doing this. I don’t enjoy most sports that require endurance, but like doing things like running in short bursts, so it works pretty well for me. The biggest downside is probably strangers looking at you weird, but I mostly don’t notice and certainly don’t mind. Walking in a group is different, of course.