The more powerful a tool is, the more important it is that the tool behaves predictably.
A chainsaw that behaves unpredictably is very, very, dangerous.
AI is, conservatively, thousands of times more powerful than a chainsaw.
And unlike an unpredictable chainsaw, there is no guarantee we will be able to turn an unpredictable AI off and fix it or replace it.
It is plausible that the danger of failing to align AI safely—to make it predictable—is such that we only have one chance to get it right.
Finally, it is absurdly cheap to make massive progress in AI safety.
This was wonderful; the post that finally got me to create an account here. I got quite a few sensible chuckles and a few hearty laughs out of your list. I think we’ve been reading similar books recently (Graeber’s Dawn of Everything? :) )
My contribution is to remind the participants that a somewhat recurring theme (something of an original in western philosophy—i.e. Socrates) in history is of wise people enjoying themselves too much and getting murdered by the people who’d grown increasingly scared/estranged/horrified by them.
Heretical thinking is fun, but in the real world there are people who would harm you for exposing them to it.
Practice safe heresy kids :)