Personally, I think this topic is worth considering since the potential downside of malevolence + AGI is so terrifying. *I have low epistemic confidence in what I’m about to say because serious thinking on the topic is only a few years old, I have no particular expertise and the landscape will probably change radically, in unpredictable ways, between now and AGI.
For a malicious actor to establish a singleton assuming a hard takeoff, basically three conditions would be necessary: there is at least one malicious actor, at least one such actor can acquire the code for the AGI, and at least one actor who obtained the information is able to use it to establish a singleton.
I think assigning probabilities 0.5 to each of those conjunctions would be reasonable. All seem quite plausibly correct, and quite plausibly incorrect. I’m not sure what could be argued to justify much lower probabilities than these.
I’m not sure this is the best way of framing the probability (but see*). I reckon:
There are many people on the planet who would act malevolently given a real chance to egt their hands on AGI. I’d say a conservative estimate would be the percentage of the population estimated to be psychopathic, which is 1%.
The vast majority of these people have a near-zero chance of getting anywhere near it. Just to throw a few numbers around wildly, maybe a very rich, very corrupt businessman would have 1% chance, while someone working on the core AGI development team could have as high as 50%. Then you’d have hackers, out-and-out criminals etc. to consider. This variable is so hard to even guess at because it depends how secret the project is, how seriously people are taking the prospect of AGI, and several other factors.
I’m agnostic on whether the 0.5 about the singleton should be higher or lower.
If security isn’t taken very seriously indeed, I don’t think we can disregard this. I’m concerned normalcy bias may cause us to be less prepared than we should be.
TL;DR: Choir agrees preacher’s sermon was very interesting.
So yes, I read this book with no small amount of motivation to enjoy it as I like Julia’s other stuff and am often terrified by the misery that irrationality causes. This is likely not a very impartial comment.
If we assume the goal was to achieve maximum possible swing in total human rationality*, I think it was correct to write the book with a less academic tone than some would have liked. If there had been a load more Bayes’ Theorem in it, people like me would have enjoyed it slightly more, but many others would have stopped reading.
Getting fresh blood to appreciate the benefits of rationality is huge. Once they’re in, they can explore more academic/technical resources if they want.
And even if you are very familiar with the subject matter, you may still need a hand in stopping your soldier mindset barging around with his ridiculous ideas. I have a Zoom call with friends in a bit, and despite just having read The Scout Mindset and being in the middle of writing this sentence, I’ll probably still get too attached to my beliefs once we start talking politics. There’s plenty of low-hanging fruit out there when it comes to walking the talk.
*Whatever the actual goal was, I don’t think this is a terrible proxy.