“That said, I’m not convinced that permanent Holocaust is worse than permanent extinction, but that’s irrelevant to my point anyway.”
Maybe it’s not. What we guess are other people’s values is heavily influenced by our own values. And if you are not convinced that permanent Holocaust is worse than permanent extinction, then, no offense, but you have a very scary value system.
“If someone isn’t convinced by the risk of permanent extinction, are you likely to convince them by the (almost certainly smaller) risk of permanent Holocaust instead?”
Naturally, because the latter is orders of magnitude worse than the former. But again, if you don’t share this view, I can’t see myself convincing you.
And we also have no idea if it really is smaller. But even a small risk of an extremely bad outcome is reason for high alarm.
You can’t get such proof. A) the future is complex and hard to predict, b) there are even many plausible scenarios (not much likely, but plausible) where it doesn’t happen.
The best argument we can come up with is that it would be reasonable to assume a high probability (say 70-90%) that it will happen through x and y, and that probability is definitely too high for it not to become the most important thing of today.
Imo the best route is: Alpha Code can already compete with human programmers in some areas. Will it take long before AI becomes better than humans at programming, and can therefore take care of AI development by itself and rewrite its own code?
I’ve found that people often don’t agree with AI x-risk because they haven’t been exposed to the key concepts like recursive self improvement and basic AI drives. I believe that once you understand those concepts it’s hard not to change your mind, because they paint such a clear picture.