I don’t have a clear sense of how dangerous a group of amoral fast-thinking miniature Isaac Newtons might be but it would surprise me if there were a particularly important risk-evaluation threshold crossed between 70 million amoral fast-thinking miniature Isaac Newtons and a mere, say, 700,000 of them.
Admittedly, I may be being distracted by the image of hundreds of thousands of miniature Isaac Newtons descending on Washington DC or something. It’s a far more entertaining idea than those interminable zombie stories.
You are right that 1% of the world population is likely too large. I probably should have said “substantial numbers in existence.” I’ve adjusted my estimate, so amoral Newtons don’t worry me unless they are secret or exist (>1000). And the minimum number gets bigger unless there is reason to think amoral Newtons will cooperate amongst themselves to dominate humanity.
I don’t have a clear sense of how dangerous a group of amoral fast-thinking miniature Isaac Newtons might be but it would surprise me if there were a particularly important risk-evaluation threshold crossed between 70 million amoral fast-thinking miniature Isaac Newtons and a mere, say, 700,000 of them.
Admittedly, I may be being distracted by the image of hundreds of thousands of miniature Isaac Newtons descending on Washington DC or something. It’s a far more entertaining idea than those interminable zombie stories.
You are right that 1% of the world population is likely too large. I probably should have said “substantial numbers in existence.” I’ve adjusted my estimate, so amoral Newtons don’t worry me unless they are secret or exist (>1000). And the minimum number gets bigger unless there is reason to think amoral Newtons will cooperate amongst themselves to dominate humanity.
I don’t think the numbers I was referencing quite came across to you.
I was postulating humans:AGIs :: 1:10,000
So not 70,000 Newtons or 70 million Newtons -- 70,000 Billion Newtons.