Here’s a doubt for you: I’m a nerd, I like nerds, I’ve worked on technology, and I’ve loved techie projects since I was a kid. Grew up on SF, all of that.
My problem lately is that I can’t take Friendly AI arguments seriously. I do think AI is possible, that we will invent it. I do think that at some point in the next hundreds of years, it will be game over for the human race. We will be replaced and/or transformed.
I kind of like the human race! And I’m forced to conclude that a human race without that tiny fraction of nerds could last a good long time yet (tens of thousands of years) and would change only slowly, through biological evolution. They would not do much technology, since it takes nerds (in the broadest sense) to do this. But, they would still have fulfilling, human, lives.
On the other hand, I don’t think a human race with nerds can forever avoid inventing a self-destructive technology like AI. So much as I have been brought up to think of politicians and generals as destroyers, and scientists and other nerds as creators, I have to admit that it’s the other way around, ultimately.
The non-nerds can’t destroy the human race. Only we nerds can do that.
That’s my particular crisis of faith. Care to take a side?
There’s that old quote: “never let your sense of morality keep you from doing what you know is right.”
I’d still like an answer to the most basic Friendly AI question: what do you want it to do? Forget the implementation problems for a second, and just give me a scenario where the AI is doing what you want it to do. What does that world look like? Because I don’t even know what I want from that future.