but didn’t know enough about the real world to know what code to write.
This requires two things: knowing what you want, and learning about the world.
I don’t see the fundamental problem in getting an AI to learn about the world. The informal human epistemic process has been analyzed into components, and these have been formalized and implemented in ways far more powerful than an unaided human can manage. It’s a lot of work to put it all together in a self-consistent package, and to give it enough self-knowledge and world-knowledge to set it in motion, and it would require a lot of computing power. But I don’t see any fundamental difficulty.
What the AI wants is utterly contingent on initial conditions. But an AI that can represent the world and learn about it, can also represent just about any goal you care to give it, so there’s no extra problem to solve here. (Except for Friendliness. But that is the specific problem of identifying a desirable goal, not the general problem of implementing goal-directed behavior.)
Just reviewing this basic argument reinforces the prior impression that we are already drifting towards transhuman AI and that there’s no fundamental barrier in the way. We already know enough for hard work alone to get us there—I mean the hard work of tens of thousands of researchers in many fields, not one person or one group making a super-duper effort. The other factor which seals our fate is distributed computing. Even if Moore’s law breaks down, computers can be networked, and there are lots of computers.
So, we are going to face something smarter than human, which means something that can outwit us, which means something that should win if its goals are ever in conflict with ours. And there is no law of nature to guarantee that its goals will be humanly benevolent. On the contrary, it seems like anything might serve as the goal of an AI, just as “any” numerical expression might be fed to a calculator for evaluation.
What we don’t know is how likely it is that the first transhuman AI’s goals will be bad for us. A transhuman AI may require something like the resources of a large contemporary server farm to operate, in which case it’s not going to happen by accident. There is some possibility that the inherent difficulty of getting there renders it more likely that by the time you get to transhuman AI, the people working on it have thought ahead to the time when the AI is autonomous and in fact beyond stopping, and realized that it had better have “ethics”. But that just means that by the time that the discipline of AI is approaching the transhuman threshold, people are probably becoming aware of what we have come to call the problem of friendliness. It doesn’t mean that the problem is sure to have been solved by the time the point of no return is reached.
All in all, therefore, I conclude (1) the Singularity concept makes sense (2) it is a matter for concern in the present and near future, not the far future (3) figuring out the appropriate initial conditions for an ethical AI is the key problem to solve (4) SIAI is historically important as the first serious attempt to solve this problem.
This requires two things: knowing what you want, and learning about the world.
I don’t see the fundamental problem in getting an AI to learn about the world. The informal human epistemic process has been analyzed into components, and these have been formalized and implemented in ways far more powerful than an unaided human can manage. It’s a lot of work to put it all together in a self-consistent package, and to give it enough self-knowledge and world-knowledge to set it in motion, and it would require a lot of computing power. But I don’t see any fundamental difficulty.
What the AI wants is utterly contingent on initial conditions. But an AI that can represent the world and learn about it, can also represent just about any goal you care to give it, so there’s no extra problem to solve here. (Except for Friendliness. But that is the specific problem of identifying a desirable goal, not the general problem of implementing goal-directed behavior.)
Just reviewing this basic argument reinforces the prior impression that we are already drifting towards transhuman AI and that there’s no fundamental barrier in the way. We already know enough for hard work alone to get us there—I mean the hard work of tens of thousands of researchers in many fields, not one person or one group making a super-duper effort. The other factor which seals our fate is distributed computing. Even if Moore’s law breaks down, computers can be networked, and there are lots of computers.
So, we are going to face something smarter than human, which means something that can outwit us, which means something that should win if its goals are ever in conflict with ours. And there is no law of nature to guarantee that its goals will be humanly benevolent. On the contrary, it seems like anything might serve as the goal of an AI, just as “any” numerical expression might be fed to a calculator for evaluation.
What we don’t know is how likely it is that the first transhuman AI’s goals will be bad for us. A transhuman AI may require something like the resources of a large contemporary server farm to operate, in which case it’s not going to happen by accident. There is some possibility that the inherent difficulty of getting there renders it more likely that by the time you get to transhuman AI, the people working on it have thought ahead to the time when the AI is autonomous and in fact beyond stopping, and realized that it had better have “ethics”. But that just means that by the time that the discipline of AI is approaching the transhuman threshold, people are probably becoming aware of what we have come to call the problem of friendliness. It doesn’t mean that the problem is sure to have been solved by the time the point of no return is reached.
All in all, therefore, I conclude (1) the Singularity concept makes sense (2) it is a matter for concern in the present and near future, not the far future (3) figuring out the appropriate initial conditions for an ethical AI is the key problem to solve (4) SIAI is historically important as the first serious attempt to solve this problem.