I work in this field, and was under approximately the opposite impression; that voice and visual recognition are rapidly approaching human levels. If I’m wrong and there are sharp limits, I’d like to know. Thanks!
Simulation_Brain
Karma: 46
Now this is an interesting thought. Even a satisficer with several goals but no upper bound on each will use all available matter on the mix of goals it’s working towards. But a limited goal (make money for GiantCo, unless you reach one trillion, then stop) seems as though it would be less dangerous. I can’t remember this coming up in Eliezer’s CFAI document, but suspect it’s in there with holes poked in its reliability.
I think the concern stands even without a FOOM; if AI gets a good bit smarter than us, however that happens (design plus learning, or self-improvement), it’s going to do whatever it wants.
As for your “ideal Bayesian” intuition, I think the challenge is deciding WHAT to apply it to. The amount of computational power needed to apply it to every thing and every concept on earth is truly staggering. There is plenty of room for algorithmic improvement, and it doesn’t need to get that good to outwit (and out-engineer) us.
I think there are very good questions in here. Let me try to simplify the logic:
First, the sociological logic: if this is so obviously serious, why is no one else proclaiming it? I think the simple answer is that a) most people haven’t considered it deeply and b) someone has to be first in making a fuss. Kurzweil, Stross, and Vinge (to name a few that have thought about it at least a little) seem to acknowledge a real possibility of AI disaster (they don’t make probability estimates).
Now to the logical argument itself:
a) We are probably at risk from the development of strong AI. b) The SIAI can probably do something about that.
The other points in the OP are not terribly relevant; Eliezer could be wrong about a great many things, but right about these.
This is not a castle in the sky.
Now to argue for each: There’s no good reason to think AGI will NOT happen within the next century. Our brains produce AGI; why not artificial systems? Artificial systems didn’t produce anything a century ago; even without a strong exponential, they’re clearly getting somewhere.
There are lots of arguments for why AGI WILL happen soon; see Kurzweil among others. I personally give it 20-40 years, even allowing for our remarkable cognitive weaknesses.
Next, will it be dangerous? a) Something much smarter than us will do whatever it wants, and very thoroughly. (this doesn’t require godlike AI, just smarter than us. Self-improving helps, too.) b) The vast majority of possible “wants” done thoroughly will destroy us. (Any goal taken to extremes will use all available matter in accomplishing it.) Therefore, it will be dangerous if not VERY carefully designed. Humans are notably greedy and bad planners individually, and often worse in groups.
Finally, it seems that SIAI might be able to do something about it. If not, they’ll at least help raise awareness of the issue. And as someone pointed out, achieving FAI would have a nice side effect of preventing most other existential disasters.
While there is a chain of logic, each of the steps seems likely, so multiplying probabilities gives a significant estimate of disaster, justifying some resource expenditure to prevent it (especially if you want to be nice). (Although spending ALL your money or time on it probably isn’t rational, since effort and money generally have sublinear payoffs toward happiness).
Hopefully this lays out the logic; now, which of the above do you NOT think is likely?
I think the point is that not valuing non-interacting copies of oneself might be inconsistent. I suspect it’s true; that consistency requires valuing parallel copies of ourselves just as we value future variants of ourselves and so preserve our lives. Our future selves also can’t “interact” with our current self.
Quality matters if you have a community that’s interested in your work; you’ll get more “nice job” comments if it IS a nice job.
I don’t think the lack of an earth-shattering ka-FOOM changes much of the logic of FAI. Smart enough to take over the world is enough to make human existence way better, or end it entirely.
It’s quite tricky to ensure that your superintelligent AI does anything like what you wanted it to. I don’t share the intuition that creating a “homeostasis” AI is any easier than an FAI. I think one move Eliezer is making in his “Creating Friendly AI” strategy is to minimize the goals you’re trying to give the machine; just CEV.
I think this makes apparent what a good CEV seeker needs anyway; some sense of restraint when CEV can’t be reliably extrapolated in one giant step. It’s less than certain that even a full FOOM AI could reliably extrapolate to some final most-preferred world state.
I’d like to see a program where humanity actually chooses its own future; we skip the extrapolation and just use CV repeatedly; let people live out their own extrapolation.
Does just CV work all right? I don’t know, but it might. Sure, Palestinians want to kill Israelis and vice versa; but they both want to NOT be killed way more than they want to kill, and most other folks don’t want to see either of them killed.
Or perhaps we need a much more cautious, “OK, let’s vote on improvements, but they can’t kill anybody and benefits have to be available to everyone...” policy for the central guide of AI.
CEV is a well thought out proposal (perhaps the only one—counterexamples?), but we need more ideas in the realm of AI motivation/ethics systems. Particularly, ways to get from a practical AI with goals like “design neat products for GiantCo” or “obey orders from my commanding officer” to ensure that they don’t ruin everything if they start to self-improve. Not everyone is going to want to give their AI CEV as its central goal, at least not until it’s clear it can/will self improve, at which point it’s probably too late.
Well, yes; it’s not straightforward to go from brains to preferences. But for any particular definition of preference, a given brain’s “preference” is just a fact about that brain. If this is true, it’s important to understanding morality/ethics/volition.
I think this is an excellent question. I’m hoping it leads to more actual discussion of the possible timeline of GAI.
Here’s my answer, important points first, and not quite as briefly as I’d hoped.
1) even if uFAI isn’t the biggest existential risk, the very low investment and interest in it might make it the best marginal value for investment of time or money. As someone noted, having at least a few people thinking about the risk far in advance seems like a great strategy if the risk is unknown.
2) No one but SIAI is taking donations to mitigate the risk (as far as I know) so your point 2 is all but immaterial right now.
3) I personally estimate the risk of uFAI to be vastly higher than any other, although I am as you point out quite biased in that direction. I don’t think other existential threats come close (although I don’t have the expertise to evaluate “gray goo” self replicator dangers) . a) AI is a new risk; (plagues and nuclear wars have failed to get us so far) b) it can be deadly in new ways (outsmarting/out-teching us); c) we don’t know for certain that it won’t happen soon.
How hard is AI? We actually don’t know. I study not just the brain but how it gets computation and thinking done (a rare and fortunate job; most neuroscientists study neurons, not the whole mind) - and I think that its principles aren’t actually all that complex. To put it this way: algorithms are rapidly approaching the human level in speech and vision, and the principles of higher-level thinking appear to be similar. (as an aside, EYs now-outdated Levels of General Intelligence does a remarkably good job of converging with my independently-developed opinion on principles of brain function) In my limited (and biased) experience, those with similar jobs tend to have similar opinions. But the bottom line is that we don’t know either how hard, or how easy, it could turn out to be. Failure to this point is not strong evidence of continued failure.
And people will certainly try. The financial and power incentives are such that people will continue their efforts on narrow AI, and proceed to general AI when it helps solve problems. Recent military and intelligence grants indicate a trend in increasing interest in getting beyond narrow AI to get more useful AI; things that can make intelligence and military decisions and actions more cheaply (and eventually reliably) than a human. Industry similarly has a strong interest in narrow AI (e.g, sensory processing) but they will probably be a bit later to the GAI party given their track record of short term thinking. Academics are certainly are doing GAI research, in addition to lots of narrow AI stuff. Have a look at the BICA (biologically inspired cognitive architecture) conference for some academic enthusiasts with baby GAI projects.
So, it could happen soon. If it gets much smarter than us, it will do whatever it wants; and if we didn’t build its motivational system veeery carefully, doing what it wants will eventually involve using all the stuff we need to live.
Therefore, I’d say the threat is on the order of 10-50%, depending on how fast it develops, how easy making GAI friendly turns out to be, and how much attention the issue gets. That seems huge relative to other truly existential threats.
If it matters, I believed very similar things before stumbling on LW and EY’s writings.
I hope this thread is attracting some of the GAI sceptics; I’d like to stress-test this thinking.