Regarding the vast ‘mind design space’, it gets infinitely smaller when you are to stop considering the theoretical stuff based on oracles and realize that the classical computing AI—the one still competing with us for resources—can only square or cube current processing power before it runs out of things in the universe.
But we don’t know how much it can improve its algorithms before it hits the theoretical limit of efficient resource utilization and has to start expanding outward. You sound a bit like you’re assuming that human brains are already near that limit (so that the only way that an AI could beat us was by grabbing lots and lots of resources). So resource-boundedness doesn’t really tell us anything about the upper bound on AI harmfulness. That’s one of the ways you could try to defeat the argument about AI risk. Another would be arguing about the thesis that AIs are likely to be harmful if safety isn’t carefully engineered into them. I don’t see how we could deduce anything reassuring about that issue from the fact that AIs will have bounded resources.
But we don’t know how much it can improve its algorithms before it hits the theoretical limit of efficient resource utilization and has to start expanding outward. You sound a bit like you’re assuming that human brains are already near that limit (so that the only way that an AI could beat us was by grabbing lots and lots of resources). So resource-boundedness doesn’t really tell us anything about the upper bound on AI harmfulness.
You forget one important bit: There are other sources of insights about AI than human analogies and speculations what oracle would do.
It stands that no amount of optimization of AI’s ‘predictor of the future’ can beat Lyapunov’s exponent ; the AI, however effective, still can’t forecast jack shit. On top of that, the AI has enormous number of actions it can take, far far larger than it can process if it were to process them by forecasting the outcomes very accurately.
The very important optimization is not thinking about stuff that has low payoff/thought ratio. Long forecasts have very low payoff, as the cost is exponential in time. Inventions on the other hand should pay off very well. Survival of mankind is not on hand of a calculation ‘okay, action A destroys mankind and action B does not, and action A leads to ever so slightly higher utility 1000 years from now; the mankind got to go’. That sort of forecasting is infeasible. The one with short time span is foolish and near-sighted, while the far future is unknown.
It’s on hands of some general purpose, effective strategies of the kind ‘it is valuable to maximize the choices in the future’ , ‘information is valuable’ (curiosity), ‘penalize actions depending to how badly you can undo them’. Versus ‘if they all act paranoid on Eliezer’s suggestion, they might try damage me’. (Though, hopefully, if the AI is engineering a virus against mankind, the virus won’t exterminate but would medicate for paranoia because that solves immediate problem just as well while leaving more options for future and avoiding actions that can’t be undone even approximately). We are still made of atoms, that AI could use, but there’s quite a plenty of other atoms around. I’d be more worried about it wanting our computation hardware (brains). Crappy it might be, but it is already around, and doesn’t need to be manufactured.
It is very ineffective to just go ahead and limit future options and do entirely irreversible things simply because you don’t have simple expected utility based answer to”why not”. The future you will still be trying to achieve what ever goals you want to achieve, and choosing among choices it has available, and giving the future self more choices is extremely solid heuristic even though you can’t straightforwardly calculate expected utility of doing so (due to recursion).
The oracle is the first mental superpower (the idea dates back quite a while), the least feasible, but the easiest for uneducated to speculate about. It is the easiest way to portray super-intelligence without being super intelligent—why, the super intelligence just knows complete outcomes of it’s actions and chooses best action. That’s easy to think of, and also is extremely inefficient approach to maximization of anything.
The point here is not that AI would necessarily be unable to get rid of mankind. The point is that AI is not particularly more likely to do so than the mankind itself (and may well be less likely). The risks are differential. People are prone to retarded ideologies too. Other people are not you-friendly intelligences, which is totally obvious when you are not living all your life in the privileged class. Groups of other people are unfriendly non-you intelligences, too, highly dangerous and prone to hurting you even if it hurts them as well. Presumably the AI will at least be friendly enough not to hurt you on it’s own expense; you can’t assume even this rudimentary friendliness of your fellow ferocious survival machines, crazed and meme-infested to the brim. It remains to be shown that AI is any more of the existential risk than human all natural stupidity. When one’s speculating up scary stuff one can speculate up the scary human ideologies and social orders as easily as scary AI goal systems. The AI can stop us from killing ourselves, or may kill us, that is not yet a risk until you show that the former is significantly less than latter.
But we don’t know how much it can improve its algorithms before it hits the theoretical limit of efficient resource utilization and has to start expanding outward. You sound a bit like you’re assuming that human brains are already near that limit (so that the only way that an AI could beat us was by grabbing lots and lots of resources). So resource-boundedness doesn’t really tell us anything about the upper bound on AI harmfulness. That’s one of the ways you could try to defeat the argument about AI risk. Another would be arguing about the thesis that AIs are likely to be harmful if safety isn’t carefully engineered into them. I don’t see how we could deduce anything reassuring about that issue from the fact that AIs will have bounded resources.
You forget one important bit: There are other sources of insights about AI than human analogies and speculations what oracle would do.
It stands that no amount of optimization of AI’s ‘predictor of the future’ can beat Lyapunov’s exponent ; the AI, however effective, still can’t forecast jack shit. On top of that, the AI has enormous number of actions it can take, far far larger than it can process if it were to process them by forecasting the outcomes very accurately.
The very important optimization is not thinking about stuff that has low payoff/thought ratio. Long forecasts have very low payoff, as the cost is exponential in time. Inventions on the other hand should pay off very well. Survival of mankind is not on hand of a calculation ‘okay, action A destroys mankind and action B does not, and action A leads to ever so slightly higher utility 1000 years from now; the mankind got to go’. That sort of forecasting is infeasible. The one with short time span is foolish and near-sighted, while the far future is unknown.
It’s on hands of some general purpose, effective strategies of the kind ‘it is valuable to maximize the choices in the future’ , ‘information is valuable’ (curiosity), ‘penalize actions depending to how badly you can undo them’. Versus ‘if they all act paranoid on Eliezer’s suggestion, they might try damage me’. (Though, hopefully, if the AI is engineering a virus against mankind, the virus won’t exterminate but would medicate for paranoia because that solves immediate problem just as well while leaving more options for future and avoiding actions that can’t be undone even approximately). We are still made of atoms, that AI could use, but there’s quite a plenty of other atoms around. I’d be more worried about it wanting our computation hardware (brains). Crappy it might be, but it is already around, and doesn’t need to be manufactured.
It is very ineffective to just go ahead and limit future options and do entirely irreversible things simply because you don’t have simple expected utility based answer to”why not”. The future you will still be trying to achieve what ever goals you want to achieve, and choosing among choices it has available, and giving the future self more choices is extremely solid heuristic even though you can’t straightforwardly calculate expected utility of doing so (due to recursion).
The oracle is the first mental superpower (the idea dates back quite a while), the least feasible, but the easiest for uneducated to speculate about. It is the easiest way to portray super-intelligence without being super intelligent—why, the super intelligence just knows complete outcomes of it’s actions and chooses best action. That’s easy to think of, and also is extremely inefficient approach to maximization of anything.
The point here is not that AI would necessarily be unable to get rid of mankind. The point is that AI is not particularly more likely to do so than the mankind itself (and may well be less likely). The risks are differential. People are prone to retarded ideologies too. Other people are not you-friendly intelligences, which is totally obvious when you are not living all your life in the privileged class. Groups of other people are unfriendly non-you intelligences, too, highly dangerous and prone to hurting you even if it hurts them as well. Presumably the AI will at least be friendly enough not to hurt you on it’s own expense; you can’t assume even this rudimentary friendliness of your fellow ferocious survival machines, crazed and meme-infested to the brim. It remains to be shown that AI is any more of the existential risk than human all natural stupidity. When one’s speculating up scary stuff one can speculate up the scary human ideologies and social orders as easily as scary AI goal systems. The AI can stop us from killing ourselves, or may kill us, that is not yet a risk until you show that the former is significantly less than latter.