Edited for clarity, thanks. I ought have added that the Vizier can take no action of its own—perform experiments, expand its brainpower, ask us to clarify what we mean—to get in a better position to answer the question. Its only power is to give one best guess from its prior information.
. I ought have added that the Vizier can take no action of its own—perform experiments, expand its brainpower, ask us to clarify what we mean—to get in a better position to answer the question.
And how does one define this rigorously enough? Humans have an intuition about what constitutes performing experiments or expanding brainpower but it isn’t obvious what those mean when one is trying to be precise. For example, is a human expanding its brainpower when it writes notes on paper? That’s not directly analogous to making things into computronium, but if an AI decides to convert Earth into memory storage we’re in about as bad shape. “Perform experiment” has similar definitional problems.
It means that the AI can’t do anything outside of its box, aside from taking in 1s and 0s and spitting out 1s and 0s. (Obviously that still allows it to “perform experiments” in the sense of running Monte Carlo simulations or whatever.) Getting it to not torture virtual people would admittedly be an additional problem which this doesn’t cover. The AI has no means with which to convert Earth into memory storage aside from manipulating us. But it doesn’t have any motivation to manipulate us, because the multiplication of question-specific demons means it has a low time horizon—it treats each answer as the final answer; is a deontologist rather than consequentialist.
It means that the AI can’t do anything outside of its box, aside from taking in 1s and 0s and spitting out 1s and 0s.
Really? And are you sure this is all it will do? How do you know for example that it won’t manipulate other objects by fooling with its power source? Or by rapidly turning on and off components send out very specific radio signals to nearby electronic devices? These can both be possibly handld but these are only the most obvious extra angles of attack for the AI.
I think that a properly designed Oracle AI might be possible, but that may be due more to a failure of imagination on my part and my general skepticism of fooming than anything else.
In general, most of the concern on this site regarding unfriendly AI involves the problems associated with autonomous AI and self-improving AI.
You’re right that an AI that cannot do anything unauthorized on its own, and cannot improve itself beyond its initial architecture, is not particularly dangerous. (It can of course be enormously dangerous in short-term practical terms, like many other new technologies, but it does not pose a significant existential risk.)
“Does P=NP?”
turns universe to computronium in order to answer an unanswerable question
Edited for clarity, thanks. I ought have added that the Vizier can take no action of its own—perform experiments, expand its brainpower, ask us to clarify what we mean—to get in a better position to answer the question. Its only power is to give one best guess from its prior information.
And how does one define this rigorously enough? Humans have an intuition about what constitutes performing experiments or expanding brainpower but it isn’t obvious what those mean when one is trying to be precise. For example, is a human expanding its brainpower when it writes notes on paper? That’s not directly analogous to making things into computronium, but if an AI decides to convert Earth into memory storage we’re in about as bad shape. “Perform experiment” has similar definitional problems.
It means that the AI can’t do anything outside of its box, aside from taking in 1s and 0s and spitting out 1s and 0s. (Obviously that still allows it to “perform experiments” in the sense of running Monte Carlo simulations or whatever.) Getting it to not torture virtual people would admittedly be an additional problem which this doesn’t cover. The AI has no means with which to convert Earth into memory storage aside from manipulating us. But it doesn’t have any motivation to manipulate us, because the multiplication of question-specific demons means it has a low time horizon—it treats each answer as the final answer; is a deontologist rather than consequentialist.
Really? And are you sure this is all it will do? How do you know for example that it won’t manipulate other objects by fooling with its power source? Or by rapidly turning on and off components send out very specific radio signals to nearby electronic devices? These can both be possibly handld but these are only the most obvious extra angles of attack for the AI.
I think that a properly designed Oracle AI might be possible, but that may be due more to a failure of imagination on my part and my general skepticism of fooming than anything else.
In general, most of the concern on this site regarding unfriendly AI involves the problems associated with autonomous AI and self-improving AI.
You’re right that an AI that cannot do anything unauthorized on its own, and cannot improve itself beyond its initial architecture, is not particularly dangerous. (It can of course be enormously dangerous in short-term practical terms, like many other new technologies, but it does not pose a significant existential risk.)