We can presume that a scientist wants to still exist, and hence doesn’t want to destroy the world. This seems much stronger than a presumption that an Oracle AI will be safe. Of course, an AI might be safe, and a scientist might be out to get us; but the balance of probability says otherwise.
You don’t have any evidence that AIs are generally dangerous (since we have AIs and the empirical evidence is that they are not), and you don’t have a basis for theorising that Oracles are dangerous, because there are a number of different kinds of oracle.
An AI can fool us better simply because it’s smarter (by assumption).
So are out current AIs fooling is? We build them because they are better than us at specific things, but that doesn’t give them the motivation or the ability to fool us. Smartness isn’t a single one-size-all thing and AIs aren’t uniform in their abilities an properties. Once you shed those two illusions, you can see much easier methods of AI safety than those put forward by MIRI.
I still think you’re using “non-agent” as magical thinking.
I still think that if you can build it, it isn’t magic.
But if it tell us X, we won’t accept it. If the Oracle cares that we adopt X, it might answer Y, which does the same as X but looks more appealing.
A narrowly defined AI won’t “care” about anything except answering questions, so it won’t try to second guess us.
Or more subtly, if the AI comes up with Y, it might not tell us that it causes X, because it doesn’t care that X doesn’t fulfil our values, whereas a scientist would note all the implications.
I have dealt with that objection several times. People know that when you use databases and search engines, they don’t fully contextualise things, and the user of the information therefore has to exercise caution.
If humans are incapable of recognizing whether the plan is dangerous or not, it doesn’t matter how much scrutiny they put it through, they won’t be able to discern the danger.
That’s an only-perfection-will-do objection. Of course, humans can’t perfectly scrutinise scientific discovery, etc, so that changes nothing.
You don’t have any evidence that AIs are generally dangerous (since we have AIs and the empirical evidence is that they are not), and you don’t have a basis for theorising that Oracles are dangerous, because there are a number of different kinds of oracle.
So are out current AIs fooling is? We build them because they are better than us at specific things, but that doesn’t give them the motivation or the ability to fool us. Smartness isn’t a single one-size-all thing and AIs aren’t uniform in their abilities an properties. Once you shed those two illusions, you can see much easier methods of AI safety than those put forward by MIRI.
I still think that if you can build it, it isn’t magic.
A narrowly defined AI won’t “care” about anything except answering questions, so it won’t try to second guess us.
I have dealt with that objection several times. People know that when you use databases and search engines, they don’t fully contextualise things, and the user of the information therefore has to exercise caution.
That’s an only-perfection-will-do objection. Of course, humans can’t perfectly scrutinise scientific discovery, etc, so that changes nothing.