Think of Solomonoff Induction or some approximation of it.
Which is uncomputable, and an approximation would presumably benefit from increased computing power.
It is not an agent. It just tries every possible hypothesis on that data and does a Bayesian update.
Like that’s simple? How exactly do you make an AI that isn’t an agent? With no goals, why does it do anything?
I don’t know exactly what strategies it will use, but it’s utility function is literally to minimize computing power. If you trust the AI to fulfill it’s utility function, then you can trust it will do this to the best of it’s ability.
But why are plans that take less computing power to come up with more likely to be safe? Besides, if it calculates that searching for simple solutions is likely not going to meet the 90% criteria, it can forgoe that and jump straight to complicated ones.
Which is uncomputable, and an approximation would presumably benefit from increased computing power.
I gave the simplest possible counter example to your objection. I never proposed that we actually use pure Solomonoff induction.
EDIT: I realize you said something different. You implied that an approximation of Solomonoff induction would benefit from more computing power, and so would act as an agent to obtain it. This is totally incorrect. Solomonoff induction can be approximated in various ways by bounding the run time of the programs, or using simpler models instead of computer programs, etc. None of these create any agentness. They still just do prediction. I’m not sure you understand the distinction between agents and predictive non-agents, and this is very important for FAI work.
Like that’s simple? How exactly do you make an AI that isn’t an agent? With no goals, why does it do anything?
The entire field of machine learning is about building practical approximations of Solomonoff inductions. Algorithms which can predict things and which are not agents. Agents are just special cases of prediction algorithms, where they take the action that has the highest predicted reward.
But why are plans that take less computing power to come up with more likely to be safe?
Because they are plans that less powerful intelligences could have come up with. We don’t worry about humans taking over the world, because they aren’t intelligent enough. The danger of superintelligence is because it could be far more powerful than us. This is a limit on that power.
Besides, if it calculates that searching for simple solutions is likely not going to meet the 90% criteria, it can forgoe that and jump straight to complicated ones.
That’s a feature. We don’t know how much computing power is necessary. We just want it to minimize it.
EY is talking about oracles which answer questions. I am just talking about prediction.
But yes you do have a point that building a powerful predictive AI is not completely trivial. But it’s certainly possible. If you have infinite computing power, you can just run Solomonoff induction.
Realistically we will have to find good approximations, and this might require using agenty-AI. And if so we will have to do work on controlling that AI. I believe this is possible, because it’s a simple domain with a well specified goal, and no output channels except a single number.
Anyway, the other AI judge isn’t an important or necessary part of my idea. I just wanted to have a simple outside judge of solutions. You could make the judge internal, have the AI use it’s own probability estimates to decide when to output a solution. It is essentially doing that already by trying to predict what the judge will say to it’s plan. The judge is redundant.
Which is uncomputable, and an approximation would presumably benefit from increased computing power.
Like that’s simple? How exactly do you make an AI that isn’t an agent? With no goals, why does it do anything?
But why are plans that take less computing power to come up with more likely to be safe? Besides, if it calculates that searching for simple solutions is likely not going to meet the 90% criteria, it can forgoe that and jump straight to complicated ones.
Your idea is similar to http://lesswrong.com/lw/854/satisficers_want_to_become_maximisers/, have you seen that?
I gave the simplest possible counter example to your objection. I never proposed that we actually use pure Solomonoff induction.
EDIT: I realize you said something different. You implied that an approximation of Solomonoff induction would benefit from more computing power, and so would act as an agent to obtain it. This is totally incorrect. Solomonoff induction can be approximated in various ways by bounding the run time of the programs, or using simpler models instead of computer programs, etc. None of these create any agentness. They still just do prediction. I’m not sure you understand the distinction between agents and predictive non-agents, and this is very important for FAI work.
The entire field of machine learning is about building practical approximations of Solomonoff inductions. Algorithms which can predict things and which are not agents. Agents are just special cases of prediction algorithms, where they take the action that has the highest predicted reward.
Because they are plans that less powerful intelligences could have come up with. We don’t worry about humans taking over the world, because they aren’t intelligent enough. The danger of superintelligence is because it could be far more powerful than us. This is a limit on that power.
That’s a feature. We don’t know how much computing power is necessary. We just want it to minimize it.
I think several of your objections were addressed in http://lesswrong.com/lw/tj/dreams_of_friendliness/. That’s pretty much where I’m coming from. Do you have good responses to the arguments there?
Response
EY is talking about oracles which answer questions. I am just talking about prediction.
But yes you do have a point that building a powerful predictive AI is not completely trivial. But it’s certainly possible. If you have infinite computing power, you can just run Solomonoff induction.
Realistically we will have to find good approximations, and this might require using agenty-AI. And if so we will have to do work on controlling that AI. I believe this is possible, because it’s a simple domain with a well specified goal, and no output channels except a single number.
Anyway, the other AI judge isn’t an important or necessary part of my idea. I just wanted to have a simple outside judge of solutions. You could make the judge internal, have the AI use it’s own probability estimates to decide when to output a solution. It is essentially doing that already by trying to predict what the judge will say to it’s plan. The judge is redundant.