Its goal is to predict as accurately as it can. Clearly taking over the world and reassigning all computing power to calculate the prediction is the best move.
Think of Solomonoff Induction or some approximation of it. It is not an agent. It just tries every possible hypothesis on that data and does a Bayesian update.
How does it know how long a plan will take to design until it actually designs it? (I’m assuming your “time” is time to design a plan).
It doesn’t know. It needs to predict. But in general humans have a general idea of how solvable a problem is before they solve it. An engineer knows that building a certain kind of machine is possible, long before he works out the exact specification. A computer programmer knows a problem is probably solvable with a certain approach before they work out the exact computer code to produce it.
This AI is highly incentivized to work fast. Searching down the totally wrong tree is punished highly. Trying simpler ideas before more complex ones is rewarded. So is being able to quickly come up with possible solutions that might work, before reviewing them in more depth.
I don’t know exactly what strategies it will use, but it’s utility function is literally to minimize computing power. If you trust the AI to fulfill it’s utility function, then you can trust it will do this to the best of it’s ability.
How do we know the fastest designed plan is the safest? Maybe this AI generates unsafe plans faster than safe ones.
Safety is not guaranteed with this approach. I am fully upfront about this. What it does is minimize optimization. The plan you get will be the stupidest one the AI can come up with. This significantly decreases risk.
Tl;Dr not pessimistic enough.
IMHO extreme pessimism leads to throwing out a huge number of ideas. Some of which might be practical, or lead to more practical approaches. I was extremely pessimistic about FAI for a long time until reading some of the recent proposals to actually attack the problem. None of the current ideas are sufficient, but they show it’s at least approachable.
Think of Solomonoff Induction or some approximation of it.
Which is uncomputable, and an approximation would presumably benefit from increased computing power.
It is not an agent. It just tries every possible hypothesis on that data and does a Bayesian update.
Like that’s simple? How exactly do you make an AI that isn’t an agent? With no goals, why does it do anything?
I don’t know exactly what strategies it will use, but it’s utility function is literally to minimize computing power. If you trust the AI to fulfill it’s utility function, then you can trust it will do this to the best of it’s ability.
But why are plans that take less computing power to come up with more likely to be safe? Besides, if it calculates that searching for simple solutions is likely not going to meet the 90% criteria, it can forgoe that and jump straight to complicated ones.
Which is uncomputable, and an approximation would presumably benefit from increased computing power.
I gave the simplest possible counter example to your objection. I never proposed that we actually use pure Solomonoff induction.
EDIT: I realize you said something different. You implied that an approximation of Solomonoff induction would benefit from more computing power, and so would act as an agent to obtain it. This is totally incorrect. Solomonoff induction can be approximated in various ways by bounding the run time of the programs, or using simpler models instead of computer programs, etc. None of these create any agentness. They still just do prediction. I’m not sure you understand the distinction between agents and predictive non-agents, and this is very important for FAI work.
Like that’s simple? How exactly do you make an AI that isn’t an agent? With no goals, why does it do anything?
The entire field of machine learning is about building practical approximations of Solomonoff inductions. Algorithms which can predict things and which are not agents. Agents are just special cases of prediction algorithms, where they take the action that has the highest predicted reward.
But why are plans that take less computing power to come up with more likely to be safe?
Because they are plans that less powerful intelligences could have come up with. We don’t worry about humans taking over the world, because they aren’t intelligent enough. The danger of superintelligence is because it could be far more powerful than us. This is a limit on that power.
Besides, if it calculates that searching for simple solutions is likely not going to meet the 90% criteria, it can forgoe that and jump straight to complicated ones.
That’s a feature. We don’t know how much computing power is necessary. We just want it to minimize it.
EY is talking about oracles which answer questions. I am just talking about prediction.
But yes you do have a point that building a powerful predictive AI is not completely trivial. But it’s certainly possible. If you have infinite computing power, you can just run Solomonoff induction.
Realistically we will have to find good approximations, and this might require using agenty-AI. And if so we will have to do work on controlling that AI. I believe this is possible, because it’s a simple domain with a well specified goal, and no output channels except a single number.
Anyway, the other AI judge isn’t an important or necessary part of my idea. I just wanted to have a simple outside judge of solutions. You could make the judge internal, have the AI use it’s own probability estimates to decide when to output a solution. It is essentially doing that already by trying to predict what the judge will say to it’s plan. The judge is redundant.
Think of Solomonoff Induction or some approximation of it. It is not an agent. It just tries every possible hypothesis on that data and does a Bayesian update.
It doesn’t know. It needs to predict. But in general humans have a general idea of how solvable a problem is before they solve it. An engineer knows that building a certain kind of machine is possible, long before he works out the exact specification. A computer programmer knows a problem is probably solvable with a certain approach before they work out the exact computer code to produce it.
This AI is highly incentivized to work fast. Searching down the totally wrong tree is punished highly. Trying simpler ideas before more complex ones is rewarded. So is being able to quickly come up with possible solutions that might work, before reviewing them in more depth.
I don’t know exactly what strategies it will use, but it’s utility function is literally to minimize computing power. If you trust the AI to fulfill it’s utility function, then you can trust it will do this to the best of it’s ability.
Safety is not guaranteed with this approach. I am fully upfront about this. What it does is minimize optimization. The plan you get will be the stupidest one the AI can come up with. This significantly decreases risk.
IMHO extreme pessimism leads to throwing out a huge number of ideas. Some of which might be practical, or lead to more practical approaches. I was extremely pessimistic about FAI for a long time until reading some of the recent proposals to actually attack the problem. None of the current ideas are sufficient, but they show it’s at least approachable.
Which is uncomputable, and an approximation would presumably benefit from increased computing power.
Like that’s simple? How exactly do you make an AI that isn’t an agent? With no goals, why does it do anything?
But why are plans that take less computing power to come up with more likely to be safe? Besides, if it calculates that searching for simple solutions is likely not going to meet the 90% criteria, it can forgoe that and jump straight to complicated ones.
Your idea is similar to http://lesswrong.com/lw/854/satisficers_want_to_become_maximisers/, have you seen that?
I gave the simplest possible counter example to your objection. I never proposed that we actually use pure Solomonoff induction.
EDIT: I realize you said something different. You implied that an approximation of Solomonoff induction would benefit from more computing power, and so would act as an agent to obtain it. This is totally incorrect. Solomonoff induction can be approximated in various ways by bounding the run time of the programs, or using simpler models instead of computer programs, etc. None of these create any agentness. They still just do prediction. I’m not sure you understand the distinction between agents and predictive non-agents, and this is very important for FAI work.
The entire field of machine learning is about building practical approximations of Solomonoff inductions. Algorithms which can predict things and which are not agents. Agents are just special cases of prediction algorithms, where they take the action that has the highest predicted reward.
Because they are plans that less powerful intelligences could have come up with. We don’t worry about humans taking over the world, because they aren’t intelligent enough. The danger of superintelligence is because it could be far more powerful than us. This is a limit on that power.
That’s a feature. We don’t know how much computing power is necessary. We just want it to minimize it.
I think several of your objections were addressed in http://lesswrong.com/lw/tj/dreams_of_friendliness/. That’s pretty much where I’m coming from. Do you have good responses to the arguments there?
Response
EY is talking about oracles which answer questions. I am just talking about prediction.
But yes you do have a point that building a powerful predictive AI is not completely trivial. But it’s certainly possible. If you have infinite computing power, you can just run Solomonoff induction.
Realistically we will have to find good approximations, and this might require using agenty-AI. And if so we will have to do work on controlling that AI. I believe this is possible, because it’s a simple domain with a well specified goal, and no output channels except a single number.
Anyway, the other AI judge isn’t an important or necessary part of my idea. I just wanted to have a simple outside judge of solutions. You could make the judge internal, have the AI use it’s own probability estimates to decide when to output a solution. It is essentially doing that already by trying to predict what the judge will say to it’s plan. The judge is redundant.