Designing intelligent machines is an incremental process
Launching a particular design (version x.y.z) is a one-time event. From that perspective, AI is not a web site. It’s a rocket.
Let me rephrase: the math behind AI will take some time do be discovered. The discovery process will be incremental, with some important milestones (“Conciousness Reduced”, “Bypassing Löbs Theorem”…). At one point, the FAI team will have realized they have found the solution. They will know what to do to make Friendly AI happen. This realization will also be incremental. They will likely discuss the solution before everyone agrees it is the right one.
One key point about that solution is that it is provably safe. To prove a solution is safe, you have to describe it with mathematical precision. When you can do that, you can also write a computer program that implements that solution. And that’s exactly what they will do, because they don’t want to leave any room for human error.
At one point, the program that implements the solution will be build. Again, this will be incremental. There will be bugs, which will be corrected. Then there will be no bug left. Then the FAI team will, gradually, realize there is no bug.
And then, and only then, someone will push the red button, type build && run or something. The provably correct program will then execute the provably correct solution to make FAI happen. The ultimate milestone.
We’re not finished. The program runs, but we don’t have a super-intelligence capable of fixing our problems yet. But we will. Singularity will happen, gradually, self improvement by self improvement. Humans will probably be kept in the loop as well, for preference extraction. But they won’t be holding the steering wheel.
That looks like a very detailed story, but the core point is very simple: design, write the program, and prove (possibly iteratively), and only then, launch. Because running a program that’s still in development is not safe —the video is a joke, but its core point stands: why in the name of Azathoth did the guy launched a self-improving sentient program without knowing for sure it will be safe? If we win, that sure won’t be by trial and error.
We don’t seem to agree. This isn’t how technology gets built. Nobody proved the first aeroplane was safe. Nobody proved the first space rocket was safe. They weren’t safe. No areoplanes or spaceships have ever been proven safe. You may be able to prove some things about the design process, but not safety—security doesn’t work like that.
There is something called “provable security” in cryptography—but it doesn’t really mean what its title says. It means that you can prove something relating to security in a particular model—not that you can prove something is safe.
On this day of the Great Launch, we have won. That’s the whole point of the thread.
If we didn’t take the extreme precautions I mentioned, we would most certainly have lost. That’s the Singularity Institute Scary Idea, which I actually believe.
You, on the other hand, say that we will most certainly not take those drastic precautions. And you know the worst part? I agree. Which take us back to square one: by default, we’re doomed. (There. You’ve done it. Now I’m scared.)
Evolution isn’t really a win/lose game. Organisms succeed in perpetuating themselves—and the things they value—to varying degrees. Humans seem set to survive in the history books, but our overall influence on the future looks set to be quite marginal—due to the substantial influence of technological determinism—except in the unlikely case where we permanently destroy civilization. Of course we can still try—but I just don’t think our best shot looks much like what you described. Of course it might be fun if we had time for all that stuff about provable security—but at the moment, the situation looks a lot like a frantic race, and that looks like exactly the sort of thing that will be first to go up against the wall.
I said I considered destroying “civilization” to be unlikely. Going by this:
progressing toward advanced AGI without a design for “provably non-dangerous AGI” (or something closely analogous, often called “Friendly AI” in SIAI lingo) is highly likely to lead to an involuntary end for the human race.
...the scary idea claims to be about “the human race”. I don’t define “civilization” in a human-centric way—so I don’t class those as being the same thing—for instance, I think that civilization might well continue after an “involuntary” robot takeover.
Well, a civilization with humanity all dead is pretty much certainly not what we want. I don’t care if in the grand scheme of things, this isn’t a win/lose game. I think I have something like a utility function, and I want it maximized, period.
Back to my question: do you see any other path to building a future we want than the one I described?
Well, a civilization with humanity all dead is pretty much certainly not what we want.
Well, humans will live on via historical simulations, with pretty good probability. Humans won’t remain the dominant species, though. Those hoping for that have unrealistic expectations. Machines won’t remain human tools, they are likely to be in charge.
I think I have something like a utility function, and I want it maximized, period.
Sure, but it’s you and billions of other organisms—with often-conflicting futures in mind—and so most won’t have things their way.
do you see any other path to building a future we want than the one I described?
IIRC, your proposal put considerable emphasis on proof. We’ll prove what we can, but proof often lags far behind the leading edge of computer science. There are many other approaches to building mission critical systems incrementally—I expect we will make more use those.
Historical simulations: assuming it preserves identity etc, why not…
Utility function: I know that my chances of maximizing my utiliy function are quite… slim, to say the least.
Path to best future(humanity): proofs do not lag so far behind right now. Modern type systems are now pretty good, and we have proof assistants that makes the “prove your whole program” quite feasible –though not cheap yet. Plus, the leading edge is generally the easiest to prove, because it tends to lie on solid mathematical ground. We don’t do proofs because they’re generally expensive, and we use ancient technologies that leak lots of low-level details, and make proofs much harder. (I program for a living.)
But I see at least the possibility of a slightly different path: still take precautions, just don’t prove the thing.
Oh, and I forgot: if we solve safety before capability, incrementally designing the AI by trial-and-error would be quite reasonable. The definite milestone will be harder to define in this case. I guess I’ll have to update a bit.
Launching a particular design (version x.y.z) is a one-time event. From that perspective, AI is not a web site. It’s a rocket.
Let me rephrase: the math behind AI will take some time do be discovered. The discovery process will be incremental, with some important milestones (“Conciousness Reduced”, “Bypassing Löbs Theorem”…). At one point, the FAI team will have realized they have found the solution. They will know what to do to make Friendly AI happen. This realization will also be incremental. They will likely discuss the solution before everyone agrees it is the right one.
One key point about that solution is that it is provably safe. To prove a solution is safe, you have to describe it with mathematical precision. When you can do that, you can also write a computer program that implements that solution. And that’s exactly what they will do, because they don’t want to leave any room for human error.
At one point, the program that implements the solution will be build. Again, this will be incremental. There will be bugs, which will be corrected. Then there will be no bug left. Then the FAI team will, gradually, realize there is no bug.
And then, and only then, someone will push the red button, type
build && run
or something. The provably correct program will then execute the provably correct solution to make FAI happen. The ultimate milestone.We’re not finished. The program runs, but we don’t have a super-intelligence capable of fixing our problems yet. But we will. Singularity will happen, gradually, self improvement by self improvement. Humans will probably be kept in the loop as well, for preference extraction. But they won’t be holding the steering wheel.
That looks like a very detailed story, but the core point is very simple: design, write the program, and prove (possibly iteratively), and only then, launch. Because running a program that’s still in development is not safe —the video is a joke, but its core point stands: why in the name of Azathoth did the guy launched a self-improving sentient program without knowing for sure it will be safe? If we win, that sure won’t be by trial and error.
We don’t seem to agree. This isn’t how technology gets built. Nobody proved the first aeroplane was safe. Nobody proved the first space rocket was safe. They weren’t safe. No areoplanes or spaceships have ever been proven safe. You may be able to prove some things about the design process, but not safety—security doesn’t work like that.
There is something called “provable security” in cryptography—but it doesn’t really mean what its title says. It means that you can prove something relating to security in a particular model—not that you can prove something is safe.
I made 2 assumptions here
On this day of the Great Launch, we have won. That’s the whole point of the thread.
If we didn’t take the extreme precautions I mentioned, we would most certainly have lost. That’s the Singularity Institute Scary Idea, which I actually believe.
You, on the other hand, say that we will most certainly not take those drastic precautions. And you know the worst part? I agree. Which take us back to square one: by default, we’re doomed. (There. You’ve done it. Now I’m scared.)
Evolution isn’t really a win/lose game. Organisms succeed in perpetuating themselves—and the things they value—to varying degrees. Humans seem set to survive in the history books, but our overall influence on the future looks set to be quite marginal—due to the substantial influence of technological determinism—except in the unlikely case where we permanently destroy civilization. Of course we can still try—but I just don’t think our best shot looks much like what you described. Of course it might be fun if we had time for all that stuff about provable security—but at the moment, the situation looks a lot like a frantic race, and that looks like exactly the sort of thing that will be first to go up against the wall.
Are you saying that you don’t buy the scary idea?
I said I considered destroying “civilization” to be unlikely. Going by this:
...the scary idea claims to be about “the human race”. I don’t define “civilization” in a human-centric way—so I don’t class those as being the same thing—for instance, I think that civilization might well continue after an “involuntary” robot takeover.
Well, a civilization with humanity all dead is pretty much certainly not what we want. I don’t care if in the grand scheme of things, this isn’t a win/lose game. I think I have something like a utility function, and I want it maximized, period.
Back to my question: do you see any other path to building a future we want than the one I described?
Well, humans will live on via historical simulations, with pretty good probability. Humans won’t remain the dominant species, though. Those hoping for that have unrealistic expectations. Machines won’t remain human tools, they are likely to be in charge.
Sure, but it’s you and billions of other organisms—with often-conflicting futures in mind—and so most won’t have things their way.
IIRC, your proposal put considerable emphasis on proof. We’ll prove what we can, but proof often lags far behind the leading edge of computer science. There are many other approaches to building mission critical systems incrementally—I expect we will make more use those.
Historical simulations: assuming it preserves identity etc, why not…
Utility function: I know that my chances of maximizing my utiliy function are quite… slim, to say the least.
Path to best future(humanity): proofs do not lag so far behind right now. Modern type systems are now pretty good, and we have proof assistants that makes the “prove your whole program” quite feasible –though not cheap yet. Plus, the leading edge is generally the easiest to prove, because it tends to lie on solid mathematical ground. We don’t do proofs because they’re generally expensive, and we use ancient technologies that leak lots of low-level details, and make proofs much harder. (I program for a living.)
But I see at least the possibility of a slightly different path: still take precautions, just don’t prove the thing.
Oh, and I forgot: if we solve safety before capability, incrementally designing the AI by trial-and-error would be quite reasonable. The definite milestone will be harder to define in this case. I guess I’ll have to update a bit.