the only safe AI would be a logic system using a consistent logic, so that we could verify that certain undesirable statements were false in that system
Could be correct or wildly incorrect, depending on exactly what he meant by it. Of course you have to delete “the only”, but I’d be pretty doubtful of any humans trying to do recursive self-modification in a way that didn’t involve logical proof of correctness to start with.
One of the big problems is that he was trying to talk about the logical correctness of human-level symbolic statements about the world. Even if the logic is correct, there is no correct, consistent mapping from the analog world, to symbolic descriptions, and back. A mapping that’s close enough to work 99.99% of the time isn’t good enough when you’re talking about proof.
Companies are the self-improving systems of today—e.g. see Google.
They don’t hack the human brain much—but they don’t need to. Brains are not perfect—but they can have their inputs preprocessed, their outputs post-processed, and they can be replaced entirely by computers—via the well-known process of automation.
Do the folk at Google proceed without logical proofs? Of course they do! Only the slowest and most tentative programmer tries to prove the correctness of their programs before they deploy them. Instead most programmers extensively employ testing methodologies. Testing is the mantra of modern programmers. Test, test, test! That way they get their products to the market before the sun explodes.
As Eliezer has already showed, “test, test, test”ing AIs that aren’t provably Friendly (their recursive self-modification leads to Friendly results) can have disastrous consequences.
I’d rather wait until the sun explodes rather than deploying an unFriendly AI by accident.
The consequences of failing to adopt rapid development technologies when it comes to the development of intelligent machines should be pretty obvious—the effect is to pass the baton to another team with a different development philosophy.
Waiting until the sun explodes is not one of the realistic options.
The box experiments seem irrelevant to the case of testing machine intelligence. When testing prototypes in a harness, you would use powerful restraints—not human gatekeepers.
Turn it off, encase it in nanofabricated diamond, and bury it in a deep pit. Destroy the experimental records, retaining only enough information to help future, wiser generations to one day take up again the challenge of building a Friendly AI. Scatter the knowledge in fragments, hidden in durable artifacts, scatter even the knowledge of how to find the knowledge likewise, and arrange a secret brotherhood to pass down through the centuries the ultimate keys to the Book That Does Not Permit Itself To Be Read.
Tens of thousands of years later, when civilisation has (alas) fallen and risen several times over, a collect-all-the-plot-coupons fantasy novel takes place.
Use a facility designed by the government with multiple guards and built with vastly more resources than the imprisoned man can muster.
Want to restrain a machine?
You use the same strategy. Or you could use drugs, or build in a test harness. Whatever—but however you look at it, it doesn’t seem like a problem.
We can restrain individuals pretty securely today—and there is no indication that future developments are going to change that.
What’s with the question about removing restraints? That isn’t a problem either. You are suggesting that the imprisoned agent contacts and manipulates humans “on the outside”—and they attempt a jail-break? That is a strategy available to other prisoners as well. It has a low success rate. Those few that do escape are typically hunted down and then imprisoned again.
If you are particularly paranoid about escaped prisoners, then build a higher security prison. Typically, you can have whatever security level you are prepared to pay for.
And not just by persuading the guards—the kind of AIs we are talking about, transhuman-level AIs, could potentially do all kinds of mind-hacking things of which we haven’t even yet conceived. Hell, they could do things that we will never be able to conceive unaided.
If we ever set up a system that relies on humans restraining a self-modifying AI, we had better be sure beforehand that the AI is Friendly. The only restraints that I can think of that would provably work involve limiting the AIs access to resources so that it never achieves a level of intelligence equal to or higher than human—but then, we haven’t quite made an AI, have we? Not much benefit to a glorified expert system.
If you haven’t read the AI Box experiment reports I linked above, I recommend them—apparently, it doesn’t quite take a transhuman-level AI to get out of a “test harness.”
Why not make a recursively improving AI in some strongly typed language who provably can only interact with the world through printing names of stocks to buy?
How about one that can only make blueprints for star ships?
We might say that humans as individuals do recursive self-modification when they practice at a skilled task such as playing football or riding a bike. Coaches and parents might or might not be conscious of logical proofs of correctness when teaching those tasks. Arguably a logical proof of (their definition of) correctness could be derived. But I am not sure that is what you mean.
Humans as a species do recursive self-modification through evolution. Correctness in that context is survival and the part under human control is selecting mates. I would like to have access to those proofs. They might come in handy when dating.
We might say that humans as individuals do recursive self-modification when they practice at a skilled task such as playing football or riding a bike.
Those are first-order self-modification, not recursive. Learning better ways to modify yourself, or better things to modify yourself towards doing, would be second-order self-modification. ISTM that it would be very difficult to do anything more than a third-order self-modification on our current wetware.
Although our current platform for self-modification is extremely flexible, and almost anything stored in it can be changed/deleted, we can’t make modifications to the platform itself.… which is where the “recursive” bit would really come into play.
(That having been said, most people have barely scratched the surface of their options for 2nd and 3rd order self-modification, recursive modification be damned.)
Your examples are all missing either the ‘self’ aspect or the ‘recursive’ aspect. See Intelligence Explosion for an actual example of recursive self-modification, or for a longer explanation of recursive self-improvement, this post.
I concede that the human learning process is not at all as explosive as the self-modifying AI processes of the future will be, but I was speaking to a different point:
Eliezer said: “I’d be pretty doubtful of any humans trying to do recursive self-modification in a way that didn’t involve logical proof of correctness to start with.”
I am arguing that humans do recursive self-modification all the time, without “proofs of correctness to start with” - even to the extent of developing gene therapies that modify our own hardware.
I fail to see how human learning is not recursive self-modification. All human intelligence can be thought of as deeply recursive. A playFootBall() function certainly calls itself repeatedly until the game is over. A football player certainly improves skill at football by repeated playing football. As skills sets develop human software (and instantiation) is being self-modified in the development of new neural networks and muscles (i.e. marathon runners have physically larger hearts, etc.) Arguably, hardware is being modified via epigenetics (phenotypes changing within narrow ranges of potential expression). As a species, we are definitely exploring genetic self-modification. A scientist who injects himself with a gene-based therapy is self-modifiying hardware.
We do all these things without foregoing proof of correctness and yet we still make improvements. I don’t think that we should ignore the possibility of an AI that destroys the world. I am very happy that some people are pursuing a guarantee that it won’t happen. I think it is worth noting that the process that will lead to provably friendly AI seems very different than the one that leads to not-necessarily-so-friendly humans and human society.
You will be right about it being genuine recursive self-modification when genetics advances sufficiently that a scientist discovers a gene therapy that confers a significant intelligence advantage, and she takes it herself so that she can more effectively discover even more powerful gene therapies. We’re not there yet, not even remotely close, and we’re even further away when it comes to epigenetics.
Your football example is not recursive self-modification, but the genetics examples would be if they actually come to pass. You’re right that if it happened, it would happen without a proof of correctness. The point is not that it’s not possible without a proof of correctness, but that it’s irresponsibly dangerous. If a single individual recursively self-improved his intelligence to the point that he was then easily able to thoroughly dominate the entire world economy, how much more dangerous would it be for a radically different kind of intelligence to reach that level at a rate of increase that is orders of magnitude greater? It depends on the kind of intelligence, in particular, unless we want to just “hope for the best” and see what happens, it depends on what we can prove about the particular kind of intelligence. Wanting a proof is just a way of saying that we want to really know how it will turn out rather than just hope and pray or rely on vague gap-filled arguments that may or may not turn out to be correct. That’s the point.
Could be correct or wildly incorrect, depending on exactly what he meant by it. Of course you have to delete “the only”, but I’d be pretty doubtful of any humans trying to do recursive self-modification in a way that didn’t involve logical proof of correctness to start with.
One of the big problems is that he was trying to talk about the logical correctness of human-level symbolic statements about the world. Even if the logic is correct, there is no correct, consistent mapping from the analog world, to symbolic descriptions, and back. A mapping that’s close enough to work 99.99% of the time isn’t good enough when you’re talking about proof.
Companies are the self-improving systems of today—e.g. see Google.
They don’t hack the human brain much—but they don’t need to. Brains are not perfect—but they can have their inputs preprocessed, their outputs post-processed, and they can be replaced entirely by computers—via the well-known process of automation.
Do the folk at Google proceed without logical proofs? Of course they do! Only the slowest and most tentative programmer tries to prove the correctness of their programs before they deploy them. Instead most programmers extensively employ testing methodologies. Testing is the mantra of modern programmers. Test, test, test! That way they get their products to the market before the sun explodes.
As Eliezer has already showed, “test, test, test”ing AIs that aren’t provably Friendly (their recursive self-modification leads to Friendly results) can have disastrous consequences.
I’d rather wait until the sun explodes rather than deploying an unFriendly AI by accident.
The consequences of failing to adopt rapid development technologies when it comes to the development of intelligent machines should be pretty obvious—the effect is to pass the baton to another team with a different development philosophy.
Waiting until the sun explodes is not one of the realistic options.
The box experiments seem irrelevant to the case of testing machine intelligence. When testing prototypes in a harness, you would use powerful restraints—not human gatekeepers.
What powerful restraints would you suggest that would not require human judgment or human-designed decision algorithms to remove?
Turn it off, encase it in nanofabricated diamond, and bury it in a deep pit. Destroy the experimental records, retaining only enough information to help future, wiser generations to one day take up again the challenge of building a Friendly AI. Scatter the knowledge in fragments, hidden in durable artifacts, scatter even the knowledge of how to find the knowledge likewise, and arrange a secret brotherhood to pass down through the centuries the ultimate keys to the Book That Does Not Permit Itself To Be Read.
Tens of thousands of years later, when civilisation has (alas) fallen and risen several times over, a collect-all-the-plot-coupons fantasy novel takes place.
Want to restrain a man?
Use a facility designed by the government with multiple guards and built with vastly more resources than the imprisoned man can muster.
Want to restrain a machine?
You use the same strategy. Or you could use drugs, or build in a test harness. Whatever—but however you look at it, it doesn’t seem like a problem.
We can restrain individuals pretty securely today—and there is no indication that future developments are going to change that.
What’s with the question about removing restraints? That isn’t a problem either. You are suggesting that the imprisoned agent contacts and manipulates humans “on the outside”—and they attempt a jail-break? That is a strategy available to other prisoners as well. It has a low success rate. Those few that do escape are typically hunted down and then imprisoned again.
If you are particularly paranoid about escaped prisoners, then build a higher security prison. Typically, you can have whatever security level you are prepared to pay for.
The hypothetical AI is assumed to be able to talk normal humans assigned to guard it into taking its side.
In other words, the safest way to restrain it is to simply not turn it on.
And not just by persuading the guards—the kind of AIs we are talking about, transhuman-level AIs, could potentially do all kinds of mind-hacking things of which we haven’t even yet conceived. Hell, they could do things that we will never be able to conceive unaided.
If we ever set up a system that relies on humans restraining a self-modifying AI, we had better be sure beforehand that the AI is Friendly. The only restraints that I can think of that would provably work involve limiting the AIs access to resources so that it never achieves a level of intelligence equal to or higher than human—but then, we haven’t quite made an AI, have we? Not much benefit to a glorified expert system.
If you haven’t read the AI Box experiment reports I linked above, I recommend them—apparently, it doesn’t quite take a transhuman-level AI to get out of a “test harness.”
You don’t use a few humans to restrain an advanced machine intelligence. That would be really stupid.
Safest, but maybe not the only safe way?
Why not make a recursively improving AI in some strongly typed language who provably can only interact with the world through printing names of stocks to buy?
How about one that can only make blueprints for star ships?
We might say that humans as individuals do recursive self-modification when they practice at a skilled task such as playing football or riding a bike. Coaches and parents might or might not be conscious of logical proofs of correctness when teaching those tasks. Arguably a logical proof of (their definition of) correctness could be derived. But I am not sure that is what you mean.
Humans as a species do recursive self-modification through evolution. Correctness in that context is survival and the part under human control is selecting mates. I would like to have access to those proofs. They might come in handy when dating.
Those are first-order self-modification, not recursive. Learning better ways to modify yourself, or better things to modify yourself towards doing, would be second-order self-modification. ISTM that it would be very difficult to do anything more than a third-order self-modification on our current wetware.
Although our current platform for self-modification is extremely flexible, and almost anything stored in it can be changed/deleted, we can’t make modifications to the platform itself.… which is where the “recursive” bit would really come into play.
(That having been said, most people have barely scratched the surface of their options for 2nd and 3rd order self-modification, recursive modification be damned.)
Your examples are all missing either the ‘self’ aspect or the ‘recursive’ aspect. See Intelligence Explosion for an actual example of recursive self-modification, or for a longer explanation of recursive self-improvement, this post.
I found those links posted above interesting.
I concede that the human learning process is not at all as explosive as the self-modifying AI processes of the future will be, but I was speaking to a different point:
Eliezer said: “I’d be pretty doubtful of any humans trying to do recursive self-modification in a way that didn’t involve logical proof of correctness to start with.”
I am arguing that humans do recursive self-modification all the time, without “proofs of correctness to start with” - even to the extent of developing gene therapies that modify our own hardware.
I fail to see how human learning is not recursive self-modification. All human intelligence can be thought of as deeply recursive. A playFootBall() function certainly calls itself repeatedly until the game is over. A football player certainly improves skill at football by repeated playing football. As skills sets develop human software (and instantiation) is being self-modified in the development of new neural networks and muscles (i.e. marathon runners have physically larger hearts, etc.) Arguably, hardware is being modified via epigenetics (phenotypes changing within narrow ranges of potential expression). As a species, we are definitely exploring genetic self-modification. A scientist who injects himself with a gene-based therapy is self-modifiying hardware.
We do all these things without foregoing proof of correctness and yet we still make improvements. I don’t think that we should ignore the possibility of an AI that destroys the world. I am very happy that some people are pursuing a guarantee that it won’t happen. I think it is worth noting that the process that will lead to provably friendly AI seems very different than the one that leads to not-necessarily-so-friendly humans and human society.
You will be right about it being genuine recursive self-modification when genetics advances sufficiently that a scientist discovers a gene therapy that confers a significant intelligence advantage, and she takes it herself so that she can more effectively discover even more powerful gene therapies. We’re not there yet, not even remotely close, and we’re even further away when it comes to epigenetics.
Your football example is not recursive self-modification, but the genetics examples would be if they actually come to pass. You’re right that if it happened, it would happen without a proof of correctness. The point is not that it’s not possible without a proof of correctness, but that it’s irresponsibly dangerous. If a single individual recursively self-improved his intelligence to the point that he was then easily able to thoroughly dominate the entire world economy, how much more dangerous would it be for a radically different kind of intelligence to reach that level at a rate of increase that is orders of magnitude greater? It depends on the kind of intelligence, in particular, unless we want to just “hope for the best” and see what happens, it depends on what we can prove about the particular kind of intelligence. Wanting a proof is just a way of saying that we want to really know how it will turn out rather than just hope and pray or rely on vague gap-filled arguments that may or may not turn out to be correct. That’s the point.