The simplest way to solve the jester’s puzzle is to make a table of the four cases (where the frog is, where the true inscription is), then determine for each case whether the inscriptions are in fact true or false as required for that case. (All the while making la-la-la-can’t-hear-you noises at any doubts one might have about whether self-reference can be formalised at all.) The conclusion is that the first box has the frog and the true inscription. That assumes that the jester was honest in stating his puzzle, but given his shock at the outcome of the king’s puzzle, that appears to be so.
But can self-reference be formalised? How, for example, do two perfect reasoners negotiate a deal? In general, how can two perfect reasoners in an adversarial situation ever interpret the other’s words as anything but noise?
“Are you the sort of man who would put the poison into his own goblet or his enemy’s? Now, a clever man would put the poison into his own goblet because he would know that only a great fool would reach for what he was given. I am not a great fool so I can clearly not choose the wine in front of you...But you must have known I was not a great fool; you would have counted on it, so I can clearly not choose the wine in front of me.” …etc.
Or consider a conversation between an FAI that wants to keep the world safe for humans, and a UFAI that wants to turn the world into paperclips.
I understand that the “turn the world into paperclips” thing comes from a writing of Eliezer, but it is shorthand for a very unlikely scenario. Moreover, this site has gotten really far away from actually dealing with the problems that an unfriendly AGI is likely to cause. Instead, it seems to deal with stupid human problems, foibles, unreason, etc.
The problem with this, as I see it, is that humans are a diverse group, and what’s rational for those without much brainpower is totally irrational for those with a lot of brainpower.
If you have the capacity to develop a functional AGI with mirror neurons, then that’s what you should be doing. If you have the capacity to develop a part of such an AGI, then that’s what you should be doing.
If you don’t have such a capacity (in brains, or in some other necessary capital, such as monetary/material capital), then you shouldn’t waste your time trying to shape the post-singularity future.
Most of this site is word games that point out that words are inadequate communicators, and generally only used to signal status of one primate to another. True enough, but not related to the domain in question: how to stop (or “make less likely”) homo economicus (var. sociopathicus) from killing/displacing the lesser primates?
First, we must realize that sociopathy is our primary problem. We are entering the singularity using sociopath-defined social systems, sociopath-controlled social systems, and sociopath-populated social systems. The tools of liberal democracy have been abandoned, incrementally, due to the former facts / situation(s). Now it’s true that I’ve used a lot of what Marvin Minsky (more succinctly than this site) called “suitcase words.”
You can either debate me in what Ray Kurzweil calls “slow, serial, and imprecise” language (again, more succinctly than this site, in his book “The Age of Spiritual Machines”), or you can “most favorably” interpret what I say, and realize I’m right, and make your way to the fire escape.
Time is short. Human stupidity is long. We will all likely perish. Make haste.
Just to clarify: I think it’s smart to build AGI right now that starts off not knowing much, build it with a weak robot body that can interact with the world towards a goal, allow unlimited self-improvement, and raise the child with love and respect. I think it’s also good to have multiple such “Mind Children.” The more there are, the more likelihood that the non-sociopaths will be able to ameliorate the damage from the sociopaths, both by destroying them, and by designing systems that reward them enough so that the destructiveness of their sociopathy is not fully realized (as humans have tried and failed to do with their own systems).
The simplest way to solve the jester’s puzzle is to make a table of the four cases (where the frog is, where the true inscription is), then determine for each case whether the inscriptions are in fact true or false as required for that case. (All the while making la-la-la-can’t-hear-you noises at any doubts one might have about whether self-reference can be formalised at all.) The conclusion is that the first box has the frog and the true inscription. That assumes that the jester was honest in stating his puzzle, but given his shock at the outcome of the king’s puzzle, that appears to be so.
But can self-reference be formalised? How, for example, do two perfect reasoners negotiate a deal? In general, how can two perfect reasoners in an adversarial situation ever interpret the other’s words as anything but noise?
“Are you the sort of man who would put the poison into his own goblet or his enemy’s? Now, a clever man would put the poison into his own goblet because he would know that only a great fool would reach for what he was given. I am not a great fool so I can clearly not choose the wine in front of you...But you must have known I was not a great fool; you would have counted on it, so I can clearly not choose the wine in front of me.” …etc.
Or consider a conversation between an FAI that wants to keep the world safe for humans, and a UFAI that wants to turn the world into paperclips.
I understand that the “turn the world into paperclips” thing comes from a writing of Eliezer, but it is shorthand for a very unlikely scenario. Moreover, this site has gotten really far away from actually dealing with the problems that an unfriendly AGI is likely to cause. Instead, it seems to deal with stupid human problems, foibles, unreason, etc.
The problem with this, as I see it, is that humans are a diverse group, and what’s rational for those without much brainpower is totally irrational for those with a lot of brainpower.
If you have the capacity to develop a functional AGI with mirror neurons, then that’s what you should be doing. If you have the capacity to develop a part of such an AGI, then that’s what you should be doing.
If you don’t have such a capacity (in brains, or in some other necessary capital, such as monetary/material capital), then you shouldn’t waste your time trying to shape the post-singularity future.
Most of this site is word games that point out that words are inadequate communicators, and generally only used to signal status of one primate to another. True enough, but not related to the domain in question: how to stop (or “make less likely”) homo economicus (var. sociopathicus) from killing/displacing the lesser primates?
First, we must realize that sociopathy is our primary problem. We are entering the singularity using sociopath-defined social systems, sociopath-controlled social systems, and sociopath-populated social systems. The tools of liberal democracy have been abandoned, incrementally, due to the former facts / situation(s). Now it’s true that I’ve used a lot of what Marvin Minsky (more succinctly than this site) called “suitcase words.”
You can either debate me in what Ray Kurzweil calls “slow, serial, and imprecise” language (again, more succinctly than this site, in his book “The Age of Spiritual Machines”), or you can “most favorably” interpret what I say, and realize I’m right, and make your way to the fire escape.
Time is short. Human stupidity is long. We will all likely perish. Make haste.
Just to clarify: I think it’s smart to build AGI right now that starts off not knowing much, build it with a weak robot body that can interact with the world towards a goal, allow unlimited self-improvement, and raise the child with love and respect. I think it’s also good to have multiple such “Mind Children.” The more there are, the more likelihood that the non-sociopaths will be able to ameliorate the damage from the sociopaths, both by destroying them, and by designing systems that reward them enough so that the destructiveness of their sociopathy is not fully realized (as humans have tried and failed to do with their own systems).