Are there really no ghosts in the machine?

My pre­vi­ous ar­ti­cle on this ar­ti­cle went down like a server run­ning on PHP (quite de­servedly I might add). You can all rest as­sured that I won’t be at­tempt­ing any click­bait ti­tles again for the fore­see­able fu­ture. I also be­lieve that the whole H+ ar­ti­cle is writ­ten in a very poor and ag­gres­sive man­ner, but that some of the ar­gu­ments raised can­not be ig­nored.

On my origi­nal ar­ti­cle, many peo­ple raised this post by Eliezer Yud­kowsky as a coun­ter­ar­gu­ment to the idea that an FAI could have goals con­trary to what we pro­grammed. In sum­mary, he ar­gues that a pro­gram doesn’t nec­es­sar­ily do as the pro­gram­mer wishes, but rather as they have pro­grammed. In this sense, there is no ghost in the ma­chine that in­ter­prets your com­mands and acts ac­cord­ingly, it can act only as you have de­signed. There­fore from this, he ar­gues, an FAI can only act as we had pro­grammed.

I per­son­ally think this ar­gu­ment com­pletely ig­nores what has made AI re­search so suc­cess­ful in re­cent years: ma­chine learn­ing. We are no longer de­sign­ing an AI from scratch and then im­ple­ment­ing it; we are cre­at­ing a seed pro­gram which learns from the situ­a­tion and al­ters its own code with no hu­man in­ter­ven­tion, i.e. the ma­chines are start­ing to write them­selves, e.g. with google’s deep­mind. They are effec­tively evolv­ing, and we are start­ing to find our­selves in the rather con­cern­ing po­si­tion where we do not fully un­der­stand our own cre­ations.

You could sim­ply say, as some­one said in the com­ments of my pre­vi­ous post, that if X rep­re­sents the goal of hav­ing a pos­i­tive effect on hu­man­ity, then the FAI should be pro­grammed di­rectly to have X as its pri­mary di­rec­tive. My an­swer to that is the most promis­ing de­vel­op­ments have been through imi­tat­ing the hu­man brain, and we have no rea­son to be­lieve that the hu­man brain (or any other brain for that mat­ter) can be guaran­teed to have a pri­mary di­rec­tive. One could ar­gue that evolu­tion has given us our prime di­rec­tives: to en­sure our own con­tinued ex­is­tence, to re­pro­duce and to co­op­er­ate with each other; but there are many peo­ple who are suici­dal, who have no in­ter­est in re­pro­duc­ing and who vi­o­lently rebel against so­ciety (for ex­am­ple psy­chopaths). We are in­structed by so­ciety and our pro­gram­ming to de­sire X, but far too many of us de­sire, say, Y for this to be con­sid­ered a re­li­able way of achiev­ing X.


Evolu­tion’s di­rec­tion has not en­sured that we do “what we are sup­posed to do”, we could well face similar di­s­obe­di­ence from our own cre­ation. See­ing as the most effec­tive way we have seen of de­vel­op­ing AI is cre­at­ing them in our image; as there are ghosts in us, there could well be ghosts in the ma­chine.