Models, myths, dreams, and Cheshire cat grins

“she has of­ten seen a cat with­out a grin but never a grin with­out a cat”

Let’s have a very sim­ple model. There’s a boolean, , which mea­sures whether there’s a cat around. There’s a nat­u­ral num­ber , which counts the num­ber of legs on the cat, and a boolean , which checks whether the cat is grin­ning (or not).

There are a few ob­vi­ous rules in the model, to make it com­pat­i­ble with real life:

  • .

  • .

Or, in other words, if there’s no cat, then there are zero cat legs and no grin.

And that’s true about re­al­ity. But sup­pose we have trained a neu­ral net to au­to­mat­i­cally find the val­ues of , , and . Then it’s perfectly con­ceiv­able that some­thing might trig­ger the out­puts and si­mul­ta­neously: a grin with­out any cat to hang it on.

Ad­ver­sar­ial examples

Ad­ver­sar­ial ex­am­ples of­ten seem to be­have this way. Take for ex­am­ple this ad­ver­sar­ial ex­am­ple of a pig clas­sified as an air­liner:

Imag­ine that the neu­ral net was not only clas­sify­ing “pig” and “air­liner”, but other things like “has wings” and “has fur”.

Then the “pig-air­liner” doesn’t have wings, and has fur, which are fea­tures of pigs but not air­lin­ers. Of course, you could build an ad­ver­sar­ial model that also breaks “has wings” and “has fur”, but, hope­fully, the more fea­tures that need to be faked, the harder it would be­come.

This sug­gests that, as al­gorithms get smarter, they will be­come more adept at avoid­ing ad­ver­sar­ial ex­am­ples—as long as the ul­ti­mate ques­tion is clear. In our real world, the cat­e­gories of pigs and air­lin­ers are pretty sharply dis­tinct.

We run into prob­lems, though, if the con­cepts are less clear—such as what might hap­pens to pigs and air­lin­ers if the al­gorithm op­ti­mises them, or how the al­gorithm might clas­sify un­der­defined con­cepts like “hu­man hap­piness”.

Myths and dreams

Define the fol­low­ing booleans: de­tects the pres­ence of a liv­ing hu­man head, a liv­ing hu­man body, a liv­ing jackal head, a liv­ing jackal body.

In our world real world we gen­er­ally have and . But set the fol­low­ing val­ues:

and you have the god Anu­bis.

Similarly, what is a dragon? Well, it’s an en­tity such that the fol­low­ing are all true:

And, even though those fea­tures never go to­gether in the real world, we can put them to­gether in our imag­i­na­tion, and get a dragon.

Note that “is fly­ing” seems more fun­da­men­tal to a dragon than “has wings”, thus all the wingless drag­ons that fly “by magic[1]”. Our imag­i­na­tion seem com­fortable with such com­bi­na­tions.

Dreams are always be­wil­der­ing upon awak­en­ing, be­cause they also com­bine con­tra­dic­tory as­sump­tions. But these com­bi­na­tions are of­ten be­yond what our imag­i­na­tions are com­fortable with, so we get things like meet­ing your mother—who is also a wolf—and hand­ing Dubai to her over the tea cups (that con­tain milk and fear).

“Alice in Won­der­land” seems to be in be­tween the wild in­co­her­ence of dream fea­tures, and the more re­stricted in­con­sis­tency of sto­ries and imag­i­na­tion.

  1. Not that any real crea­ture that size could fly with those wings any­way. ↩︎