The Quotation is not the Referent

In clas­si­cal logic, the op­er­a­tional defi­ni­tion of iden­tity is that when­ever ‘A=B’ is a the­o­rem, you can sub­sti­tute ‘A’ for ‘B’ in any the­o­rem where B ap­pears. For ex­am­ple, if (2 + 2) = 4 is a the­o­rem, and ((2 + 2) + 3) = 7 is a the­o­rem, then (4 + 3) = 7 is a the­o­rem.

This leads to a prob­lem which is usu­ally phrased in the fol­low­ing terms: The morn­ing star and the evening star hap­pen to be the same ob­ject, the planet Venus. Sup­pose John knows that the morn­ing star and evening star are the same ob­ject. Mary, how­ever, be­lieves that the morn­ing star is the god Lu­cifer, but the evening star is the god Venus. John be­lieves Mary be­lieves that the morn­ing star is Lu­cifer. Must John there­fore (by sub­sti­tu­tion) be­lieve that Mary be­lieves that the evening star is Lu­cifer?

Or here’s an even sim­pler ver­sion of the prob­lem. 2 + 2 = 4 is true; it is a the­o­rem that (((2 + 2) = 4) = TRUE). Fer­mat’s Last The­o­rem is also true. So: I be­lieve 2 + 2 = 4 ⇒ I be­lieve TRUE ⇒ I be­lieve Fer­mat’s Last The­o­rem.

Yes, I know this seems ob­vi­ously wrong. But imag­ine some­one writ­ing a log­i­cal rea­son­ing pro­gram us­ing the prin­ci­ple “equal terms can always be sub­sti­tuted”, and this hap­pen­ing to them. Now imag­ine them writ­ing a pa­per about how to pre­vent it from hap­pen­ing. Now imag­ine some­one else dis­agree­ing with their solu­tion. The ar­gu­ment is still go­ing on.

P’rsnally, I would say that John is com­mit­ting a type er­ror, like try­ing to sub­tract 5 grams from 20 me­ters. “The morn­ing star” is not the same type as the morn­ing star, let alone the same thing. Beliefs are not planets.

morn­ing star = evening star
”morn­ing star” ≠ “evening star”

The prob­lem, in my view, stems from the failure to en­force the type dis­tinc­tion be­tween be­liefs and things. The origi­nal er­ror was writ­ing an AI that stores its be­liefs about Mary’s be­liefs about “the morn­ing star” us­ing the same rep­re­sen­ta­tion as in its be­liefs about the morn­ing star.

If Mary be­lieves the “morn­ing star” is Lu­cifer, that doesn’t mean Mary be­lieves the “evening star” is Lu­cifer, be­cause “morn­ing star” ≠ “evening star”. The whole para­dox stems from the failure to use quote marks in ap­pro­pri­ate places.

You may re­call that this is not the first time I’ve talked about en­forc­ing type dis­ci­pline—the last time was when I spoke about the er­ror of con­fus­ing ex­pected util­ities with util­ities. It is im­mensely helpful, when one is first learn­ing physics, to learn to keep track of one’s units—it may seem like a bother to keep writ­ing down ‘cm’ and ‘kg’ and so on, un­til you no­tice that (a) your an­swer seems to be the wrong or­der of mag­ni­tude and (b) it is ex­pressed in sec­onds per square gram.

Similarly, be­liefs are differ­ent things than planets. If we’re talk­ing about hu­man be­liefs, at least, then: Beliefs live in brains, planets live in space. Beliefs weigh a few micro­grams, planets weigh a lot more. Planets are larger than be­liefs… but you get the idea.

Merely putting quote marks around “morn­ing star” seems in­suffi­cient to pre­vent peo­ple from con­fus­ing it with the morn­ing star, due to the vi­sual similar­ity of the text. So per­haps a bet­ter way to en­force type dis­ci­pline would be with a visi­bly differ­ent en­cod­ing:

morn­ing star = evening star
13.15.18.14.9.14.7.0.19.20.1.18 ≠ 5.22.5.14.9.14.7.0.19.20.1.18

Study­ing math­e­mat­i­cal logic may also help you learn to dis­t­in­guish the quote and the refer­ent. In math­e­mat­i­cal logic, |- P (P is a the­o­rem) and |- []‘P’ (it is prov­able that there ex­ists an en­coded proof of the en­coded sen­tence P in some en­coded proof sys­tem) are very dis­tinct propo­si­tions. If you drop a level of quo­ta­tion in math­e­mat­i­cal logic, it’s like drop­ping a met­ric unit in physics—you can de­rive visi­bly ridicu­lous re­sults, like “The speed of light is 299,792,458 me­ters long.”

Alfred Tarski once tried to define the mean­ing of ‘true’ us­ing an in­finite fam­ily of sen­tences:

(“Snow is white” is true) if and only (snow is white)
(“Weasels are green” is true) if and only if (weasels are green)
...

When sen­tences like these start seem­ing mean­ingful, you’ll know that you’ve started to dis­t­in­guish be­tween en­coded sen­tences and states of the out­side world.

Similarly, the no­tion of truth is quite differ­ent from the no­tion of re­al­ity. Say­ing “true” com­pares a be­lief to re­al­ity. Real­ity it­self does not need to be com­pared to any be­liefs in or­der to be real. Re­mem­ber this the next time some­one claims that noth­ing is true.