# The dangers of zero and one

Eliezer wrote a post warn­ing against un­re­al­is­ti­cally con­fi­dent es­ti­mates, in which he ar­gued that you can’t be 99.99% sure that 53 is prime. Chris Hal­lquist replied with a post ar­gu­ing that you can.

That par­tic­u­lar case is tricky. There have been many in­de­pen­dent calcu­la­tions of the first hun­dred prime num­bers. 53 is a small enough num­ber that I think some­one would no­tice if Wikipe­dia in­cluded it er­ro­neously. But can you be 99.99% con­fi­dent that 1159 is a prime? You found it in one par­tic­u­lar source. Can you trust that source? It’s large enough that no one would no­tice if it were wrong. You could try to ver­ify it, but if I write a Perl or C++ pro­gram, I can’t even be 99.9% sure that the com­piler or in­ter­preter will in­ter­pret it cor­rectly, let alone that the pro­gram is cor­rect.

Rather than ar­gue over the num­ber of nines to use for a spe­cific case, I want to em­pha­size the the im­por­tance of not as­sign­ing things prob­a­bil­ity zero or one. Here’s a real case where ap­prox­i­mat­ing 99.9999% con­fi­dence as 100% had dis­as­trous con­se­quences.

I de­vel­oped a new gene-caller for JCVI. Genes are in­ter­preted in units of 3 DNA nu­cleotides called codons. A bac­te­rial gene starts with a start codon (usu­ally ATG, TTG, or GTG) and ends at the first stop codon (usu­ally TAG, TGA, or TAA). Most such se­quences are not genes. A gene-caller is a com­puter pro­gram that takes a DNA se­quence and guesses which of them are genes.

The first thing I tried was to cre­ate a sec­ond-or­der Markov model on codons, and train it on all of the large pos­si­ble genes in the genome. (Long se­quences with­out stop codons are un­likely to oc­cur by chance and are prob­a­bly genes.) That means that you set P = 1 and go down the se­quence of each large pos­si­ble gene, codon by codon, mul­ti­ply­ing P by the prob­a­bil­ity of see­ing each of the 64 pos­si­ble codons in the third po­si­tion given the codons in the first and sec­ond po­si­tions. Then I cre­ated a sec­ond Markov model from the en­tire genome. This took about one day to write, and plug­ging these two mod­els into Bayes’ law as shown be­low turned out to work bet­ter than all the other sin­gle-method gene-pre­dic­tion al­gorithms de­vel­oped over the past 30 years.

But what prob­a­bil­ity should you as­sign to a codon se­quence that you’ve never seen? A bac­te­rial genome might have 4 mil­lion base pairs, about half of which are in long pos­si­ble genes and will be used for train­ing. That means your train­ing data for one genome has about 2 mil­lion codon triplets. Sur­pris­ingly, a lit­tle less than half of all pos­si­ble codon triplets do not oc­cur at all in that data (DNA se­quences are not ran­dom). What prob­a­bil­ity do you as­sign to an event that oc­curs zero times out of 2 mil­lion?

This came up re­cently in an on­line ar­gu­ment. Another per­son said that, if the prob­a­bil­ity that X is true is be­low your de­tec­tion thresh­old or your digits of ac­cu­racy, you should as­sign P(X) = 0, since any other num­ber is just made up.

Well, I’d already em­piri­cally de­ter­mined whether that was true for the gene caller. First, due to a cod­ing er­ror, I as­signed such events P(X) = 1 /​ (64^3 * size of train­ing set), which is too small by about 64^3. Next I tried P(X) = 0.5 /​ (size of train­ing set), which is ap­prox­i­mately cor­rect. Fi­nally I tried P(X) = 0. I tested the re­sults on genomes where I had strong ev­i­dence for what where and were not genes.

How well do you think each P(X) worked?

The two non-zero prob­a­bil­ities gave nearly the same re­sults, de­spite differ­ing by 6 or­ders of mag­ni­tude. But us­ing P(X) = 0 caused the gene-caller to miss hun­dreds of genes per genome, which is a dis­as­trous re­sult. Why?

Any par­tic­u­lar codon triplet that was never found in the train­ing set would have a prior of less than one in 4 mil­lion. But be­cause a large num­ber of triplets are in genes out­side the train­ing set, that meant some of those triplets (not most, but about a thou­sand of them) had true pri­ors of be­ing found some­where in those genes of nearly one half. (You can work it out in more de­tail by as­sum­ing a Zipf law dis­tri­bu­tion of pri­ors, but I won’t get into that.)

So some of them did oc­cur within genes in that genome, and each time one did, its as­signed prob­a­bil­ity of zero an­nihilated all the hun­dreds of other pieces of ev­i­dence for the ex­is­tence of that gene, mak­ing the gene im­pos­si­ble to de­tect.

You can think of this us­ing log­a­r­ithms. I computed

P(gene | se­quence) = P(se­quence | gene) * P(gene) /​ P(se­quence)

where P(se­quence) and P(se­quence | gene) are com­puted us­ing the two Markov mod­els. Each of them is the product of a se­quence of Markov prob­a­bil­ities. Ig­nor­ing P(gene), which is con­stant, we can compute

log(P(gene|se­quence)) ~ log(P(se­quence | gene)) - log(P(se­quence)) =

sum (over all codon triplets in the se­quence) [ log(P(codon3 | codon1, codon2, gene)) - log(P(codon3 | codon1, codon2)) ]

You can think of this as adding the bits of in­for­ma­tion it would take to spec­ify that triplet out­side of a gene, and sub­tract­ing the bits of in­for­ma­tion it would take to spec­ify that in­for­ma­tion in­side a gene, leav­ing bits of ev­i­dence that it is in a gene.

If we as­sign P(codon3 | codon1, codon2, gene) = 0, the num­ber of bits of in­for­ma­tion it would take to spec­ify “codon3 | codon1, codon2” in­side a gene is -log(0) = in­finity. As­sign P(X) = 0 is claiming to have in­finite bits of in­for­ma­tion that X is false.

Go­ing back to the ar­gu­ment, the ac­cu­racy of the prob­a­bil­ities as­signed by the Markov model are quite low, prob­a­bly one to three digits of ac­cu­racy in most cases. Yet it was im­por­tant to as­sign pos­i­tive prob­a­bil­ities to events whose prob­a­bil­ities were at least seven or­ders of mag­ni­tude be­low that.

It didn’t mat­ter what prob­a­bil­ity I as­signed to them! Given hun­dreds of other bits scores to add up, chang­ing the num­ber of bits taken away by one highly im­prob­a­ble event by 10 had lit­tle im­pact. It just mat­ters not to make it zero.

• Nice ex­am­ple of how us­ing a prob­a­bil­ity of ex­actly zero can screw you over. Two ob­ser­va­tions.

1. Could have done with a link to Eliezer’s 0 and 1 are not prob­a­bil­ities from back in 2008.

2. I say you can get to 99.99% con­fi­dence that 1159 is prime (if it ac­tu­ally is; I haven’t checked); prob­a­bly 99.9999%. Sup­pose you (a) write a pro­gram to check all pos­si­ble di­vi­sors, test that it gives the right an­swers for ev­ery­thing up to 100, and run it mul­ti­ple times in case of cos­mic rays; (b) look it up in a table of prime num­bers; (c) ap­ply, by hand, one of the fancy num­ber-the­o­ret­i­cal pri­mal­ity tests (most of these are prob­a­bil­is­tic—but again you can find state­ments of the form “If a num­ber less than 10^12 passes these spe­cific tests and isn’t one of the fol­low­ing short list of ex­cep­tions, it is prime”). Then I reckon that apart from the­o­ries where what’s wrong is your brain a,b,c are clearly ex­tremely close to in­de­pen­dent; (a) has well less than 0.001 chance of failure, (b) well less than 0.01, and (c) well less than 0.1; so the only hy­poth­e­sis you need to con­sider that might take the prob­a­bil­ity above 10^-6 is that you’re delu­sional in some way that speci­fi­cally messes up your abil­ity to tell whether 1159 is prime. Now (d) this is surely ex­traor­di­nar­ily rare—delu­sions in gen­eral aren’t so rare, but this is some­thing very spe­cific and weird; and (e) if your mind is that badly screwed up then at­tempt­ing to work with prob­a­bil­ities is prob­a­bly kinda mean­ingless any­way. (The­o­rems like Cox’s say that you should use prob­a­bil­ities as mea­sures of con­fi­dence, and com­pute with them in the stan­dard ways, if you are a ra­tio­nal agent. If you are known to be a catas­troph­i­cally ir­ra­tional agent, then what prob­a­bil­ities you as­sign is prob­a­bly not the great­est of your wor­ries.)

• Can you re­ally be sure that a pro­gram that you write has at least a 99.9% chance of be­ing cor­rect with­out perform­ing mod­er­ately ex­ten­sive test­ing? Per­son­ally, I’d prob­a­bly put sig­nifi­cantly more con­fi­dence in (b).

• A pro­gram this sim­ple? Yes.

[EDITED to add: And I did say to test it on the primes up to 100.]

[EDITED again to add …] Here, just for refer­ence, is what I would write, in less than a minute, to do the job. It is only in­tended to work for in­te­gers >= 2, and need not be bear­ably effi­cient for in­te­gers that aren’t very small.

``````def is_prime(n):
for i in range(2,n):
if n%i == 0: re­turn False
re­turn True
``````

Four short, sim­ple lines cor­re­spond­ing quite ex­actly to the defi­ni­tion of pri­mal­ity.

So, what could be wrong in this pro­gram that could make it give a wrong an­swer for 1159? I can think of the fol­low­ing things. (1) I could have messed up the range of val­ues of i to test against. (2) I could have got my vari­ables mud­dled up, test­ing i%n in­stead of n%i or some­thing of the kind. (3) I could have got my re­turn con­di­tions back­wards, mak­ing this a not-prime test. (4) I could have messed up the con­trol flow in some way that, e.g., does some­thing crazy when we fall off the end of that loop. (5) I could have mixed up the % (mod­u­lus) op­er­a­tor with some­thing else like di­vi­sion or ex­clu­sive-or. (6) I could have got my­self con­fused about what the con­di­tion for pri­mal­ity ac­tu­ally is. (7) I could have done some­thing else that I haven’t been able to think of.

Well, most of those are very low-prob­a­bil­ity, es­pe­cially given that I’ve listed them ex­plic­itly and checked the pro­gram. Fur­ther, it’s easy to see that all of them would al­most cer­tainly fail on test­ing against the num­bers from 2 to 100, and it seems likely that most er­rors in cat­e­gory 7 would do so too. (E.g., one can imag­ine mess­ing up the test in such a way that num­bers of the form prime^2 get mis­tak­enly iden­ti­fied as prime, or so that it only catches primes of the form 4k+1, or some­thing.)

And, lo and be­hold, I then did

``````print [p for p in range(2,100) if is_prime(p)]
``````

and got

``````[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]
``````

which has 25 en­tries (which I know, or think I know, is the right an­swer), con­sists en­tirely of num­bers I be­lieve I know to be prime, and in­cludes ev­ery num­ber < 100 that I be­lieve I know to be prime.

I am very com­fortable in­deed reck­on­ing this as at least 1000:1 ev­i­dence that any spe­cific pos­i­tive in­te­ger on the or­der of 1000 is prime iff my is_prime func­tion says it is and gives the same re­sult on re­peated runs.

Other failure modes I haven’t listed above but want to men­tion that I have thought of: There could be in­te­ger overflow prob­lems. (But not for num­bers this small, and not in Python un­less there’s a weird Python bug I haven’t heard of.) There could be ridicu­lous er­rors in Python it­self that make it get sim­ple ar­ith­metic on small num­bers wrong. (But I’m run­ning Python 2.7.3, which was re­leased some time ago and was for some time the most com­monly used Python ver­sion; I would surely have heard if it had that sort of er­ror; in any case that class of er­ror is very, very rare.) My com­puter could be messed up by cos­mic rays or some­thing. (Which is why I’d run the pro­gram mul­ti­ple times on 1159.)

Or, of course, my brain could be messed up in some way that speci­fi­cally in­terferes with my un­der­stand­ing of prime num­bers or of what it takes to check a pro­gram’s cor­rect­ness. The first is already dealt with in the dis­cus­sion above. The sec­ond is one of the rea­sons why I would also check against some­one else’s list of small­ish prime num­bers and do a num­ber-the­o­retic test by hand. (And also, I think, rather im­prob­a­ble.)

• (7): in­den­ta­tion er­ror. But I guess the in­ter­preter will tell you `i` is used out of scope. That, or you would have got­ten an­other catas­trophic re­sult on num­bers be­low 10.

``````def is_prime(n):
for i in range(2,n):
if n%i == 0: re­turn False
re­turn True
``````

(Edit: okay, that was LessWrong screw­ing up lead­ing spaces. We can cheat that with un­break­able spaces.)

• in­den­ta­tion error

Yes, good point. I’d gen­er­al­ize: “I could have com­mit­ted some syn­tax er­ror—wrong in­den­ta­tion, wrong sort of brack­ets some­where, etc. -- that just hap­pens to leave a pro­gram with cor­rect syn­tax but un­in­tended se­man­tics.”

I guess the in­ter­preter will tell you `i` is used out of scope.

Depends on ex­actly what er­ror oc­curs. (With the for­mat­ting the way it ac­tu­ally looks here it would just be a syn­tax er­ror.)

That, or you would have got­ten an­other catas­trophic re­sult on num­bers be­low 10.

This (aside from the sim­plic­ity of the code) is the real rea­son why I’m still happy with an er­ror prob­a­bil­ity less than 0.001. It takes a re­ally odd sort of er­ror to pro­duce cor­rect re­sults for the first 100 can­di­date primes and wrong re­sults there­after. I could write some pro­grams with that sort of er­ror, but they wouldn’t look at all like naive pri­mal­ity check­ers with sim­ple ty­pos.

• OK. I agree. Check­ing the out­put on num­bers less than 100 vastly de­creases the prob­a­bil­ity of er­ror, es­pe­cially af­ter enu­mer­at­ing sim­ple er­rors and re­al­iz­ing that none of them would re­turn any­thing re­motely cor­rect on those num­bers.

• Well, but you can (a) pre­form mod­er­ately ex­ten­sive test­ing, and (b) do re­dun­dancy.

If you write 3 pro­grams for ver­ify­ing prime­ness (us­ing differ­ent al­gorithms and pos­si­bly pro­gram­ming lan­guages/​ap­proaches); and if all their re­sults match, then you can as­sume a much higher con­fi­dence in cor­rect­ness than for a sin­gle such pro­gram.

• I think mul­ti­ple pro­grams wouldn’t help very much un­less they were run on differ­ent pro­ces­sors and writ­ten in differ­ent lan­guages.

• Yes. I agree. I am cer­tainly not try­ing to say that 99.99% con­fi­dence in the pri­mal­ity sta­tus of a four digit num­ber is not achiev­able.

• I can take a stab at it (in python):

``````> def isprime(n):
> > if n == 1 or n % 2 == 0: re­turn False
> > i = 3
> > while i*i ⇐ n:
> > > if n % i == 0: re­turn False
> > > i += 2
> > re­turn True
>
> print(all(map(isprime, [3,5,7,11,13,17,23,53])))
> print(isprime(1159))
``````

(ap­par­ently LW’s mark­down im­ple­men­ta­tion butch­ers whites­pace in code)

when I run that it tells me that 1159 isn’t prime. This pro­gramme is sim­ple enough that I would af­ter spend­ing an hour or so, at my men­tal peak con­di­tion, prov­ing it cor­rect, trust it with very high stakes.

• And… You’d lose the bet. That pro­gram does have a mis­take in it. See if you can find it.

(An­swer: Gjb vf cevzr, ohg lbhe shapgvba pnyyf vg pbzcbfvgr.)

• Good catch, but I bet Mag­ne­toHy­droDy­nam­ics would not lose the bet “af­ter spend­ing an hour or so, at [their] men­tal peak con­di­tion, prov­ing it cor­rect”.

• True. It’s not a perfect illus­tra­tion. But it’s a good, ironic ex­am­ple of how the point that Mag­ne­toHy­droDy­nam­ics was try­ing to make can have a pretty se­ri­ous flaw.

• Meh. It was a spe­cial case he already knew to call out for spe­cial treat­ment. Would have been caught VERY quickly in test­ing, just like the same sort of er­ror oc­cur­ring in the gen­eral case (hmm. the primes are 4, 6, 8, 9, 10, 12? nooo...)

• 99.99% con­fi­dence im­plies that he could write TEN THOUSAND pro­grams of similar difficulty (with hours spent ver­ify­ing each at peak men­tal con­di­tion) and make only ONE mis­take.

As some­one who’s done quite a bit of pro­gram­ming and read­ing about pro­gram­ming, I’d be im­pressed if he made it passed a hun­dred with­out mak­ing two mis­takes.

The in­dus­try av­er­age is 15-50 er­rors /​ 1000 lines of code. The amount of effort re­quired to get be­low 0.1 er­rors/​kLoC, like they do for the space shut­tle, is very, very large. One per­son check­ing their own work for an hour doesn’t stand a chance.

• Only make one mis­take that makes it past test­ing.

• That’s what I meant, though I can see that it’s not clear.

In this case a mis­take would be writ­ing the code, iter­at­ing and test­ing un­til you were satis­fied, pro­nounc­ing it done, and then af­ter­wards catch­ing the er­ror.

• Hmm. I can definitely buy that a pro­gram of more com­plex­ity than this—less eas­ily checked—would have that ac­cu­racy rate.

But prime check­ing is su­per sim­ple to write and su­per sim­ple to check. The only way you’d get an er­ror through the ob­vi­ous test­ing scheme given is to skip the test­ing.

You’re tak­ing 100 bits of test­ing (which con­tain around 80 bits of in­for­ma­tion if not pro­duced by means of the ac­tual pat­tern) and treat­ing them as around 13 bits of re­li­a­bil­ity.

• My ex­pe­rience with cod­ing is that stupid ob­vi­ous mis­takes are way more likely than 1/​10000. You write some­thing slightly wrong, keep read­ing it as if it were right, and that’s that.

Deter­min­ing if a num­ber is prime is a bit of a nice case, I sup­pose, be­cause it’s so amenable to test­ing. The struc­ture of the mis­takes you make is un­likely to match the struc­ture of primes, so you’ll catch any mis­takes more eas­ily.

I’d still con­sider do­ing it 10000 times to be ex­tremely difficult. Just adding 10000 six-digit num­bers by hand, even with some cross-check­ing, is quite difficult.

• Yes, stupid cod­ing mis­takes are more like 1 in 2 than 1 in 10^4; it is the test­ing that helps here.

• Are the pro­grams writ­ten for the space shut­tle the same level of difficulty as this one?

• Over­all the space shut­tle soft­ware is definitely more com­pli­cated than con­firm­ing a small prime, but on a line by line ba­sis I don’t know. The space shut­tle is more of a spe­cial case ex­am­ple; not some­thing you com­pare against.

I’d use the in­dus­try av­er­age rate un­til you showed me you could hit 0.1 er­rors/​kloc for a few years. For ex­am­ple, Microsoft hits 10-20 defects/​kloc and they put sig­nifi­cant effort into test­ing.

The most likely rea­son that your bug rate on these pro­grams would be anoma­lously low is the fact that they’re small. In­stead of writ­ing one hun­dred thou­sand line pro­gram, you’re writ­ing 10000 ten line pro­grams. The com­plex­ities won’t com­pound as much.

• The prob­a­bil­ity of mis­take in a pro­gram is not the same as the prob­a­bil­ity of be­ing wrong about 1159 be­ing a com­pos­ite. At high re­li­a­bil­ity level, you fac­tor the num­ber, then you mul­ti­ply up the fac­tors, and check that as well.

• apart from the­o­ries where what’s wrong is your brain

(just amused by the pos­si­bil­ity)

Also, it is pos­si­ble that Peano ar­ith­metic isn’t con­sis­tent; if so, ei­ther the very con­cept of ‘pri­mal­ity’ doesn’t make any sense, or it can just mess up the pri­mal­ity tests which were used in cre­ation of (b) and (c), and the con­nec­tion be­tween “1159 if prime” and “this pro­gram out­puts True and halts” as well.

Of course, it screws up any ap­pli­ca­tion of Cox’s the­o­rem here, even worse than in delu­sion case.

• But can you be 99.99% con­fi­dent that 1159 is a prime?

This doesn’t af­fect the thrust of the post but 1159 is not prime. Prime fac­tors are 19 and 61.

• That may have, in fact, been the point. I doubt many peo­ple both­ered to check.

• In­deed. :)

• As soon as I saw that in the post, I be­gan check­ing it. In my head. Have I men­tioned I’m good at math?

• That only means you are merely good at ar­ith­metic. Can you prove, say, that there are no perfect squares of the form

3^p + 19(p-1)

where p is prime?

• I was about 80% sure that 1159 was not prime, based on read­ing that sen­tence. It took me <1 minute to con­firm this. I can to­tally be more than 99.99% sure of the pri­mal­ity of any given four-digit num­ber.

In fact, those odds sug­gest that I’d ex­pect to make one mis­take with prob­a­bil­ity >0.5 if I were to go through a list of all the num­bers be­low 10,000 and clas­sify them as prime or not prime. I think this is ridicu­lous. I’m quite will­ing to take a bet at 100 to 1 odds that I can pro­duce an ex­haus­tive list of all the prime num­bers be­low 1,000,000 (which con­tains no com­pos­ite num­bers), if any­one’s will­ing to stump up at least \$10 for the other side of the bet.

• You can only sim­ply ex­po­nen­ti­ate the chance of suc­cess if it doesn’t cor­re­late over mul­ti­ple rep­e­ti­tions. I would say that if the list of primes be­low 10^6 you were refer­enc­ing has at least one er­ror in the first 10^5, it would be more likely to be faulty later, and vice versa, which means that your gut es­ti­mates on the two scales might be non­con­tra­dic­tory.

• Right. Say you write code to gen­er­ate primes. If there’s no bug, all your an­swers are cor­rect. If there’s a bug in your code, prob­a­bly lots of your an­swers are in­cor­rect.

• The Pen­tium FDIV bug was ac­tu­ally dis­cov­ered by some­one writ­ing code to com­pute prime num­bers.

• Another ex­am­ple where sim­ple prob­a­bil­ity as­sign­ment fails is from “Rea­son­ing with Limited Re­sources and As­sign­ing Prob­a­bil­ities to Arith­meti­cal State­ments” by Haim Gaif­man:( http://​​www.columbia.edu/​​~hg17/​​RLR.pdf )

A coin is tossed 50 times. Jack an­a­lyzes the se­quence of 0’s (heads) and 1’s (tails) and con­cludes that the se­quence is ran­dom, with nearly equal prob­a­bil­ities for each side. Then Jill in­forms him that the coin con­tains a tiny mechanism, con­trol­led by a microchip, which shifts its grav­ity cen­ter so as to pro­duce a pseudo ran­dom se­quence–– one that passes the stan­dard ran­dom­ness tests but is in fact defined by a math­e­mat­i­cal al­gorithm. [...] Jill gives Jack the al­gorithm and the out­comes of ad­di­tional 50 tosses fully ac­cord with the calcu­lated pre­dic­tions. Let h be the hy­poth­e­sis that the out­come se­quence co­in­cides with the se­quences pro­duced by the al­gorithm. Let the ev­i­dence be e. Given e, it would be ir­ra­tional of Jack not to ac­cord h ex­tremely high prob­a­bil­ity. Had P(h)–– Jack’s prior sub­jec­tive prob­a­bil­ity for h––been non-zero, con­di­tion­al­iza­tion on e could have pushed it up. But if P(h) = 0, then, P(h|e) = 0, for ev­ery e for which P(e) > 0. Now Jack’s prior prob­a­bil­ity ac­com­mo­dates the pos­si­bil­ity of in­de­pen­dent tosses with non-zero prob­a­bil­ities for ‘0’ and ‘1’. There­fore P( ) as­signs each finite se­quence a nonzero value, hence P(e) > 0. If P(h) = 0, then Jack can­not learn from e what he should, un­less he changes prob­a­bil­ities in a way that vi­o­lates the Bayesian pre­scrip­tion.

And it ex­plic­itly ques­tions the mod­el­ing of ‘un­known’ pos­si­bil­ites as 0:

Jack has never con­sid­ered the pos­si­bil­ity of a coin that pro­duces, de­ter­minis­ti­cally, a math­e­mat­i­cally defined se­quence; hence, if the prior prob­a­bil­ity is to re­flect any­thing like Jack’s psy­cholog­i­cal state, P(h) = 0. But could he, in prin­ci­ple, have made place for such pos­si­bil­ities in ad­vance?

• Ac­tu­ally “could he, in prin­ci­ple, have made place for such pos­si­bil­ities in ad­vance?” is very, very ex­cel­lent ques­tion.

We can al­lo­cate for such pos­si­bil­ities in ad­vance. For ex­am­ple, we can use a sim­ple statis­ti­cal model for limi­ta­tions of our own un­der­stand­ing of re­al­ity—I have a cer­tain num­ber of years ex­pe­rience in mak­ing judge­ments and as­sump­tions about re­al­ity; I know that I don’t con­sider all pos­si­ble ex­pla­na­tions, and I can es­ti­mate that in x% cases the ‘true’ ex­pla­na­tion was one that I hadn’t con­sid­ered. So I can make a ‘be­lief bud­get’ for the ‘other’ cat­e­gory. For ex­am­ple, any ques­tion like ‘will the coin fall heads or tails’ has to in­clude ‘other’ op­tion. It may fall on it’s side.

A great ex­am­ple is the quo­ta­tion “One of these days in your trav­els, a guy is go­ing to show you a brand-new deck of cards on which the seal is not yet bro­ken. Then this guy is go­ing to offer to bet you that he can make the jack of spades jump out of this brand-new deck of cards and squirt cider in your ear. But, son, do not ac­cept this bet, be­cause as sure as you stand there, you’re go­ing to wind up with an ear full of cider.”

If you want to han­dle re­al­ity, you have to model the prob­a­bil­ity of ‘jack of spades jump­ing out of this brand-new deck of cards and squirt­ing cider in your ear’ as non-zero. 10^-99 might be ok, but not 0.

• Another per­son said that, if the prob­a­bil­ity that X is true is be­low your de­tec­tion thresh­old or your digits of ac­cu­racy, you should as­sign P(X) = 0, since any other num­ber is just made up.

This im­me­di­ately struck me as so very wrong. The worse you can mea­sure, the more events you feel jus­tified as­sign­ing zero prob­a­bil­ity?

It’s strange that such claims make it out into the wild. Maybe there was ad­di­tional con­text that made it make sense for some­one once upon a time, and they gen­er­al­ized to an ill fit­ting con­text.

• My own re­sponse to this is “P(X) = 0 is made up, too. If I want to avoid us­ing a made-up value, I should stop think­ing about P(X) al­to­gether. Alter­na­tively, if I want to think about P(X), I should as­sign it a made-up value that works well. In many con­texts ep­silon works far bet­ter than zero.”

• I think Phil’s ex­pe­rience sug­gests a rea­son­able way to at­tack these prob­lems.

Do the anal­y­sis for a se­ries of ep­silons, start­ing with your mea­sure­ment delta and work­ing down, and see if it makes any differ­ence to the re­sults.

Also, in one of Jaynes’ pa­per on the marginal­iza­tion “para­dox” he sug­gested that ig­nor­ing the ex­is­tence of a vari­able gives you a differ­ent re­sult than ap­ply­ing an ig­no­rance prior (show­ing yet again that the term “ig­no­rance prior” is a stu­pefy­ing oxy­moron).

• Re: ig­nor­ing the ex­is­tence of a vari­able… yes, ab­solutely. I meant in the more sweep­ing, less use­ful sense of “go work on a differ­ent prob­lem al­to­gether, or per­haps just have a beer and watch telly.”

Re: see­ing how re­sults vary for differ­ent pos­si­ble val­ues of an un­known vari­able… yup, agreed.

• This im­me­di­ately struck me as so very wrong. The worse you can mea­sure, the more events you feel jus­tified as­sign­ing zero prob­a­bil­ity?

As­sign­ing more events zero prob­a­bil­ity leaves you worse off com­pared to some­one who makes ac­cu­rate es­ti­mates, but it doesn’t nec­es­sar­ily leave you worse off com­pared to some­one else who mea­sures as poorly as you and makes poorly es­ti­mated mea­sure­ments..

It’s a way of miti­gat­ing the dam­age by not be­ing able to mea­sure well. You’re still worse off than a per­son who can mea­sure well, you’re just not as worse off.

• No. If you’d only ever seen TAAGCC, pe­riod, you would NOT have any sort of li­cense to com­pletely rule out the pos­si­bil­ity of any­thing else. In­deed, the prob­a­bil­ities should be nearly even with a lit­tle more weight given to that par­tic­u­lar ob­ser­va­tion.

Ap­ply­ing the Sun­rise for­mula seems ap­pro­pri­ate here.

• Re­gard­ing 53 be­ing a prime, you can’t just do some naive Bayesian math­e­mat­ics on some real val­ued prob­a­bil­ity of 53 be­ing a prime be­cause that prob­a­bil­ity would be heav­ily cor­re­lated with un­cer­tainty about ev­ery other num­ber and ev­ery other math­e­mat­i­cal op­er­a­tion, in­clud­ing con­ver­sion from dec­i­mal no­ta­tion to other rep­re­sen­ta­tion, in­ter­nally done math­e­mat­ics in vi­sual char­ac­ter recog­ni­tion, and so on. Such prob­a­bil­ity is en­tirely mean­ingless. One could speak of prob­a­bil­ity of (53 is a prime | math­e­mat­ics is not com­pletely wrong), that would be a num­ber very close to 1 (due to re­peat test­ing).

• I want to em­pha­size the im­por­tance of not as­sign­ing things prob­a­bil­ity zero or one.

Un­less you are deal­ing with an officer of the law. An­swers like, “Yes, I’m 99% cer­tain X” do not go over well with them.

Also, I’m sure Godel would have en­joyed your post. Or at least 99% sure.

• I know this is not your main topic, but are you fa­mil­iar with Good-Tur­ing es­ti­ma­tion? It’s a way of as­sign­ing non-ar­bi­trary prob­a­bil­ity to un­ob­served events.

• In­ter­est­ing dis­cus­sion but I sus­pect an im­por­tant dis­tinc­tion may be re­quired be­tween logic and prob­a­bil­ity the­ory. Logic is a spe­cial case of prob­a­bil­ity the­ory where val­ues are re­stricted to only 0 and 1, that is to 0% and 100% prob­a­bil­ity. Within logic you may ar­rive at cer­tain con­clu­sions but gen­er­ally within prob­a­bil­ity the­ory con­clu­sions are not cer­tain but rather as­signed a de­gree of plau­si­bil­ity.

If logic pro­vides, in some con­texts, a valid method of rea­son­ing then con­clu­sions ar­rived at will be ei­ther 0% or 100% true. Deny­ing that 100% con­fi­dence is ever ra­tio­nal seems to be equiv­a­lent to deny­ing that logic ever ap­plies to any­thing.

It is cer­tainly true that many phe­nom­ena are bet­ter de­scribed by prob­a­bil­ity than by logic but can we deny logic any val­idity. I un­der­stand math­e­mat­i­cal proofs as be­ing within the realm of logic where things may of­ten be de­ter­mined as be­ing ei­ther true or false. For in­stance Eu­clid is cred­ited with first prov­ing that there is no largest prime. I be­lieve most math­e­mat­i­ci­ans ac­cept this as a true state­ment and that most would agree that 53 is eas­ily proven to be prime.

• Deny­ing that 100% con­fi­dence is ever ra­tio­nal seems to be equiv­a­lent to deny­ing that logic ever ap­plies to any­thing.

It’s just say­ing that logic is a model that can’t de­scribe any­thing in the real world fully liter­ally. That doesn’t mean it’s not use­ful. Ab­stract­ing away ir­rele­vant de­tails is bread and but­ter re­duc­tion­ism.

• Yes I agree, there is only a rough iso­mor­phism be­tween the math­e­mat­ics of bi­nary logic and the real world; bi­nary logic seems to de­scribe a limit that re­al­ity ap­proaches but never reaches.

We should con­sider that the math­e­mat­ics of bi­nary logic are the limit­ing case of prob­a­bil­ity the­ory; it is prob­a­bil­ity the­ory where the prob­a­bil­ities may only take the val­ues of 0 or 1. Prob­a­bil­ity the­ory can do ev­ery­thing that logic can but it can also han­dle those real world cases where the prob­a­bil­ity of know­ing some­thing is some­thing other than 0 or 1, as is the usual case with sci­en­tific knowl­edge.

• Yeah, I came across that idea in the Jaynes book, and was very im­pressed.

• When you prove some­thing in math­e­mat­ics, at very least you im­plic­itly as­sume you have made no mis­takes any­where, are not hal­lu­ci­nat­ing, etc. Your “real” sub­jec­tive de­gree of be­lief in some math­e­mat­i­cal propo­si­tion, on the other hand, must take all these things into ac­count.

For prac­ti­cal pur­poses the prob­a­bil­ity of hal­lu­ci­na­tions etc. may be very small and so you can usu­ally ig­nore them. But the OP is right to demon­strate that in some cases this is a bad ap­prox­i­ma­tion to make.

De­duc­tive logic is just the spe­cial limit­ing case of prob­a­bil­ity the­ory where you al­low your­self the lux­ury of an ideal­ised box of thought iso­lated from “real world” small prob­a­bil­ities.

edit: Per­haps I could say it a differ­ent way. It may be rea­son­able for cer­tain con­di­tional prob­a­bil­ities to be zero or one, so long as they are con­di­tioned on enough as­sump­tions, e.g. P(“51 is a prime” given “I did my math cor­rectly, I am not hal­lu­ci­nat­ing, the ex­ter­nal world is real, etc...”)=1 might be achiev­able. But if you try to re­move the con­di­tional on all that other stuff you can­not keep this cer­tainty.

• HELLO.Dr Lawrence saved my mar­riage within 3days of con­tact,i con­tacted him in re­gard of my hus­band who left me for an­other woman i tried all the meth­ods i know to get him back but to no avail then a good friend of mine Mrs maria in­tro­duce me to dr­lawrence­spel­ltem­ple@hot­mail.com who cast a pow­er­ful and won­der­ful spell that brought him back to me in just 3days i re­ally want to use this medium to ad­vice that for solu­tion re­gard­ing any re­la­tion­ship is­sues con­tact the tem­ple and all your worry s will be gone: dr­lawrence­spel­ltem­ple@hot­mail.com...DONNA

• Down­voted for be­ing spam.

• I’m over 99.99% cer­tain it’s spam.

• Can you give me an ex­am­ple of a propo­si­tion ar­rived at by what you’re call­ing “logic” here that cor­re­sponds to an ex­pected ob­ser­va­tion in which you have 0% or 100% con­fi­dence?

• “Tau­tolo­gies are tau­tolog­i­cal” is the state­ment for which I am most cer­tain.

Cer­tainly, if the fate of hu­man­ity was the con­se­quence of a false pos­i­tive and a slap on my wrist was the con­se­quence of a false nega­tive for an af­fir­ma­tive an­swer to “tau­tolo­gies are tau­tolog­i­cal”, I would give a nega­tive an­swer...so by that lit­mus test, I don’t re­ally have 100% con­fi­dence. But I’ve still have a strong log­i­cal in­tu­ition that I aught to be that con­fi­dent, and that my emo­tional un­der­con­fi­dence in the pro­posed sce­nario is a bit silly.

The prob­lem is, the above hy­po­thet­i­cal sce­nario in­volves weird and self con­tra­dic­tory stuff (an en­tity which knows the right an­swer, my 100% cer­tainty that the en­tity knows the right an­swer and can in­ter­pret my an­swer, etc). I think it may be best to re­strict prob­a­bil­ity es­ti­mates to em­piri­cal mat­ters (I’m 99.99% cer­tain that my calcu­la­tor will out­put “2” in re­sponse to the in­put “1+1=”) and keep it away from pure logic realms.

• +1 for the thought ex­per­i­ment test. I haven’t seen that one be­fore; any source or is it some­thing you came up with?

• thanks! the spe­cific thought ex­per­i­ment is some­thing I came up with but the gen­eral no­tion of us­ing hy­po­thet­i­cal sce­nar­ios to gauge one’s true cer­tainty is some­thing that I think many peo­ple have done at one time or an­other, though I can’t think of a spe­cific ex­am­ple.

• One ex­am­ple in clas­si­cal logic is the syl­l­o­gism where if the premises are true then the con­clu­sion is by ne­ces­sity true:

Socrates is a man

All men are mortal

there­fore it is true that Socrates is mortal

Another ex­am­ple is math­e­mat­i­cal proofs. Here is the Wikipe­dia pre­sen­ta­tion of Eu­clids proof from 300 BC that there is an in­finite num­ber of prime num­bers. Per­haps In your terms this proof pro­vides 0% con­fi­dence that we will ob­serve the largest prime num­ber.

Take any finite list of prime num­bers p1, p2, …, pn. It will be shown that at least one ad­di­tional prime num­ber not in this list ex­ists. Let P be the product of all the prime num­bers in the list: P = p1p2...pn. Let q = P + 1. Then, q is ei­ther prime or not:

1) If q is prime then there is at least one more prime than is listed.

2) If q is not prime then some prime fac­tor p di­vides q. If this fac­tor p were on our list, then it would di­vide P (since P is the product of ev­ery num­ber on the list); but as we know, p di­vides P + 1 = q. If p di­vides P and q then p would have to di­vide the differ­ence of the two num­bers, which is (P + 1) − P or just 1. But no prime num­ber di­vides 1 so there would be a con­tra­dic­tion, and there­fore p can­not be on the list. This means at least one more prime num­ber ex­ists be­yond those in the list.

This proves that for ev­ery finite list of prime num­bers, there is a prime num­ber not on the list. There­fore there must be in­finitely many prime num­bers.

• The “Socrates is mor­tal” one is a good ex­am­ple be­cause nowa­days its con­clu­sion has a prob­a­bil­ity of less than one.

• I would be in­ter­ested if you would care to elab­o­rate a lit­tle.Syl­l­o­gisms have been a main­stay of philos­o­phy for over two mil­len­nium and un­doubt­edly I have a lot to learn about them.

In my ad­mit­tedly limited un­der­stand­ing of syl­l­o­gisms the con­clu­sion is true given the premises be­ing true. Truth is more in the struc­ture of the ar­gu­ment than in its con­clu­sion. If Socrates is not mor­tal than ei­ther he is not a man or not all men are mor­tal.

• It is the “all men are mor­tal” propo­si­tion that is in dan­ger of be­ing ren­dered false by suffi­ciently ad­vanced tech­nol­ogy (at least, de­pend­ing on what you mean by “mor­tal”).

• Or by “man.”

• Per­haps In your terms this proof pro­vides 0% con­fi­dence that we will ob­serve the largest prime num­ber.

Sure, I’m will­ing to con­sider that a pre­dic­tion about the num­bers that cor­re­spond to ob­serv­able phe­nom­ena.

And you’re as­sert­ing that the chance that Eu­clid was wrong about the prop­er­ties of the num­bers we ob­serve is not just van­ish­ingly small, but in fact zero, such that no amount of ob­served ev­i­dence could pos­si­bly prop­erly change our minds about it.

Yes?

• I have some skep­ti­cism about ab­solute cer­tainty. Logic deals in cer­tain­ties but it seems un­clear if it ab­solutely de­scribes any­thing in the real world. I am not sure if ob­served ev­i­dence plays a role in logic. If all men are mor­tal and if Socrates is a man then Socrates is mor­tal ap­pears to be true. If we were to ob­serve Socrates be­ing im­mor­tal the syl­l­o­gism would still be true but one of the con­di­tional premises that all men are mor­tal or that Socrates is a man would not be true.

In sci­ence at least where ev­i­dence plays a de­ci­sive role there is no cer­tainty; sci­en­tific the­o­ries must be falsifi­able, there is always some pos­si­bil­ity that an ex­per­i­men­tal re­sult will not agree with the­ory.

The ex­am­ples I gave are true by virtue of log­i­cal re­la­tion­ships such as if all A are B and all B are C then all A are C. In this vein it might seem cer­tain that if some­thing is here it can­not be there, how­ever this is not true for quan­tum sys­tems; due to su­per­po­si­tion a quan­tum en­tity can be said to be both here and there.

Another in­ter­est­ing ap­proach to this prob­lem was taken by David Deutsch. He con­sid­ers that any math­e­mat­i­cal proof is a form of calcu­la­tion and all calcu­la­tion is phys­i­cal just as all in­for­ma­tion has a phys­i­cal form. Thus math­e­mat­i­cal proofs are no more cer­tain than the phys­i­cal laws in­voked to calcu­late them. All math­e­mat­i­cal proofs re­quire our math­e­mat­i­cal in­tu­ition, the in­tu­ition that one step of the proof fol­lows log­i­cally from the other. Un­doubt­edly such in­tu­ition is the re­sult of our long evolu­tion­ary his­tory that has built knowl­edge of how the world works into our brains. Although these in­tu­itions are formed from prin­ci­ples en­coded in our ge­net­ics they are no more re­li­able than any other hy­poth­e­sis sup­ported by the data; they are not cer­tain.

• OK. Thanks for clar­ify­ing your po­si­tion.

• It may be in­ter­est­ing that al­though all mea­surable re­sults in quan­tum the­ory are in the form of prob­a­bil­ities there is at least one in­stance where this the­ory pre­dicts a cer­tain re­sult. If the same mea­sure­ment is im­me­di­ately made a sec­ond time on a quan­tum sys­tem the sec­ond re­sult will be the same as the first with prob­a­bil­ity 1. In other words the state of the quan­tum sys­tem re­vealed by the first mea­sure­ment is con­firmed by the sec­ond mea­sure­ment. It may seem odd that the the­ory pre­dicts the re­sult of the first mea­sure­ment as a prob­a­bil­ity dis­tri­bu­tion of pos­si­ble re­sults but pre­dicts only a sin­gle pos­si­ble re­sult for the sec­ond mea­sure­ment.

Wo­j­ciech Zu­ruk con­sid­ers this as a pos­tu­late of quan­tum the­ory (see his pa­per quan­tum Dar­winism ). (sorry for the typo in the quote).

• Pos­tu­late (iii) Im­me­di­ate rep­e­ti­tion of a mea­sure­ment yields the same out­come starts this task. This is the only un­con­tro­ver­sial mea­sure­ment pos­tu­late (even if it is difficult to ap­prox­i­mate in the lab­o­ra­tory): Such re­peata­bil­ity or pre­dictabil­ity is be­hind the very idea of a state.

If we con­sider that in­for­ma­tion ex­change took place be­tween the quan­tum sys­tem and the mea­sur­ing de­vice in the first mea­sure­ment then we might view the prob­a­bil­ity dis­tri­bu­tion im­plied by the wave func­tion as hav­ing un­der­gone a Bayesian up­date on the re­ceipt of new in­for­ma­tion. We might un­der­stand that this new in­for­ma­tion moved the quan­tum model to pre­dic­tive cer­tainty re­gard­ing the re­sult of the sec­ond mea­sure­ment.

Of course this cer­tainty is only cer­tain within the terms of quan­tum the­ory which is it­self falsifi­able.

• I fail to dis­cern your point, here; sorry. Speci­fi­cally, I don’t see what makes this more in­ter­est­ing in con­text than my ex­pec­ta­tion, within the limits of pre­ci­sion and re­li­a­bil­ity of my mea­sur­ing de­vice, that if I (e.g.) mea­sure the mass of a macro­scopic ob­ject twice I’ll get the same re­sult.

• Yes, good point. Clas­si­cal physics, deal­ing with macro­scopic ob­jects, pre­dicts definite (non-prob­a­bil­is­tic) mea­sure­ment out­comes for both the first and sec­ond mea­sure­ments.

The point I was (poorly) aiming at is that while quan­tum the­ory is in­her­ently prob­a­bil­is­tic even it some­times pre­dicts spe­cific re­sults as cer­tain­ties.

I guess the im­por­tant point for me is that while the­o­ries may pre­dict cer­tain­ties they are always falsifi­able; the the­ory it­self may be wrong.

• Ah, I see. Yes, ex­actly… the the­ory may be wrong, or we have made a mis­take in ap­ply­ing it or in­ter­pret­ing it, etc.