The Epsilon Fallacy

Link post

Pro­gram Optimization

One of the ear­lier les­sons in ev­ery pro­gram­mer’s ed­u­ca­tion is how to speed up slow code. Here’s an ex­am­ple. (If this is all greek to you, just note that there are three differ­ent steps and then skip to the next para­graph.)

//​ Step 1: import
Im­port foo­bar­lib.*
//​ Step 2: Ini­tial­ize ran­dom Foo array
Foo_field = Foo[1000]
//​ Step 3: Smooth foo field
For x in [1...998]:
Foo_field[x] = (foo_field[x+1] + foo_field[x-1])/​2

Our green­horn pro­gram­mers jump in and start op­ti­miz­ing. Maybe they de­cide to start at the top, at step 1, and think “hmm, maybe I can make this im­port more effi­cient by only im­port­ing Foo, rather than all of foo­bar­lib”. Maybe that will make step 1 ten times faster. So they do that, and they run it, and lo and be­hold, the pro­gram’s run time goes from 100 sec­onds to 99.7 sec­onds.

In prac­tice, most slow com­puter pro­grams spend the vast ma­jor­ity of their time in one small part of the code. Such slow pieces are called bot­tle­necks. If 95% of the pro­gram’s run­time is spent in step 2, then even the best pos­si­ble speedup in steps 1 and 3 com­bined will only im­prove the to­tal run­time by 5%. Con­versely, even a small im­prove­ment to the bot­tle­neck can make a big differ­ence in the run­time.

Back to our green­horn pro­gram­mers. Hav­ing im­proved the run time by 0.3%, they can re­spond one of two ways:

  • “Great, it sped up! Now we just need a bunch more im­prove­ments like that.”

  • “That was ba­si­cally use­less! We should figure out which part is slow, and then fo­cus on that part.”

The first re­sponse is what I’m call­ing the ep­silon fal­lacy. (If you know of an ex­ist­ing and/​or bet­ter name for this, let me know!)

The ep­silon fal­lacy comes in sev­eral rea­son­able-sound­ing fla­vors:

  • The sign(ep­silon) fal­lacy: this tiny change was an im­prove­ment, so it’s good!

  • The in­te­gral(ep­silon) fal­lacy: an­other 100 tiny changes like that, and we’ll have a ma­jor im­prove­ment!

  • The in­finity*ep­silon fal­lacy: this thing is re­ally ex­pen­sive, so this tiny change will save lots of money!

The ep­silon fal­lacy is tricky be­cause these all sound com­pletely rea­son­able. They’re even tech­ni­cally true. So why is it a fal­lacy?

The mis­take, in all cases, is a failure to con­sider op­por­tu­nity cost. The ques­tion is not whether our green­horn pro­gram­mers’ 0.3% im­prove­ment is good or bad in and of it­self. The ques­tion is whether our green­horn pro­gram­mers’ 0.3% im­prove­ment is bet­ter or worse than spend­ing that same amount of effort find­ing and im­prov­ing the main bot­tle­neck.

Even if the 0.3% im­prove­ment was re­ally easy—even if it only took two min­utes—it can still be a mis­take. Our pro­gram­mers would likely be bet­ter off if they had spent the same two min­utes timing each sec­tion of the code to figure out where the bot­tle­neck is. In­deed, if they just iden­tify the bot­tle­neck and speed it up, and don’t bother op­ti­miz­ing any other parts at all, then that will prob­a­bly be a big win. Con­versely, no mat­ter how much they op­ti­mize ev­ery­thing be­sides the bot­tle­neck, it won’t make much differ­ence. Any time spent op­ti­miz­ing non-bot­tle­necks, could have been bet­ter spent iden­ti­fy­ing and op­ti­miz­ing the bot­tle­neck.

This is the key idea: time spent op­ti­miz­ing non-bot­tle­necks, could have been bet­ter spent iden­ti­fy­ing and op­ti­miz­ing the bot­tle­neck. In that sense, time spent op­ti­miz­ing non-bot­tle­necks is time wasted.

In pro­gram­ming, this all seems fairly sim­ple. What’s more sur­pris­ing is that al­most ev­ery­thing in the rest of the world also works like this. Un­for­tu­nately, in the real world, so­cial mo­ti­va­tions make the ep­silon fal­lacy more in­sidious.

Car­bon Emissions

Back in col­lege, I re­mem­ber watch­ing a video about this pro­ject. The video overviews many differ­ent ap­proaches to car­bon emis­sions re­duc­tion: so­lar, wind, nu­clear, and bio power sources, grid im­prove­ments, en­g­ine/​mo­tor effi­ciency, etc. It ar­gues that none of these will be suffi­cient, on its own, to cut car­bon emis­sions enough to make a big differ­ence. But each piece can make a small differ­ence, and if we put them all to­gether, we get a vi­able car­bon re­duc­tion strat­egy.

This is called the “wedge ap­proach”.

Here’s a chart of US car­bon emis­sions from elec­tric­ity gen­er­a­tion by year, shame­lessly cribbed from a Forbes ar­ti­cle.

Note that emis­sions have dropped con­sid­er­ably in re­cent years, and are still go­ing down. Want to guess what that’s from? Hint: it ain’t a bunch of small things adding to­gether.

In the early 00’s, US oil drilling moved to­ward hori­zon­tal drilling and frack­ing. One side effect of these new tech­nolo­gies was a big boost in nat­u­ral gas pro­duc­tion—US nat­gas out­put has been grow­ing rapidly over the past decade. As a re­sult, nat­gas prices be­came com­pet­i­tive with coal prices in the mid-00’s, and elec­tric­ity pro­duc­tion be­gan to switch from coal to nat­gas. The shift is already large: elec­tric­ity from coal has fallen by 25%, while nat­gas has in­creased 35%.

The up­shot: nat­gas emits about half as much car­bon per BTU as coal, and elec­tric­ity pro­duc­tion is switch­ing from coal to nat­gas en mass. Prac­ti­cally all of the re­duc­tion in US car­bon emis­sions over the past 10 years has come from that shift.

Now, back to the wedge ap­proach. One ma­jor ap­peal of the wedge nar­ra­tive is that it’s in­clu­sive: we have all these well-mean­ing peo­ple work­ing on all sorts of differ­ent ap­proaches to car­bon re­duc­tion. The wedge ap­proach says “hey, all these ap­proaches are valuable and im­por­tant pieces of the effort, let’s all work to­gether on this”. Kum-bay-a.

But then we look at the data. Prac­ti­cally all the car­bon re­duc­tion over the past decade has come from the nat­gas tran­si­tion. Every­thing else—the col­lec­tive effort of hun­dreds of thou­sands of re­searchers and en­vi­ron­men­tal­ists on ev­ery­thing from so­lar to wind to ad cam­paigns tel­ling peo­ple to turn off their lights when not in the room—all of that adds up to barely any­thing so far, com­pared to the im­pact of the nat­gas tran­si­tion.

Now, if you’re friends with some of those re­searchers and en­vi­ron­men­tal­ists, or if you did some of that work your­self, then this will all sound like a sta­tus at­tack. We’re say­ing that all these well-mean­ing, hard-work­ing peo­ple were ba­si­cally use­less. They were the 0.3% im­prove­ment to run time. So there’s a nat­u­ral in­stinct to defend our friends/​our­selves, an in­stinct to say “no, it’s not use­less, that 0.3% im­prove­ment was valuable and mean­ingful and im­por­tant!” And we reach into our brains for a rea­son why our friends are not use­less-

And that’s when the ep­silon fal­lacy gets us.

“It’s still a pos­i­tive change, so it’s worth­while!”

“If we keep gen­er­at­ing these small changes, it will add up to some­thing even big­ger than nat­gas!”

“Car­bon emis­sions are huge, so even a small per­cent change mat­ters a lot!”

This is the ap­peal of the wedge ap­proach: the wedge ap­proach says all that effort is valuable and im­por­tant. It sounds a lot nicer than call­ing ev­ery­one use­less. It is nicer. But nice­ness does not re­duce car­bon emis­sions.

Re­mem­ber why the ep­silon fal­lacy is wrong: op­por­tu­nity cost.

Take so­lar pho­to­voltaics as an ex­am­ple: PV has been an ac­tive re­search field for thou­sands of aca­demics for sev­eral decades. They’ve had barely any effect on car­bon emis­sions to date. What would the world look like to­day if all that effort had in­stead been in­vested in ac­cel­er­at­ing the nat­gas tran­si­tion? Or in ex­tend­ing the nat­gas tran­si­tion to China? Or in so­lar ther­mal or tho­rium for that mat­ter?

Now, maybe some­day so­lar PV ac­tu­ally will be a ma­jor en­ergy source. There are le­gi­t­i­mate ar­gu­ments in fa­vor.¹ Even then, we need to ask: would the long-term re­sult be bet­ter if our efforts right now were fo­cussed el­se­where? I hon­estly don’t know. But I will make one pre­dic­tion: one wedge will end up a lot more effec­tive than all oth­ers com­bined. Car­bon emis­sion re­duc­tions will not come from a lit­tle bit of nat­gas, a lit­tle bit of PV, a lit­tle bit of many other things. That’s not how the world works.

The 8020 Rule

Sup­pose you’re a ge­netic en­g­ineer, and you want to de­sign a genome for a very tall per­son.

Our cur­rent un­der­stand­ing is that height is driven by lots of differ­ent genes, each of which has a small im­pact. If that’s true, then in­te­gral(ep­silon) isn’t a fal­lacy. A large num­ber of small changes re­ally is the way to make a tall per­son.

On the other hand, this definitely is not the case if we’re op­ti­miz­ing a com­puter pro­gram for speed. In com­puter pro­grams, one small piece usu­ally ac­counts for the vast ma­jor­ity of the run time. If we want to make a sig­nifi­cant im­prove­ment, then we need to fo­cus on the bot­tle­neck, and any im­prove­ment to the bot­tle­neck will likely be sig­nifi­cant on its own. “Lots of small changes” won’t work.

So… are things usu­ally more like height, or more like com­puter pro­grams?

A use­ful heuris­tic: the vast ma­jor­ity of real-world cases are less like height, and more like com­puter pro­grams. In­deed, this heuris­tic is already well-known in a differ­ent con­text: it’s just the 8020 rule. 20% of causes ac­count for 80% of effects.

If 80% of any given effect is ac­counted for by 20% of causes, then those 20% of causes are the bot­tle­neck. Those 20% of causes are where effort needs to be fo­cused to have a sig­nifi­cant im­pact on the effect. For ex­am­ples, here’s wikipe­dia on the 8020 rule:

  • 20% of pro­gram code con­tains 80% of the bugs

  • 20% of work­place haz­ards ac­count for 80% of injuries

  • 20% of pa­tients con­sume 80% of healthcare

  • 20% of crim­i­nals com­mit 80% of crimes

  • 20% of peo­ple own 80% of the land

  • 20% of clients ac­count for 80% of sales

You can go be­yond wikipe­dia to find whole books full of these things, and not just for peo­ple-driven effects. In the phys­i­cal sci­ences, it usu­ally goes un­der the name “power law”.

(As the ex­am­ples sug­gest, the 8020 rule is pretty loose in terms of quan­ti­ta­tive pre­ci­sion. But for our pur­poses, qual­i­ta­tive is fine.)

So we have an heuris­tic. Most of the time, the ep­silon fal­lacy will in­deed be a fal­lacy. But how can we no­tice the ex­cep­tions to this rule?

One strong hint is a nor­mal dis­tri­bu­tion. If an effect re­sults from adding up many small causes, then the effect will (typ­i­cally) be nor­mally dis­tributed. Height is a good ex­am­ple. Short-term stock price move­ments are an­other good ex­am­ple. They might not be ex­actly nor­mal, or there might be a trans­for­ma­tion in­volved (stock price move­ments are roughly log-nor­mal). If there’s an ap­prox­i­mate nor­mal dis­tri­bu­tion hid­ing some­where in there, that’s a strong hint.

But in gen­eral, omit­ting some ob­vi­ous nor­mal dis­tri­bu­tion, our prior as­sump­tion should be that most things are more like com­puter pro­grams than like height. The ep­silon fal­lacy is usu­ally fal­la­cious.

Con­clu­sion: Pro­file Your Code

Most pro­gram­mers, at some point in their ed­u­ca­tion/​ca­reer, are given an as­sign­ment to speed up a pro­gram. Typ­i­cally, they start out by try­ing things, look­ing for parts of the code which are ob­vi­ously sub­op­ti­mal. They im­prove those parts, and it does not seem to have any im­pact what­so­ever on the run­time.

After wast­ing a few hours of effort on such changes, they fi­nally “pro­file” the code—the tech­ni­cal name for timing each part, to figure out how much time is spent in each sec­tion. They find out that 98% of the run­time is in one sec­tion which they hadn’t even thought to look at. Of course all the other changes were use­less; they didn’t touch the part where 98% of the time is spent!

The in­tended les­son of the ex­pe­rience is: ALWAYS pro­file your code FIRST. Do not at­tempt to op­ti­mize any par­tic­u­lar piece un­til you know where the run­time is spent.

As in pro­gram­ming, so in life: ALWAYS iden­tify the bot­tle­neck FIRST. Do not waste time on any par­tic­u­lar small piece of a prob­lem un­til you know which piece ac­tu­ally mat­ters.


¹The Taleb ar­gu­ment pro­vides an in­ter­est­ing coun­ter­weight to the ep­silon fal­lacy. If we’re bad at pre­dict­ing which ap­proach will be big, then it makes sense to in­vest a lit­tle in many differ­ent ap­proaches. We ex­pect most of them to be use­less, but a few will have ma­jor re­sults—similar to ven­ture cap­i­tal. That said, it’s amaz­ing how of­ten peo­ple who make this ar­gu­ment just hap­pen to end up work­ing on the same things as ev­ery­one else.

No nominations.
No reviews.