I have a huge problem with the “Some problems are boring” section, and it basically boils down into the following set of rebuttals:
Some problems may seem boring, but are vital to solve anyway
Some problems may seem boring, but their generalizations are interesting
Problems that seem boring may have really interesting solutions we are unaware of
Every single one of the examples cited in that section falls into this category:
Figuring out if a blotch on a dental CT scan is more likely to indicate a streptococcus or a lactobacillus infection.
Understanding what makes an image used to advertise a hiking pole attractive to middle-class Slovenians over the age of 54.
Figuring out, using l2 data, if the spread for the price of soybean oil is too wide, and whether the bias is towards the sell or buy.
Finding the optimal price at which to pre-sell a new brand of luxury sparkling water based on yet uncertain bottling, transport, and branding cost.
Figuring out if a credit card transaction is likely to be fraudulent based on the customer’s previous buying pattern.
They all have interesting generalizations, applications and potential solutions. Identifying arbitrary blotches on dental CT scans can be generalized to early-stage gum disease prevention. Figuring out optimal pricing for any item can assist in optimal market regulation. Identifying fraud actively makes the world safer and gives us tools to understand how cheaters adapt in real-time to detection events. And, be honest, if the answer to any of these turned not to be trivial at all—if this is what our models point to—no one would be suddenly claiming the problem itself is boring.
I feel really strongly about this because dismissing any problem as “boring” is isomorphic to asking “why do we fund basic science at all if we get no applications from it” or “why study pure math”, and we all ought to know better than to advance a position so well-rebuffed as “it seems really specific and not personally interesting to me, so why should we (as a society/field) care?”
I read this article and its referenced papers when it was published on Hacker News 12 days ago, and I have reservations against accepting its conclusions regarding the broken nature of digital advertising.
The article’s conclusion is predicated on two specific kinds of evidence:
1. That brand-keyword ads overwhelmingly demonstrate selection effects.
2. That advertising for companies with large advertising impact across different channels demonstrates more selection effects than advertising effects.
The evidence is compelling, but it doesn’t warrant the conclusion that digital advertising is ineffective because:
1. Brand-keyword ads (when someone searching for “Macy’s” gets an ad linking to Macy’s website) are not the only kind or even the most common kind of keyword ads. Targeted keyword ads (having an ad for Macy’s website when someone looks up “cashmere sweater”) are more common and more competitive, yet haven’t been covered or studied in the provided literature.
2. All the studies cited in this article (such as Lewis and Rao 2015 and Gordon et al. 2018) either explicitly deal with firms that are described as “large” or “having millions of customers” (Lewis and Rao, or the eBay intervention), or neglect to disclose or characterize the firms involved in the study (such as Gordon et al). A possible selection bias might be ocurring where only brands with large pre-existing brand identity are being studied—in such a case, it would not be surprising the literature demonstrates more selection effects than advertising effects, as customers would have already heard about the brands by the time these studies ran.
Ideally, the following pieces of evidence would be needed to conclude that digital advertising as-is really is broken:
1. A survey of the effectiveness of targeted keyword ads.
2. The impact of digital advertising among companies with no large brand presence among different channels.
I was unable to find anything in the literature for either, but I confess I did not try very hard beyond a perfunctory Google Scholar search.
---
I agree that the examples cited in this article are compelling evidence for an application of Goodhart’s law in digital advertising.