I think the conclusion here is probably right, but a lot of the examples seem to exaggerate the role of DL. Like, if I thought all of the obvious-hype-bullshit put forward by big companies about DL were completely true, then it would look like this answer.
Starting from the top:
Companies like Google and even Apple talk a great deal about how they increasingly employ DL at every layer of the stack.
So, a few years back Google was pushing the idea of “AI first design” internally—i.e. design apps around the AI use-cases. By all reports from the developers I know at Google, this whole approach crashed and burned. Most ML applications didn’t generalize well beyond their training data. Also they were extremely unreliable so they always needed to either be non-crucial or to have non-ML fallbacks. (One unusually public example: that scandal where black people were auto-labelled as gorillas.) I hear that the whole “AI first” approach has basically been abandoned since then.
Of course, Google still talks about how they increasingly employ DL at every layer of the stack. It’s great hype.
DL affects the chip design...
I mean, maybe it’s used somewhere in the design loop, but I doubt it’s particularly central. My guess would be it’s used in one or two tools somewhere which are in practice not-importantly-better than the corresponding non-DL version, but someone stuck a net in there somewhere just so that they could tell a clueless middle manager “it uses deep learning!” and the clueless middle manager would buy this mediocre piece of software.
is most of what your camera does...
Misleading. Yeah, computational photography techniques have exploded, but the core tricks are not deep learning at all.
detects your face to unlock it...
This one I think I basically buy, although I don’t know much about how face detection is done today.
powers the recommendations of every single thing on it whether Youtube or app store or streaming service...
powers the ads which monetize you in the search engine results which they also power...
Misleading. Those recommenders presumably aren’t using end-to-end DL; they’re mixing it in in a few specific places. It’s a marginal value add within a larger system, not the backbone of the system.
powers the features like transcripts of calls or machine translation of pages or spam detection that you take for granted...
I basically buy the transcripts and translation examples, and basically don’t buy the spam example—we already had basically-viable spam detection before DL, the value-add there has been marginal at best.
the anti-hacking and anti-abuse measures which keep you safe (and also censor hatespeech etc on streams or social media)...
I hear companies wish they could get ML to do this well, but in practice most things still need to loop through humans. That’s epistemic status: hearsay, so not confident, but it matches my priors.
the voice synthesis you hear when you talk to it, the voice transcription when you talk to it or have your Zoom/Google videoconference sessions during the pandemic, the wake words...
These examples I basically buy.
the predictive text when you prefer to type rather than talk and the email suggestions...
These seem to work in practice exactly when they’re doing the same thing an n-gram predictor would do, and not work whenever they try to predict anything more ambitious than that.
the GNN traffic forecasts changing your Google Maps route to the meeting you emailed about...
I would be surprised if DL were doing most of the heavy lifting in Maps’ traffic forecast at this point, although I would not be surprised if it were sprinkled in and hyped up. That use-case should work really well for non-DL machine learning systems (or so I’d guess), which are a lot more transparent to the designers and developers.
the cooling systems of the data centers running all of this (not to mention optimizing the placement of the programs within the data centers both spatially in solving the placement problem and temporally in forecasting)...
Another two places where I doubt that DL is the main backbone, although it may be sprinkled in here and there and hyped up a lot. I doubt that the marginal value-add from DL is all that high in either of these use-cases, since non-DL machine learning should already be pretty good at these sorts of problems.
Stellar breakdown of hype vs. reality. Just wanted to share some news from today that Google has fired an ML scientist for challenging their paper on DL for chip placement.
The New York Times has learned Google fired machine learning scientist Satrajit Chatterjee in March, soon after it refused to publish a paper Chatterjee and others wrote challenging earlier findings that computers could design some chip components more effectively than humans. The scientist was reportedly allowed to collaborate on a paper disputing those claims after he and fellow authors expressed reservations, but was dismissed after a resolution committee rejected the paper and the researchers hoped to bring the issue to CEO Sundar Pichai and Alphabet’s board of directors.
The company hasn’t detailed why it fired Chatterjee, but told the Times he’d been “terminated with cause.” It also maintained that the original paper had been “thoroughly vetted” and peer-reviewed, and that the study challenging the claims “did not meet our standards.”
Sounds like challenging the hype is a terminable offense. But see gwern’s context for the article below.
Sounds like challenging the hype is a terminable offense.
“One story is good until another is told”. The chip design work has apparently been replicated, and Metz’s* writeup there has several red flags: in describing Gebru’s departure, he omits any mention of her ultimatum and list of demands, so he’s not above leaving out extremely important context in these departures in trying to build up a narrative of ‘Google fires researchers for criticizing research’; he explicitly notes that Chatterjee was fired ‘for cause’ which is rather eyebrow-raising when usually senior people ‘resign to spend time with their families’ (said nonfirings typically involving things like keeping their stock options while senior people are only ‘fired for cause’ when they’ve really screwed up—like, say, harassment of an attractive young woman) but he doesn’t give what that ‘cause’ was (does he really not know after presumably talking to people?) or wonder why both Chatterjee and Google are withholding it; and he uninterestedly throws in a very brief and selective quote from a presumably much longer statement by a woman involved which should be raising your other eyebrow:
Ms. Goldie said that Dr. Chatterjee had asked to manage their project in 2019 and that they had declined. When he later criticized it, she said, he could not substantiate his complaints and ignored the evidence they presented in response.
“Sat Chatterjee has waged a campaign of misinformation against me and Azalia for over two years now,” Ms. Anna Goldie said in a written statement.
She said the work had been peer-reviewed by Nature, one of the most prestigious scientific publications. And she added that Google had used their methods to build new chips and that these chips were currently used in Google’s computer data centers.
(I note that this is put at the end, which in the NYT house style, is where they bury the inconvenient facts that they can’t in good journalist conscience leave out entirely, and that makes me suspect there is more to this part than is given.)
So, we’ll see. EDIT: Timnit Gebru, perhaps surprisingly, denies any parallel and seems to say Chatterjee deserved to be fired, saying:
...But I had heard about the person from many ppl. To the extent the story is connected to mine, it’s ONLY the pattern of action on toxic men taken too late while ppl like me are retaliated against. This is NOT a story about censorship. Its a story about a toxic person who was able to stay for a long time even though many ppl knew of said toxicity. And now, they’re somehow connecting it to my story of discrimination, speaking up day in & day out & being retaliated against?
Wired has a followup article with more detailed timeline and discussion. It edges much closer to the misogyny narrative than the evil-corporate-censorship narrative.
I think the conclusion here is probably right, but a lot of the examples seem to exaggerate the role of DL. Like, if I thought all of the obvious-hype-bullshit put forward by big companies about DL were completely true, then it would look like this answer.
Starting from the top:
So, a few years back Google was pushing the idea of “AI first design” internally—i.e. design apps around the AI use-cases. By all reports from the developers I know at Google, this whole approach crashed and burned. Most ML applications didn’t generalize well beyond their training data. Also they were extremely unreliable so they always needed to either be non-crucial or to have non-ML fallbacks. (One unusually public example: that scandal where black people were auto-labelled as gorillas.) I hear that the whole “AI first” approach has basically been abandoned since then.
Of course, Google still talks about how they increasingly employ DL at every layer of the stack. It’s great hype.
I mean, maybe it’s used somewhere in the design loop, but I doubt it’s particularly central. My guess would be it’s used in one or two tools somewhere which are in practice not-importantly-better than the corresponding non-DL version, but someone stuck a net in there somewhere just so that they could tell a clueless middle manager “it uses deep learning!” and the clueless middle manager would buy this mediocre piece of software.
Misleading. Yeah, computational photography techniques have exploded, but the core tricks are not deep learning at all.
This one I think I basically buy, although I don’t know much about how face detection is done today.
Misleading. Those recommenders presumably aren’t using end-to-end DL; they’re mixing it in in a few specific places. It’s a marginal value add within a larger system, not the backbone of the system.
I basically buy the transcripts and translation examples, and basically don’t buy the spam example—we already had basically-viable spam detection before DL, the value-add there has been marginal at best.
I hear companies wish they could get ML to do this well, but in practice most things still need to loop through humans. That’s epistemic status: hearsay, so not confident, but it matches my priors.
These examples I basically buy.
These seem to work in practice exactly when they’re doing the same thing an n-gram predictor would do, and not work whenever they try to predict anything more ambitious than that.
I would be surprised if DL were doing most of the heavy lifting in Maps’ traffic forecast at this point, although I would not be surprised if it were sprinkled in and hyped up. That use-case should work really well for non-DL machine learning systems (or so I’d guess), which are a lot more transparent to the designers and developers.
Another two places where I doubt that DL is the main backbone, although it may be sprinkled in here and there and hyped up a lot. I doubt that the marginal value-add from DL is all that high in either of these use-cases, since non-DL machine learning should already be pretty good at these sorts of problems.
Stellar breakdown of hype vs. reality. Just wanted to share some news from today that Google has fired an ML scientist for challenging their paper on DL for chip placement.
From Engadget (ungated):
Sounds like challenging the hype is a terminable offense.But see gwern’s context for the article below.“One story is good until another is told”. The chip design work has apparently been replicated, and Metz’s* writeup there has several red flags: in describing Gebru’s departure, he omits any mention of her ultimatum and list of demands, so he’s not above leaving out extremely important context in these departures in trying to build up a narrative of ‘Google fires researchers for criticizing research’; he explicitly notes that Chatterjee was fired ‘for cause’ which is rather eyebrow-raising when usually senior people ‘resign to spend time with their families’ (said nonfirings typically involving things like keeping their stock options while senior people are only ‘fired for cause’ when they’ve really screwed up—like, say, harassment of an attractive young woman) but he doesn’t give what that ‘cause’ was (does he really not know after presumably talking to people?) or wonder why both Chatterjee and Google are withholding it; and he uninterestedly throws in a very brief and selective quote from a presumably much longer statement by a woman involved which should be raising your other eyebrow:
(I note that this is put at the end, which in the NYT house style, is where they bury the inconvenient facts that they can’t in good journalist conscience leave out entirely, and that makes me suspect there is more to this part than is given.)
So, we’ll see. EDIT: Timnit Gebru, perhaps surprisingly, denies any parallel and seems to say Chatterjee deserved to be fired, saying:
Wired has a followup article with more detailed timeline and discussion. It edges much closer to the misogyny narrative than the evil-corporate-censorship narrative.
* yes, the SSC Metz.
Fair enough! Great context, thanks.
In my experience, not enough people on here publically realise their errors and thank the corrector. Nice to see it happen here.