They aren’t believers in AI..but they are believers in money (to be fair to your theory, Nvidia was an early believer in AI...to make money
Nvidia wasn’t really an early believer unless you define ‘early’ so generously as to be more or less meaningless, like ‘anyone into DL before AlphaGo’. Your /r/ML link actually inadvertently demonstrates that: distributing a (note the singular in both comments) K40 (released 2013, ~$5k and rapidly declining) here or there as late as ~2014 is not a major investment or what it looks like when a large company is an ‘early believer’. The recent New Yorker profile of Huang covers this and Huang’s admission that he blew it on seeing DL coming and waited a long time before deciding to make it a priority of Nvidia’s—in 2009, they wouldn’t even give Geoff Hinton a single GPU when he asked after a major paper, and their CUDA was never intended for neural networks in the slightest.
And even now, they seem to be surprisingly reluctant to make major commitments to TSMC to ensure a big rampup of B100s and later. As I understand it, TSMC is extremely risk-averse and won’t expand as much as it could unless customers underwrite it in advance so that they can’t lose, and still thinks that AI is some sort of fad like cryptocurrencies that will go bust soon; this makes sense because that sort of deeply-hardwired conservatism is what it takes to survive the semiconductor boom-bust and gambler’s ruin and be one of the last chip fabs left standing. And why Nvidia won’t make those commitments may be Huang’s own conservatism from Nvidia’s early struggles, strikingly depicted in the profile. This sort of corporate DNA may add delay you wouldn’t anticipate from looking at how much money is on the table. I suspect that the ‘all TSMC’s capacity is turned over to AI’ point may take longer than people expect due to their stubbornness. (Which will contribute to the ‘future is already here, just unevenly distributed’ gradient between AI labs and global economy—you will have difficulty deploying your trained models at economical scale.)
And even now, they seem to be surprisingly reluctant to make major commitments to TSMC to ensure a big rampup of B100s and later. As I understand it, TSMC is extremely risk-averse and won’t expand as much as it could unless customers underwrite it in advance so that they can’t lose, and still thinks that AI is some sort of fad like cryptocurrencies that will go bust soon; this makes sense because that sort of deeply-hardwired conservatism is what it takes to survive the semiconductor boom-bust and gambler’s ruin and be one of the last chip fabs left standing.
Giving away free hardware vs extremely risk averse seems mildly contradictory, but I will assume you mean in actual magnitudes. Paying TSMC to drop everything and make only B100s is yeah, a big gamble they probably won’t make since it would cost billions, while a few free cards is nothing.
So that will slow the ramp down a little bit? Would it have mattered? 2012 era compute would be ~16 times slower per dollar, or more if we factor in lacking optimizations, transformer hasn’t been invented so less efficient networks would be used, etc.
The “it could just be another crypto bubble” is an understandable conclusion. Remember, GPT-4 requires a small fee to even use, and for the kind of senior people who work at chip companies, many of them haven’t even tried it.
You have seen the below, right? To me this looks like a pretty clear signal as to what the market wants regarding AI...
Nvidia wasn’t really an early believer unless you define ‘early’ so generously as to be more or less meaningless, like ‘anyone into DL before AlphaGo’. Your /r/ML link actually inadvertently demonstrates that: distributing a (note the singular in both comments) K40 (released 2013, ~$5k and rapidly declining) here or there as late as ~2014 is not a major investment or what it looks like when a large company is an ‘early believer’. The recent New Yorker profile of Huang covers this and Huang’s admission that he blew it on seeing DL coming and waited a long time before deciding to make it a priority of Nvidia’s—in 2009, they wouldn’t even give Geoff Hinton a single GPU when he asked after a major paper, and their CUDA was never intended for neural networks in the slightest.
And even now, they seem to be surprisingly reluctant to make major commitments to TSMC to ensure a big rampup of B100s and later. As I understand it, TSMC is extremely risk-averse and won’t expand as much as it could unless customers underwrite it in advance so that they can’t lose, and still thinks that AI is some sort of fad like cryptocurrencies that will go bust soon; this makes sense because that sort of deeply-hardwired conservatism is what it takes to survive the semiconductor boom-bust and gambler’s ruin and be one of the last chip fabs left standing. And why Nvidia won’t make those commitments may be Huang’s own conservatism from Nvidia’s early struggles, strikingly depicted in the profile. This sort of corporate DNA may add delay you wouldn’t anticipate from looking at how much money is on the table. I suspect that the ‘all TSMC’s capacity is turned over to AI’ point may take longer than people expect due to their stubbornness. (Which will contribute to the ‘future is already here, just unevenly distributed’ gradient between AI labs and global economy—you will have difficulty deploying your trained models at economical scale.)
Giving away free hardware vs extremely risk averse seems mildly contradictory, but I will assume you mean in actual magnitudes. Paying TSMC to drop everything and make only B100s is yeah, a big gamble they probably won’t make since it would cost billions, while a few free cards is nothing.
So that will slow the ramp down a little bit? Would it have mattered? 2012 era compute would be ~16 times slower per dollar, or more if we factor in lacking optimizations, transformer hasn’t been invented so less efficient networks would be used, etc.
The “it could just be another crypto bubble” is an understandable conclusion. Remember, GPT-4 requires a small fee to even use, and for the kind of senior people who work at chip companies, many of them haven’t even tried it.
You have seen the below, right? To me this looks like a pretty clear signal as to what the market wants regarding AI...