a. Just recognizing objects with something like human levels of accuracy takes hundreds of teraflops per second! We call it “TOPS” and just keeping up with a few full resolution cameras with networks that are not as robust as humans costs about 300-400 TOPS, or ‘all she’s got’ from a current gen accelerator board. This is like an inferior version of a human visual cortex, with our main issue being lack of generality and all sorts of terrible edge cases [that can misidentify objects, leading to a crash]
Hundreds of teraflops in a single chip wasn’t available until a few years ago, where several companies [Tesla, Nvidia, Waymo] developed NN accelerators.
b. You don’t understand the computer architecture consequences when we say a brain has 2.5 petabytes. This is not the same as having 2.5 petabytes of data where only a tiny fraction is being accessed at any time. At any given moment, any of the neurons in the brain might get hit with a spike train and have to give outputs, taking into account the connection weight and connectivity graph. The graph (what connects to what) and the strength and type of each connection are all information, and this is where the 2.5 petabyte estimate is coming from—the number of connections (86 billion times 1000) and how many bits of information you estimate each connection holds.
(2.5 petabyte) / (1000 * 86 billion) = 29 bytes, apparently that is all scientific America thinks a synapse holds. Say 8 bits of resolution for the weight, some “in progress” state variables (there are variables that are changed over time that are used to update the weights for learning), and enough bytes to specify uniquely that synapse’s relative position in a graph with 86 trillion entries.
Anyways your computer must be able to compute on all 2.5 petabytes at all times. Every timestep, even neurons that are not going to fire are checking to see if they should, and they must access the synaptic weights to do this and update in progress state variables. The brain isn’t synchronous but 1000 timesteps per synapse per second of realtime is a rough estimate of what you need.
This is architecturally sorta like having all 2.5 petabytes at a minimum in RAM with a very very fast bus connecting you to the chip, but if you really get serious you need massive caches or you need to just build the circuitry that evaluates the neural network directly on top of the storage medium that holds the weights. (several startups are doing this)
Let’s make a more concrete estimate. We have 2.5 petabytes of values we need to access 1000 times a second.
Therefore, our bandwidth requirement is 2.5 petabytes * 1000 = 2.5 exabytes/second . Each Nvidia A100 has 2 terabytes/second of bandwidth. Therefore for 1 brain equivalent we need 1,250,000 A100 AI accelerators, all released May 14, 2020. World’s largest supercomputer uses 158,976 nodes so this would be 10x larger.
Also the amount of traffic between A100 nodes probably exceeds available interconnects but I’m not sure about that assertion.
Each A100 is $199,000 MSRP. But probably there is a chip shortage so you would need to wait a few years for your order to fill.
So you need 248.75 billion in A100 accelerators to get to “1 brain worth” of capacity. And current AI algorithms are much less efficient than humans so..
Please note, I fully believe TAI is possible but with numbers like these it starts to become apparent why we don’t have it yet. Also this explains neatly the “AI winter”, the AI winter happened because it turned out, with computers of that era, meaningful progress wasn’t possible.
Also note the 200k pricetag is Nvidia’s MSRP, which factors in their need to pay back all the money they spent on R&D. They ‘only’ spent a few billion and maybe you could cut a deal. Each A100 likely only costs $1000 or less in real component costs.
I understand his point is not that we have enough CPU and RAM to simulate a human brain. We do not. His point seems to be that the observable memory capacity of the human brain is on the order of TB to PB. He doesn’t go too deep into the compute part but the analogy with self-driving cars seems suitable. After all quite a big part of the brain is devoted to image processing and object detection. I think it is not inconceivable that there are better algorithms than what the brain has to make do with for the intelligence part.
He’s specifically talking about building a computer not any more efficient than a brain algorithm wise and saying we have enough compute to do this.
He is incorrect because he is not factoring in the compute architecture. The reason you should consider my judgement is I am a working computer engineer and I have personally designed systems (for smaller scale tasks, I am not the architect for the self driving team I work for now)
Of course more efficient algorithms exist but by definition they take time and effort to find. And we do not know how much more efficient a system we can build and still have sentience.
Here’s why you are wrong:
a. Just recognizing objects with something like human levels of accuracy takes hundreds of teraflops per second! We call it “TOPS” and just keeping up with a few full resolution cameras with networks that are not as robust as humans costs about 300-400 TOPS, or ‘all she’s got’ from a current gen accelerator board. This is like an inferior version of a human visual cortex, with our main issue being lack of generality and all sorts of terrible edge cases [that can misidentify objects, leading to a crash]
Hundreds of teraflops in a single chip wasn’t available until a few years ago, where several companies [Tesla, Nvidia, Waymo] developed NN accelerators.
b. You don’t understand the computer architecture consequences when we say a brain has 2.5 petabytes. This is not the same as having 2.5 petabytes of data where only a tiny fraction is being accessed at any time. At any given moment, any of the neurons in the brain might get hit with a spike train and have to give outputs, taking into account the connection weight and connectivity graph. The graph (what connects to what) and the strength and type of each connection are all information, and this is where the 2.5 petabyte estimate is coming from—the number of connections (86 billion times 1000) and how many bits of information you estimate each connection holds.
(2.5 petabyte) / (1000 * 86 billion) = 29 bytes, apparently that is all scientific America thinks a synapse holds. Say 8 bits of resolution for the weight, some “in progress” state variables (there are variables that are changed over time that are used to update the weights for learning), and enough bytes to specify uniquely that synapse’s relative position in a graph with 86 trillion entries.
Anyways your computer must be able to compute on all 2.5 petabytes at all times. Every timestep, even neurons that are not going to fire are checking to see if they should, and they must access the synaptic weights to do this and update in progress state variables. The brain isn’t synchronous but 1000 timesteps per synapse per second of realtime is a rough estimate of what you need.
This is architecturally sorta like having all 2.5 petabytes at a minimum in RAM with a very very fast bus connecting you to the chip, but if you really get serious you need massive caches or you need to just build the circuitry that evaluates the neural network directly on top of the storage medium that holds the weights. (several startups are doing this)
Let’s make a more concrete estimate. We have 2.5 petabytes of values we need to access 1000 times a second.
Therefore, our bandwidth requirement is 2.5 petabytes * 1000 = 2.5 exabytes/second . Each Nvidia A100 has 2 terabytes/second of bandwidth. Therefore for 1 brain equivalent we need 1,250,000 A100 AI accelerators, all released May 14, 2020. World’s largest supercomputer uses 158,976 nodes so this would be 10x larger.
Also the amount of traffic between A100 nodes probably exceeds available interconnects but I’m not sure about that assertion.
Each A100 is $199,000 MSRP. But probably there is a chip shortage so you would need to wait a few years for your order to fill.
So you need 248.75 billion in A100 accelerators to get to “1 brain worth” of capacity. And current AI algorithms are much less efficient than humans so..
Please note, I fully believe TAI is possible but with numbers like these it starts to become apparent why we don’t have it yet. Also this explains neatly the “AI winter”, the AI winter happened because it turned out, with computers of that era, meaningful progress wasn’t possible.
Also note the 200k pricetag is Nvidia’s MSRP, which factors in their need to pay back all the money they spent on R&D. They ‘only’ spent a few billion and maybe you could cut a deal. Each A100 likely only costs $1000 or less in real component costs.
I understand his point is not that we have enough CPU and RAM to simulate a human brain. We do not. His point seems to be that the observable memory capacity of the human brain is on the order of TB to PB. He doesn’t go too deep into the compute part but the analogy with self-driving cars seems suitable. After all quite a big part of the brain is devoted to image processing and object detection. I think it is not inconceivable that there are better algorithms than what the brain has to make do with for the intelligence part.
He’s specifically talking about building a computer not any more efficient than a brain algorithm wise and saying we have enough compute to do this.
He is incorrect because he is not factoring in the compute architecture. The reason you should consider my judgement is I am a working computer engineer and I have personally designed systems (for smaller scale tasks, I am not the architect for the self driving team I work for now)
Of course more efficient algorithms exist but by definition they take time and effort to find. And we do not know how much more efficient a system we can build and still have sentience.