Comparing a H100 to “consumer CPUs” doesn’t make any sense, you should compare them to consumer GPU lines because that’s where you’d start running into this prohibition first.
And I don’t really think H100s are thousands of times better than consumer GPUs.
I don’t really think H100s are thousands of times better than consumer GPUs.
The big difference between H100s and a consumer GPU like an RTX 5080 is not the number of TFLOP/s (for both its like 50 TFLOP/s), but the VRAM, which is 80 GB for the H100 and 16 GB for the 5080.
VRAM “visual edit: video RAM” is the maximum amount of data you can store on a GPU for fast operations. This lets you more easily train bigger models on more data.
I think this is partially inaccurate, I wasn’t considering the fact that the H100 has a few optimizations for AI specific workloads (eg it is much faster when doing low-precision calculations), and their higher memory bandwidth (~the speed at which vram can move).
Hey, thanks for the feedback! I helped write this section. A few notes:
I think you’re right that comparing to consumer GPUs might make more sense, but I think comparing to other computers is still acceptable. I agree that GPUs is where you start running into prohibitions first. But I think it’s totally fair to compare to “average computers” because one of the main things I care about is the cost of the treaty. It’s not so bad if we have to ban top of the line consumer GPUs, but it would be very costly / impossible if we have to ban consumer laptops. So comparing to both of these is reasonable.
The text says “consumer CPUs” because this is what is discussed in the relevant source, and I wanted to stick to that. Due to some editing that happened, it might not have been totally clear where the claim was coming from. The text has been updated and now there’s a clear footnote.
I know that “consumer CPUs” is not literally the best comparison for, say, consumer laptops. For example, macbooks have an integrated CPU-GPU. I think it is probably true that H100s are like 3-300x better than most consumer laptops at AI tasks, but to my knowledge there is no good citable work explaining this for a wide variety of consumer hardware (I have some mentees working on it now, maybe in a month or two there will be good work!).
I’ll toss out there that NVIDIA sells personal or desktop GPUs that are marketed as for AI (like this one). These are quite powerful, often within 3x of the datacenter GPUs in terms of most of their performance. I expect these to get categorized as “AI chips” under the treaty and thus become controlled. The difference between H100s and top consumer GPUs is not 1000x, and it probably isn’t even 10x. In this tentative draft treaty, we largely try to punt questions like “what exactly counts as an AI chip” to the hypothetical technical body that helps implement the treaty, and my current opinions about this are weak.
Note that most of the compute in consumer laptops is in their GPUs not their CPUs, so comparing H100 flops to laptop CPU flops does not work for establishing the extent to which your policy would affect consumer laptops.
Comparing a H100 to “consumer CPUs” doesn’t make any sense, you should compare them to consumer GPU lines because that’s where you’d start running into this prohibition first.
And I don’t really think H100s are thousands of times better than consumer GPUs.
The big difference between H100s and a consumer GPU like an RTX 5080 is not the number of TFLOP/s (for both its like 50 TFLOP/s), but the VRAM, which is 80 GB for the H100 and 16 GB for the 5080.
VRAM “
visualedit: video RAM” is the maximum amount of data you can store on a GPU for fast operations. This lets you more easily train bigger models on more data.(the RTX 5090 has 32 GB of VRAM)
I think this is partially inaccurate, I wasn’t considering the fact that the H100 has a few optimizations for AI specific workloads (eg it is much faster when doing low-precision calculations), and their higher memory bandwidth (~the speed at which vram can move).
Hey, thanks for the feedback! I helped write this section. A few notes:
I think you’re right that comparing to consumer GPUs might make more sense, but I think comparing to other computers is still acceptable. I agree that GPUs is where you start running into prohibitions first. But I think it’s totally fair to compare to “average computers” because one of the main things I care about is the cost of the treaty. It’s not so bad if we have to ban top of the line consumer GPUs, but it would be very costly / impossible if we have to ban consumer laptops. So comparing to both of these is reasonable.
The text says “consumer CPUs” because this is what is discussed in the relevant source, and I wanted to stick to that. Due to some editing that happened, it might not have been totally clear where the claim was coming from. The text has been updated and now there’s a clear footnote.
I know that “consumer CPUs” is not literally the best comparison for, say, consumer laptops. For example, macbooks have an integrated CPU-GPU. I think it is probably true that H100s are like 3-300x better than most consumer laptops at AI tasks, but to my knowledge there is no good citable work explaining this for a wide variety of consumer hardware (I have some mentees working on it now, maybe in a month or two there will be good work!).
I’ll toss out there that NVIDIA sells personal or desktop GPUs that are marketed as for AI (like this one). These are quite powerful, often within 3x of the datacenter GPUs in terms of most of their performance. I expect these to get categorized as “AI chips” under the treaty and thus become controlled. The difference between H100s and top consumer GPUs is not 1000x, and it probably isn’t even 10x. In this tentative draft treaty, we largely try to punt questions like “what exactly counts as an AI chip” to the hypothetical technical body that helps implement the treaty, and my current opinions about this are weak.
Note that most of the compute in consumer laptops is in their GPUs not their CPUs, so comparing H100 flops to laptop CPU flops does not work for establishing the extent to which your policy would affect consumer laptops.