I don’t really think H100s are thousands of times better than consumer GPUs.
The big difference between H100s and a consumer GPU like an RTX 5080 is not the number of TFLOP/s (for both its like 50 TFLOP/s), but the VRAM, which is 80 GB for the H100 and 16 GB for the 5080.
VRAM “visual edit: video RAM” is the maximum amount of data you can store on a GPU for fast operations. This lets you more easily train bigger models on more data.
I think this is partially inaccurate, I wasn’t considering the fact that the H100 has a few optimizations for AI specific workloads (eg it is much faster when doing low-precision calculations), and their higher memory bandwidth (~the speed at which vram can move).
The big difference between H100s and a consumer GPU like an RTX 5080 is not the number of TFLOP/s (for both its like 50 TFLOP/s), but the VRAM, which is 80 GB for the H100 and 16 GB for the 5080.
VRAM “
visualedit: video RAM” is the maximum amount of data you can store on a GPU for fast operations. This lets you more easily train bigger models on more data.(the RTX 5090 has 32 GB of VRAM)
I think this is partially inaccurate, I wasn’t considering the fact that the H100 has a few optimizations for AI specific workloads (eg it is much faster when doing low-precision calculations), and their higher memory bandwidth (~the speed at which vram can move).