NVidia Hopper Speculation, Rumours and Discussion

So it's actually a bit smaller than GA100.

 

no word about clock though ....

 

Math gives 1,775 GHz for 60TF FP32 SXM5 module.
1,645 GHz for the 350W PCIE version.

 

H100 scaling to 700w on air and/or water.

 

Whitepaper table 3 says "not finalized" and figure 13 says 1.3x clock speed compared to A100.

 

There are footnotes speaking of 1,9 GHz edit: Those were given under the impression, that the SXM5 model would use 124 SMs, so maybe that's a bit on the high side for the 132 incarnation. Also not finalized though.

 

https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/

 

p18 of the Whitepaper PDF:
"Only two TPCs in both the SXM5 and PCIe H100 GPUs are graphics-capable (that is, they can run vertex, geometry, and pixel shaders)."

Note how Nvidia refers to productized H100 chips instead of the H100 GPU in general. And also, how it's not the "1 SM" as speculated solely on the basis of a slightly misaligend SM earlier.

 

Whitepaper link was added to the post above: https://nvdam.widen.net/s/9bz6dw7dqr/gtc22-whitepaper-hopper

 

Do the 64 INT32 lanes share data paths with 64 of the FP32 lanes like in Ampere?

 

AFAIU it's conceptually the same setup. Just 2x more execution ressources per register.

 

I see 32 lanes and I think it's pretty much the same as in Ampere (not GA100 but GA10x) - one SIMD is FP32/INT and another is FP32 only. Doesn't make much sense to make a separate INT32 as you won't be able to load it properly.

 

The Whitepaper says it's 16 of each INT32, FP32, FP32, FP64 plus 4 SFU and 8 L/S inside each quarter-cluster of an SM.

 

So, the H100 (1 GPU) has the following advantages vs the MI250X (2 GPUs):

FP32: 60 vs 48
FP32 Matrix: 500 vs 95 (with sparsity: 1000 vs 95)
FP16 Matrix: 1000 vs 383 (with sparsity: 2000 vs 383)
INT8 Matrix: 2000 vs 383 (with sparsity: 4000 vs 383)
INT4 Matrix: 4000 vs 383 (with sparsity: 8000 vs 383)

And all the reliability of a single monolithic GPU die, that has 3TB/s of bandwidth dedicated to it alone, vs the MI250X nature of 2 dies sharing 3.2TB of bandiwdth and a slow interconnect.

Still, the MI250X enjoys a better theoretical FP64 performance vs the H100, when it can utilize both GPUs of course:

FP64: 48 vs 30
FP64 Matrix: 97 vs 60

 

MI250X bandwidth is only 400GB/s when sharing data or 13,3% of H100. With Hopper nVidia introduces their NVLink Switch System with full speed up to 32 nodes. So even inter-node communication is now 2.25x faster than MI200 interconnect...

The loser is the US goverment still building their exascale system which will be beaten by a computer in a basement. And next year with Grace nVidia will be so far ahead that Frontier looks like doa.

 

Funny, it sounds almost like you think they don't know what they want, which in this case wasn't NVIDIA. Next year there will be new products from all which make "Frontier look DOA". But it's completely different story on when they could get a government scale supercomputer up and running on those.

 

AFAICT there's still no option to pair NV GPUs with high-end CPUs and get accelerated unified memory as well. There are virtually no high-end CPUs released yet with PCIe 5.0 either ...

 

“Hopper” GH100 GPUs Are The Heart Of A More Expansive Nvidia System (nextplatform.com)