the custom unit fallacy - modern GPU workloads more dependant on memory than computational power? *spawn

[Potato Head]

There was never any magic solution to RT or AI other than "more memory performance". More fixed function or other specialized HW logic alone won't net you major gains ...

Hmmmm, I wonder why Microsoft wants an entire nuke power plant to run GPUs if they are just sitting around underutilized waiting on memory accesses. It would seem to make more sense to load up on 3070’s instead of $15K a pop H100’s, no?

 

[MfA]

If your devs earn high 6 figures and demand the easiest to work with hardware, what you gonna do? I suspect the Chinese have far better training/inference architectures while Americans paper over poor architecture with hardware.

HBM/NVLink (and the also rans equivalents, Infinity Fabric etc) are the SMP of the modern age.