Not made for training.
Kinda redundant copium hit.
Point is, it doesn't.
So does MI300, the biggest delta is server-level perf after all.
Not really, it's like two new TOs.
Same for MI300A, still not ramped up.
Which is not the point? These are AI GPUs, who cares about HPC performance (FP64)?
Dylan Patel from SemiAnalysis (the most accurate source of information for HPC/AI on the web) says the cost is actually 2x of H100.
A is the less important product.
Are you serious?
No, he's our pet.
Aha. Gotcha (thought you were talking about HPC). However, at the AI front, there isn't any advantage for MI300X over the standard H100 (let alone the H200) aside from the more memory capacity, which is why AMD focused only on these edge cases (memory capacity sensitive inference tests). AMD also avoided using TensorRT LMM standard tests for some unknown reason, these are the defacto tests in the realm of AI.
www.semianalysis.com
Disagree here.
Well no, it has more math and more membw.
Well yeah, that's kinda the point of building a more expensive 8 stack machine.
LLaMA2 70B is the least edge cases imaginable.
They used vLLM instead which is what the industry at large uses.
No, AMD server-level perf numbers were very much in-line with what the industry expects since this is what Meta asked of them. lol.
Again, he was wrong and I was right, that's all there is to it.
MI300 is the absolut bottom for AMD. Barely faster than H100. The successor will be announced in three months. The H200 refresh will be much faster in inference. And nVidia is working with Amazon on a inference rack combining 32 Superchips with 900GB/s resulting in 20TB of memory.According to AMD, it's basically on par with H100. Which probably means it actually isn't. Also, NVIDIA has H200 now, which is even faster.
AMD showed that MI300X does win significantly against H100 in select cases, when the H100 runs out of memory, as H100 has 80GB, while the MI300X has double that: 192GB. However, H200 has 140GB, which means it fixes these cases.
MI300 is much more expensive and much more complicated to make than H100, MI300X is more power hungry too.
80+% of world's Ai training is using Nvidia HW so stop your usual BS, the standard is Nvidia, like it or not, thus the standard is whatever library runs faster on NV. End of discussion.
Dawg we're talking inference farms.
What stops AMD from bolting 8 HBM3e's on MI300?
300X already using e version
No, they're standard 5.2Gbps or whatever parts.
And it doesnt matter because Blackwell will be much faster. MI300X is barely faster than H100 in inference. Even AMD's own numbers put it only around +20%. Its a dead product.
Nope, it's using 5.2 Gbps HBM3.
Only memory bandwidth, and against the H100, not the H200. As for Math, not so much.
Disagree here, it's not, it's gonna sell whatever AMD can produce from it, the demand is so high that everything decent in the market will be gobbled up.
Keep in mind this includes their HPC contracts, as MI300 was an HPC product first and foremost, Funded by the El-Captian super computer contract. AMD modified the design on the fly once the AI bubble blew up. So yes AMD is not expecting too much money from AI contracts (relatively speaking of course). However, MI400 will be AI focused and will build on the foundation of MI300, it should also be released relatively quickly.
Not to mention the new version of TensorRT LLM released late November provides a 5x increase in tokens/second compared to the October release. Other solutions like PyTorch are available though might not be as performant, most organizations will use the solution inline with corporate AI strategy and goals.
