AMD CDNA: MI300 & MI400 (Analysis, Speculation and Rumors in 2024)

Newguy said:
Couple things looking over the whitepaper again, they doubled the vector (CU) FP32 rate?
Click to expand...
Yea.
Newguy said:
more VOPD shenanigans?
Click to expand...
No it's a more straightforward dual-issue.
940 has no VOPD (the ISA manual is out).
pTmdfx said:
HWS and ACE have known to be firmware running on embedded cores for a few generations, dated back to at least “HWS” was introduced (Polaris?).
Click to expand...
Tonga I think.
pTmdfx said:
while a single agent GFX940 configuration can have only 1 HWS by nature
Click to expand...
I'm not sure if that's correct given even the monodies have two HWS per die.
 
www.performance-intensive-computing.com

AMD and Supermicro: Pioneering AI Solutions

In the constantly evolving landscape of AI and machine learning, the synergy between hardware and software is paramount. Enter AMD and Supermicro, two industry titans who have joined forces to empower organizations in the new world of AI with cutting-edge solutions.
www.performance-intensive-computing.com www.performance-intensive-computing.com

community.amd.com

Unleashing the Open-Source Power of AI through AMD Instinct Accelerators and ROCm

At AMD, we believe that AI is the most important thing to impact computing and technology in the past 50 years, and we remain dedicated to driving innovation and collaboration within the AI industry.   Through our open software ecosystem, ROCm™, and our cutting-edge AI accelerators, we are...
community.amd.com community.amd.com

www.lamini.ai

Multi-node LLM Training on AMD GPUs

Lamini is the enterprise LLM platform for existing software teams to quickly develop and control their own LLMs. Lamini has built-in best practices for specializing LLMs on billions of proprietary documents to improve performance, reduce hallucinations, offer citations, and ensure safety. Lamini...
www.lamini.ai www.lamini.ai
 
MI325X is the HBM3E CDNA3 refresh with 288GB and 6TB/s bandwidth with increased compute coming this year.

MI350 is CDNA4 on 3nm coming in 2025. 35x inference improvement(FP4/FP6) compared to CDNA3.
1.5x more memory and 1.2x "AI compute" TFlops compared to B200.

MI400 is CDNA Next coming in 2026.
AMD Computex 2024 stream link
 
Last edited:
TopSpoiler said:
AMD’s MI300X Outperforms NVIDIA’s H100 for LLM Inference
www.blog.tensorwave.com

MI300X vs. H100 for LLM Inference | TensorWave

MI300X achieves 33% higher throughput compared to the H100 running vLLM on Mixtral 8x7B for a real-world chat use case on TensorWave cloud
www.blog.tensorwave.com www.blog.tensorwave.com

Edit: Tensorwave seems to have chosen the slowest and unoptimized inference engine for H100.
Click to expand...
The thing is that NVDA has tripled H100 performance by huge software optimizations since it was announced. AMD is not competing a still target like they did with Intel CPUs...

MLPerf-Training-4.0-Nvidia-LLM-performance.png
 
Last edited:
Based on authenticated and published test results only Intel is the second best option according the MLPerf tests.
It's a marketing attempt by Tensorwave to generate sales as they did something similar a few months ago.

Pretty unusual for AMD to miss both MLPerf tests with hardware available. They seem to want to move on pretty quick from MI300x.
 
xpea said:
The thing is that NVDA has tripled H100 performance by huge software optimizations since it was announced. AMD is not competing a still target like they did with Intel CPUs...
Click to expand...
AFAICT, they have a significant ~27% performance gain for 512 H100s (iso-GPU-count) from June 2023 to June 2024, but they are also comparing 2023 to 2024 numbers at 3.2x performance with 3.2x GPUs to claim "perfect scaling" - even though the June 2024 numbers for 3584 H100s wouldn be noticeably higher than that, so they do not actually achieve perfect scaling, but more like ~85% which is pretty standard...

EDIT: ... to bring it back to AMD/MI300X ...

For training, AMD is competing to something "close enough" to a fixed target in my opinion, +27% by SW in a year is nothing amazing in this industry, there have been a lot of optimisations in PyTorch and generic things like Flash Attention 2 which also applies to AMD. I wouldn't be surprised if NVIDIA-specific perf improvements for training were more like 10-15% which is kinda what you'd expect in that timeframe for a new-ish architecture.

For inference, both NVIDIA *and* AMD are extremely fast moving targets, partly due to inference framework improvements, and NVIDIA's unique advantage here is TensorRT which AMD doesn't have an in-house alternative to. But there are 3rd party optimised inference frameworks that can be compared, e.g. MK1 Flywheel which is used in TensorWave's blog post - however comparing that to vllm on NVIDIA is a bit silly. Either you compare the same frameworks (e.g. vllm or MK1 on both NVIDIA and AMD) or you compare the best you can find on AMD (e.g. MK1 Flywheel?) to the best you can find on NVIDIA (e.g. TensorRT). So TensorWave's blog post is definitely not a fair comparison, and there's a bit of a pattern from them of not very rigorous claims imo...
 
Arun said:
I have to hand it to NVIDIA's product & marketing teams, this is so confusingly misleading that I honestly don't know what they are trying to say...

AFAICT, they have a significant ~27% performance gain for 512 H100s (iso-GPU-count) from June 2023 to June 2024, but they are also comparing 2023 to 2024 numbers at 3.2x performance with 3.2x GPUs to claim "perfect scaling" - even though the June 2024 numbers for 3584 H100s wouldn be noticeably higher than that, so they do not actually achieve perfect scaling...
Click to expand...
I don't find to much confusing about the statement. "Nearly perfect scaling" and "perfect scaling" are not the same thing, and they also mentioned the 3.2x increase was due to larger scale and software improvements.
The result of this work is a 3.2x performance increase in just a year coming from a larger scale, and significant software improvements. This combination also delivered nearly perfect scaling — as the number of GPUs increased by 3.2x, so did the delivered performance.
Click to expand...

Anyway not the thread for this discussion.
 
You must log in or register to reply here.
Back
Top