By publishing an array of
in-house benchmark figures, British AI chip startup Graphcore has mounted a challenge against the market leader for AI acceleration in the data center, Nvidia. Graphcore is claiming significant performance advantages for its second-generation IPU versus state-of-the-art Nvidia GPUs. However, Graphcore has put systems of different sizes head-to-head, saying it has instead compared the Nvidia product that’s closest in price.
...
“The Graphcore numbers are misleading,” said Kevin Krewell, principal analyst at Tirias Research. “Many companies self-publish benchmark and performance data, but those should always be viewed sceptically. The use of performance per dollar is not a good measure for AI systems purchases because there are many other factors in the cost of ownership. Often, performance per rack space is a critical factor.”
...
The majority of Graphcore’s benchmarks compare the IPU-M2000, a system with four IPU-MK2 chips, against a single Nvidia A100 GPU. The company also compares its IPU-Pod64, a system with 64 chips, against one or two Nvidia DGX-A100 systems (8x or 16x A100 chips). The scale of the systems compared in Graphcore’s announcement seems inconsistent, but as with all performance benchmarks, the devil is in the details.
...
Graphcore has said previously that the IPU-M2000 has a recommended retail price of $32,450, though this does not include a CPU server also needed to run the system (Graphcore says this enables freedom of server choice). By comparison, the 8-GPU DGX-A100 starts at $199,000. An Nvidia A100-accelerated server with 4x A100 GPUs (Supermicro A+ Server 2124GQ-NART) including CPU starts in the region of $57,000.
...
Graphcore also announced last week that it will join MLCommons and plans to submit benchmark scores to MLPerf in 2021. The next round of MLPerf inference benchmarks scores will be published in the later part of Q1 2021, with the next round of training scores following in Q2.
“I am glad to see Graphcore join MLCommons and promise to publish further benchmarks in 2021,” said Tirias Research’s Kevin Krewell. “Those benchmark scores will have far more scrutiny by the community. I believe [Graphcore’s self-published] benchmarks will age poorly.”
