Why isn't AMD's MI300X competitive?

MI300X vs H100 vs H200 Benchmark Part 1: Training - CUDA Moat Still AliveTraining Performance, User Experience, Usability, Nvidia, AMD, GEMM, Attention, Networking, InfiniBand, Spectrum-X Ethernet, RoCEv2 Ethernet, SHARP, Total Cost of OwnershipDylan Patel, Daniel Nishball, and Reyk KnuhtsenDec 22, 2024∙ Paid3ShareIntroSemiAnalysis has been on a five-month long quest to settle the reality of MI300X. In theory, the MI300X should be at a huge advantage over Nvidia’s H100 and H200 in terms of specifications and Total Cost of Ownership (TCO). However, the reality is that the on paper specs as given below are not representative of performance that can be expected in a real-world environment. If AMD could deliver the below marketed performance with this memory, it would be a very strong competitor in the market. Source: SemiAnalysis, Nvidia, AMDToday we are going to talk through our five-month journey conducting independent analysis and training-focused benchmarking of the MI300X, the H100 and the H200, engaging with both NVIDIA and AMD. We will do a detailed overview of the numerous low-level benchmarks that we ran, see the table of contents for summary. Furthermore, we will compare the total cost of ownership of Nvidia and AMD GPUs and factor in performance. Ultimately much of what we are doing is openly giving a comprehensive public recommendation to AMD on what they need to do to be competitive and fix their software issues after five months of submitting and squashing bugs. It’s not just that it’s immature software, they need to change how they do development.In short, when comparing Nvidia’s GPUs to AMD’s MI300X, we found that the potential on paper advantage of the MI300X was not realized due to a lack within AMD public release software stack and the lack of testing from AMD.AMD’s software experience is riddled with bugs rendering out of the box training with AMD is impossible. We were hopeful that AMD could emerge as a strong competitor to NVIDIA in training workloads, but, as of today, this is unfortunately not the case. The CUDA moat has yet to be crossed by AMD due to AMD’s weaker-than-expected software Quality Assurance (QA) culture and its challenging out of the box experience. As fast as AMD tries to fill in the CUDA moat, NVIDIA engineers are working overtime to deepen said moat with new features, libraries, and performance updates.We shared benchmark source code and intermediate test results for GEMM benchmark and Single Node Training with both Nvidia and AMD and held calls and discussions to solicit feedback and implement improvements to the benchmarks, and we worked with AMD to implement bug fixes for the software stacks. Our goal with this highly iterative interaction was to ensure that our tests are an unbiased evaluation of what real-world users would experience. We initially planned to publish this article a few months ago but wanted to take the extra time to engage with the AMD team and explore possible fixes or development work. We spent a considerable time identifying and fixing AMD software bugs so that we could give AMD every chance to show MI300X unhindered by AMD software stack bugs as opposed to only showing problematic performance out of the box. To give a fair impression, we also explain the considerable amount of work on tuning and bug-squashing that it took to get there. We think this approach provides users with the best possible level of transparency. We wanted to contribute in any way we could to try to…

This excerpt is published under fair use for community discussion. Read the full article at Semianalysis.