Making Deep Learning Go Brrrr from First Principles
Optimizing deep learning performance requires understanding the underlying system bottlenecks rather than relying on ad-hoc tricks. The three main components affecting efficiency are compute, memory bandwidth, and overhead, each requiring different optimization strategies. By identifying the dominant bottleneck, developers can focus on meaningful improvements that align with hardware capabilities.
- ▪Deep learning performance optimization should be based on identifying whether the system is compute-bound, memory-bound, or overhead-limited.
- ▪Increasing GPU FLOPS won't help if the bottleneck is memory bandwidth, and reducing overhead won't help if the system is compute-bound.
- ▪Modern accelerators like GPUs achieve peak performance mainly on matrix multiplication operations, with other operations contributing negligibly to total FLOP count.
- ▪Specialized hardware such as Tensor Cores means non-matrix multiplication operations are significantly slower in comparison.
- ▪The growth rate of compute outpaces memory bandwidth, making it increasingly difficult to fully utilize hardware capacity.
Opening excerpt (first ~120 words) tap to expand
Making Deep Learning Go Brrrr From First Principles So, you want to improve the performance of your deep learning model. How might you approach such a task? Often, folk fall back to a grab-bag of tricks that might've worked before or saw on a tweet. "Use in-place operations! Set gradients to None! Install PyTorch 1.10.0 but not 1.10.1!" It's understandable why users often take such an ad-hoc approach performance on modern systems (particularly deep learning) often feels as much like alchemy as it does science. That being said, reasoning from first principles can still eliminate broad swathes of approaches, thus making the problem much more approachable. For example, getting good performance on a dataset with deep learning also involves a lot of guesswork.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Horace.
Discussion
0 commentsMore from Horace
