Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
The paper discusses the challenges of scheduling multiple Large Language Models (LLMs) on shared hardware. It highlights the performance implications of layer offloading and preemption, noting that smaller models are more sensitive to GPU residency. The authors propose key features for future schedulers to improve efficiency in managing heterogeneous, multi-model workloads.
- ▪Modern deployments of LLMs require serving multiple models with diverse architectures on shared hardware.
- ▪Offloading leads to non-linear degradation in decode throughput, particularly affecting smaller models.
- ▪Preemption incurs significant overhead, primarily due to model state reload, which varies across models and hardware.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Artificial Intelligence arXiv:2605.19593 (cs) [Submitted on 19 May 2026] Title:Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption Authors:Mert Yildiz, Pietro Spadaccino, Alexey Rolich, Francesca Cuomo, Andrea Baiocchi View a PDF of the paper titled Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption, by Mert Yildiz and 4 other authors View PDF HTML (experimental) Abstract:Modern deployments of Large Language Models (LLMs) increasingly require serving multiple models with diverse architectures, sizes, and specialization on shared, heterogeneous hardware.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI
