Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training
The paper introduces Hybrid-LoRA, a new framework for post-training large language models. This approach combines full fine-tuning with low-rank adaptation to optimize performance while reducing computational costs. Experiments demonstrate that Hybrid-LoRA achieves performance close to full fine-tuning while being more efficient.
- ▪Hybrid-LoRA selectively applies full fine-tuning to a small subset of modules that are less suited to low-rank adaptation.
- ▪The framework introduces a Hybrid-LoRA Score to rank modules based on their sensitivity to low-rank adaptation.
- ▪Experiments show improvements of up to 5.65% over state-of-the-art parameter-efficient fine-tuning baselines.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Machine Learning arXiv:2605.18822 (cs) [Submitted on 12 May 2026] Title:Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training Authors:Chengqian Zhang, Wei Zhu, Kyumin Lee View a PDF of the paper titled Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training, by Chengqian Zhang and 2 other authors View PDF HTML (experimental) Abstract:Post-training has become essential for adapting large language models (LLMs) to complex downstream behaviors, including instruction following, preference alignment, and multi-step reasoning.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI
