Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs
The paper introduces Ghosted Layers, a method designed to recover performance in layer-pruned large language models (LLMs). It addresses the activation mismatch caused by removing entire Transformer decoder blocks, which typically leads to degraded model performance. The proposed solution offers a training-free approach that consistently improves accuracy and perplexity across various LLM architectures.
- ▪Layer pruning can significantly degrade the performance of large language models by creating activation mismatches.
- ▪Ghosted Layers is a training-free recovery module that solves the boundary activation alignment problem.
- ▪The method derives a closed-form optimal linear operator from a small calibration set to reconstruct activation discrepancies.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Machine Learning arXiv:2605.15491 (cs) [Submitted on 15 May 2026] Title:Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs Authors:Vincent-Daniel Yun, Junhyuk Jo, Sai Praneeth Karimireddy, Sunwoo Lee View a PDF of the paper titled Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs, by Vincent-Daniel Yun and 3 other authors View PDF HTML (experimental) Abstract:Layer pruning removes entire Transformer decoder blocks from large language models, but introduces a mismatch between the hidden state received by the next surviving layer and the distribution it was trained to process, leading to significant performance degradation.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI
