UniSD: Towards a Unified Self-Distillation Framework for Large Language Models
May 7, 2026·
,,,,,,,,,,·
1 min read
Yiqiao Jin
Yiyang Wang
Lucheng Fu
Yijia Xiao
Yinyi Luo
Haoxin Liu
B. Aditya Prakash
Chao Zhang
Josiah Hester
Jindong Wang
Srijan Kumar
Abstract
UniSD studies how to adapt autoregressive language models through self-distillation without stronger external teachers. It combines multi-teacher agreement, EMA teacher stabilization, token-level contrastive learning, feature matching, and divergence clipping to examine supervision reliability, representation alignment, and training stability. Experiments across six benchmarks and six models identify complementary components for an integrated self-distillation pipeline.
Type
Publication
COLM 2026 LLA Workshop
Overview
Accepted at COLM 2026 LLA Workshop.
UniSD studies how to adapt autoregressive language models through self-distillation without stronger external teachers. It brings together teacher agreement, EMA stabilization, contrastive learning, feature matching, and divergence clipping, examining their individual effects and interactions across six benchmarks and six models.
