Distillation: The One Route Across Architectures and Clouds
In short: Distillation teaches a student to imitate a teacher's answers. Unlike merge/graft, which move weights (numeric parts) directly, distillation copies only the outward behavior (outputs). That difference lets the internals differ completely — and even a closed cloud model (GPT, Claude) can be the teacher.
Distillation teaches a student to imitate a teacher's answers. Unlike merge/graft, which move weights (numeric parts) directly, distillation copies only the outward behavior (outputs). That difference lets the internals differ completely — and even a closed cloud model (GPT, Claude) can be the teacher. This post builds from "how does copying answers even teach anything?" up to bridging heterogeneous and cloud models.
Why does copying answers even teach?#
The key: the teacher gives not one hard answer but a "probability distribution." Ask "what animal is this?" and the teacher doesn't just say "cat." Internally it holds "cat 92%, tiger 6%, dog 1.5%, car 0.01%…" (soft labels). That carries more than the answer — the nuance that "it's a cat but quite tiger-like, and nothing like a car."
The student imitates that nuance too, so it learns far more than "copy the answer." It's apprentice-style — learning the grain of the teacher's judgment. For generation, this extends to whole sequences (sequence-level KD), training the student to reproduce the teacher's full sentences.
White-box vs black-box?#
Two kinds. White-box learns from the teacher's internals (logits/distributions) — needs open weights (MiniLLM, GKD). Black-box learns only from the teacher's output text — an API is enough, so closed models like GPT/Claude can teach.
Famous black-box cases: Stanford Alpaca trained LLaMA on 52K OpenAI-model answers; Vicuna used ShareGPT ChatGPT conversations. The big 2025 event was DeepSeek-R1-Distill — training Qwen/Llama students on 800K reasoning traces beat running RL directly on the small base.
How do you bridge heterogeneous models / different dictionaries?#
A natural snag: distillation follows the teacher's answers, but if teacher and student have different tokenizers (the dictionary that splits words into tokens), tokens don't line up. How to bridge?
FuseLLM/FuseChat are the classic fix. They fuse several heterogeneous teachers' distributions into one student using token alignment like minimum edit distance (MinED). ULD (optimal-transport loss) even removes the shared-tokenizer requirement. In practice people also "distill heterogeneous teachers into one family, then merge" (fuse-then-merge). So distillation is the bridge across the "heterogeneity wall" that merging can't cross.
So how exactly does it differ from merging?#
Merge moves parts (weights) directly — fast, near-lossless, but walled in by same-family/open-weights. Distillation imitates answers — crosses architectures and clouds, but needs gradient training (cost) and loses a little through approximation.
| Axis | Merge | Distill |
|---|---|---|
| Operates in | Weights (parts) | Behavior (answers) |
| Gradient training | None | Required (train student) |
| Cross-architecture | No | Yes |
| Different tokenizer | No | Yes (needs alignment) |
| Cloud teacher | No | Yes (answers only) |
| Info loss | ~None | Approximation loss |
One step further — why isn't copying answers alone enough?#
Just collecting a teacher's answers offline and copying them causes distribution shift — the data mismatches the mistakes the student actually makes. So modern methods use on-policy distillation: the student generates first, and the teacher corrects that text with its judgment/probabilities (MiniLLM, GKD). Targeting real weaknesses, it learns more from the same data.
Latest trends (2026)#
Two threads. First, the DeepSeek-R1 distillation wave — training small open models on frontier outputs became standard. Second, cross-tokenizer distillation — ULD / approximate-likelihood matching made distillation work even across different tokenizers, widening the heterogeneity wall that merging can't cross. To bridge different families, the answer is always distillation.
One-line: distillation learns by imitating the teacher's "answers (distribution)," so it moves ability across the heterogeneous/cloud boundary merging can't — at the cost of student training and slight approximation loss.
Note: As of 2026-07-01. Training on a teacher's outputs is subject to each model's license/terms (some restrict training competing models).
Responses
No responses yet. Be the first to respond.