How Models Inherit Skills: Merge, Chimera, or Distill
In short: Every week the open-source world ships powerful new models. Yet most were not trained from scratch — they recombine abilities "inherited" from models that are already good. There are three ways to inherit: merge (blend weights), graft (splice whole parts, a chimera), and distill (learn by imitating answers). This series maps all three so a beginner can follow every step.
Every week the open-source world ships powerful new models. Yet most were not trained from scratch — they recombine abilities "inherited" from models that are already good. There are three ways to inherit: merge (blend weights), graft (splice whole parts, a chimera), and distill (learn by imitating answers). This series maps all three so a beginner can follow every step.
Why inherit instead of learning from scratch?#
Training a giant model from zero is like raising a newborn and teaching it language and every field from nothing — it costs enormous data and compute. But the world already has open-weight "experts": a math expert, a Korean expert, a coding expert. There's no reason to re-raise them. Moving learned ability is far cheaper. That's why GPU-poor individuals built leaderboard-topping models this way, and it became a real engineering field.
All three share one trait: they recombine others' ability without bottom-up learning. The upside is cheap and fast; the downside is they cannot invent ability that was in no source — you can't cook a flavor that isn't in the ingredients.
So where exactly do the three differ?#
The key split is one thing: where they operate. Hold this and the whole series gets easy.
Merge and graft live in "weight space." A model's skill lives in billions of numbers (parameters/weights). Merge directly averages/interpolates those numbers; graft splices whole layer blocks. Both touch the numbers directly, so the two models' tensor shapes must match — usually only same-family models (e.g., both Llama-derived).
Distillation lives in "behavior space." You query a teacher, take only its answers, and the student imitates them. Since you never touch the numbers, the internals can differ completely — even a closed cloud teacher (GPT, Claude) works (you only need its outputs).
A natural question: "If they're the same family, can I just average any two?" No — there's a trap, the "coordinate system / dialect" story of Post 2. For now, hold the big map.
| Route | One-line analogy | Operates in | Gradient training | Cross-family | Cloud teacher |
|---|---|---|---|---|---|
| Merge | Average two chefs' hands | Weights | None (combine/search) | No | No |
| Graft (chimera) | Transplant an organ | Weights | None (splice) | No | No |
| Distill | Apprentice watching a master | Behavior (answers) | Required (train student) | Yes | Yes |
So when do you pick which?#
Rule of thumb: same-family, different flavors (math + Korean) → merge is cheapest and smoothest. Different family, or a cloud teacher → distillation is the only road. Want a bigger model → stack layers (graft/chimera), but the seam is rough, so "healing" (Post 5) follows.
Latest trends (2026)#
Three threads dominate the past year. (1) Evolutionary merging went mainstream — Sakana AI showed how to auto-optimize merge "recipes" with an evolutionary algorithm (Post 3), now open-sourced as mergekit-evolve. (2) A distillation wave — in 2025, training small models on frontier reasoning outputs (DeepSeek-R1-Distill) exploded (Post 4). (3) Bridging tokenizers — distillation used to be hard across different tokenizers; new methods make it work anyway. In practice all three are layered together.
One-line: blend (merge), splice (chimera), imitate (distill) — a triangle. The split is "touch weights vs imitate behavior." The next posts dig into the real mechanics of each.
Note: Compiled 2026-07-01 from public research and repos. Cited numbers/cases are as reported in the papers/model cards; verify current terms in the originals.
Responses
No responses yet. Be the first to respond.