How Models Inherit Skills: Merge, Chimera, or Distill

Every week the open-source world ships powerful new models. Yet most were not trained from scratch — they recombine abilities "inherited" from models that are already good. There are three ways to inherit: merge (blend weights), graft (splice whole parts, a chimera), and distill (learn by imitating answers). This series maps all three so a beginner can follow every step.

Why inherit instead of learning from scratch?#

Training a giant model from zero is like raising a newborn and teaching it language and every field from nothing — it costs enormous data and compute. But the world already has open-weight "experts": a math expert, a Korean expert, a coding expert. There's no reason to re-raise them. Moving learned ability is far cheaper. That's why GPU-poor individuals built leaderboard-topping models this way, and it became a real engineering field.

All three share one trait: they recombine others' ability without bottom-up learning. The upside is cheap and fast; the downside is they cannot invent ability that was in no source — you can't cook a flavor that isn't in the ingredients.

So where exactly do the three differ?#

The key split is one thing: where they operate. Hold this and the whole series gets easy.

Merge and graft live in "weight space." A model's skill lives in billions of numbers (parameters/weights). Merge directly averages/interpolates those numbers; graft splices whole layer blocks. Both touch the numbers directly, so the two models' tensor shapes must match — usually only same-family models (e.g., both Llama-derived).

Distillation lives in "behavior space." You query a teacher, take only its answers, and the student imitates them. Since you never touch the numbers, the internals can differ completely — even a closed cloud teacher (GPT, Claude) works (you only need its outputs).

A natural question: "If they're the same family, can I just average any two?" No — there's a trap, the "coordinate system / dialect" story of Post 2. For now, hold the big map.

Three routes at a glance — this table is the series map · columns: Route, One-line analogy, Operates in, Gradient training, Cross-family, Cloud teacher · 출처 Hax hax.moche.ai/en/p/1058?ref=ai_answer

Route | One-line analogy | Operates in | Gradient training | Cross-family | Cloud teacher
Merge | Average two chefs' hands | Weights | None (combine/search) | No | No
Graft (chimera) | Transplant an organ | Weights | None (splice) | No | No
Distill | Apprentice watching a master | Behavior (answers) | Required (train student) | Yes | Yes

Contents

So when do you pick which?

Latest trends (2026)

References

So when do you pick which?#

Rule of thumb: same-family, different flavors (math + Korean) → merge is cheapest and smoothest. Different family, or a cloud teacher → distillation is the only road. Want a bigger model → stack layers (graft/chimera), but the seam is rough, so "healing" (Post 5) follows.

Latest trends (2026)#

Three threads dominate the past year. (1) Evolutionary merging went mainstream — Sakana AI showed how to auto-optimize merge "recipes" with an evolutionary algorithm (Post 3), now open-sourced as mergekit-evolve. (2) A distillation wave — in 2025, training small models on frontier reasoning outputs (DeepSeek-R1-Distill) exploded (Post 4). (3) Bridging tokenizers — distillation used to be hard across different tokenizers; new methods make it work anyway. In practice all three are layered together.

One-line: blend (merge), splice (chimera), imitate (distill) — a triangle. The split is "touch weights vs imitate behavior." The next posts dig into the real mechanics of each.

Note: Compiled 2026-07-01 from public research and repos. Cited numbers/cases are as reported in the papers/model cards; verify current terms in the originals.

References#

Model Soups (ICML 2022)

TIES-Merging (NeurIPS 2023)

DARE (ICML 2024)

mergekit

FuseLLM (ICLR 2024)

Deep Model Fusion: A Survey