Chimera & Frankenmerge: Splicing Models by the Part

A chimera does not "blend" parents — it "splices" parts. Like the lion-head, goat-body, snake-tail monster of myth, each part is kept original and grafted whole. In model land this is frankenmerge / passthrough merging: stack whole layers from one or more models. This post builds up, from scratch, why you'd do this and why the price is a rough "seam."

Why splice instead of average?#

Weight-averaging blends same-shape models into a same-size result — size unchanged. Splicing is different: stacking whole layers grows the parameter count itself. Examples: goliath-120b (two Llama-2-70B interleaved in 16-layer blocks → 118B) and Upstage SOLAR-10.7B (32 layers up-scaled to 48, then retrained). Making "a bigger model you could never get by averaging" is the whole point. Transplanting only a specific block (like a knowledge-heavy FFN) into another body is the same "organ graft" idea.

But why does splicing get rough?#

This is the heart of the post. To get it, you first need how attention and FFN talk inside a transformer.

Bottom line: they are not wired to each other. They communicate only through a shared bulletin board — the residual stream. Down this conveyor-belt document, attention reads → adds a context memo, then FFN reads → adds a knowledge-processed result. Attention is a "meeting" that moves info between positions; FFN is "per-position processing" that looks up knowledge. Both only read and write the same document; no direct link.

The problem is that document's "coordinate system." What each dimension means is a private dialect each model learned. Gemma writes in its dialect ("topic features go in the X direction"); another model uses a different dialect ("topic features in Y").

Now graft a foreign FFN into Gemma's body. The width matches, so it physically fits — no crash. But the FFN tries to read Gemma's memo in its own dialect. Same document, wrong dictionary → it writes nonsense, and the next layer reads that polluted document, spreading the mess downward. That mismatch is the "seam." The attention line is alive and well, but the attention↔FFN handshake speaks two languages, so the conversation breaks.

So how do you stitch the seam?#

Luckily, each junction has a small translator — LayerNorm. A short post-graft retraining mostly re-tunes this normalization and nearby scales so the dialects connect — that's "healing" (Post 5). SOLAR even trimmed some middle layers to shrink the mismatch, then recovered with continued pretraining. The lesson: a chimera is not "splice and done" but splice then rehab.

Latest trends (2026)#

After goliath, hobbyists self-merged Llama-3-70B up to 122B/225B (e.g., Llama-3-120B). MoE frankenmerge (mergekit-moe) also spread — keep the base's attention, place several models' FFNs side by side as experts, cutting the seam burden. The community lesson: "bigger lifts benchmarks, but real-use coherence needs healing." So focus is shifting from the splice to post-splice rehab (healing).

One-line: a chimera splices parts instead of blending parents. You can grow size or move specific parts, but attention↔FFN read the shared document in different dialects, so the seam is rough — LayerNorm re-tuning + finetuning "rehab" always follows.

Note: As of 2026-07-01. Many frankenmerges ship without retraining and can ramble despite good benchmarks (ties to the traps in Posts 3 and 5).

References#

• mergekit — passthrough/Frankenmerging
• SOLAR-10.7B: Depth Up-Scaling
• goliath-120b (model card)
• Merge LLMs with mergekit (Labonne)
• Transformer Circuits — residual stream