Finetuning vs Healing: Stitching the Merge Seam

Healing is a light retraining right after a merge/graft to stitch the broken seam. In Post 2 we said "a graft needs rehab" — that rehab's (community) name is healing. The key is its purpose: healing does not inject new knowledge; it restores the coherence/fluency the graft broke. This post lays out what healing fixes, how it differs from finetuning, and when it's essential.

What exactly does healing fix?#

Quick recap of Post 2. In a frankenmerge/FFN graft, attention writes memos to the shared document (residual stream) in its dialect, and the grafted FFN reads them in a different dialect, writing nonsense — the seam. Healing re-tunes that attention↔FFN handshake. A little retraining re-aligns LayerNorm (the junction translator) and nearby scales so the dialects connect. After healing, the grafted part understands the body's document "in its own dictionary."

Healing vs finetuning vs from-scratch?#

They sound alike but have opposite goals. Let's separate them cleanly.

Healing uses a little general corpus to make the seam "speak smoothly again" — the goal is recovered fluency, not new ability. Domain finetuning injects new skill/knowledge with target-domain data. From-scratch pretraining builds "how to speak" from zero on massive data. Costs are opposite too — healing starts from a near-working merged checkpoint (very cheap); from-scratch is enormous.

How is healing actually done?#

The recipe is surprisingly simple: run a short continued-pretraining (or LoRA) on the merged checkpoint with a little general text — general, not domain data. SOLAR-10.7B's curve is emblematic — right after up-scaling 32→48 layers, it dipped below the base, then recovered fast during retraining and finally surpassed the base. It shows "the splice isn't the end; the post-splice rehab is the real work."

When is it needed — and not?#

It splits cleanly. Chimera (frankenmerge, FFN graft) has rough seams, so healing is essential. Same-lineage weight averaging (SLERP, TIES, DARE) shares an ancestor, so coordinates already line up and fluency rarely breaks — little healing needed. The rule running through this whole series: blending needs less healing; splicing needs it most. That's exactly why same-lineage averaging is so appealing in LLM merging.

Caution — healing is not magic#

Two traps. First, if the graft is fundamentally bad, healing can't save it — too-different dialects don't connect, and you just retrain a broken skeleton. Second, many big frankenmerges chase benchmark scores without enough healing: great on leaderboards, rambling in real use — exactly the "benchmark gaming" trap of Post 3. Plausible-sounding but incoherent.

Latest trends (2026)#

As merging enters production, healing is being automated: pipelines bundle "merge → short heal → eval," and lightly mix healing data toward the target personality. And a new yardstick emerged — "how much did you heal" became a trust metric for merged models, because post-splice rehab decides quality more than the splice itself.

One-line: healing re-aligns the graft-broken attention↔FFN handshake via light retraining to restore fluency. Its purpose differs from finetuning or from-scratch; it's essential for chimeras and rarely needed for same-lineage averages. And a graft that healing can't fix simply can't be fixed.

Note: As of 2026-07-01. "Healing" is community vocabulary, not a formal term; the underlying technique (post-graft continued pretraining) is rigorously shown by the SOLAR/DUS paper.

References#

• SOLAR-10.7B: Depth Up-Scaling
• mergekit
• Merge LLMs with mergekit (Labonne)
• Model merging community writeup
• ConStat — merge leaderboard contamination