Darwinian (Evolutionary) Model Merging: Recipe as Genome

The "Darwin" route is an evolutionary algorithm that treats a merge recipe as a genome and runs it across generations. Sakana AI's Evolutionary Model Merge is the flagship: a computer converges on blend ratios humans could never find by hand. This post unpacks, without skipping steps, what "running generations" really means — and the crux question, "who scores it so it improves on its own?"

What does "recipe = genome" mean?#

Clear up one misconception first: what evolves is not the model but the numeric settings for "how to mix" — the recipe. A recipe is, say: "layer-1 FFN = 70% model A + 30% B; layer-2 = 40% A + 60% C; graft at layers 5, 11…" — a bundle of numbers. That bundle is the gene. Tweak it and a different child (merged model) is born.

The analogy to livestock breeding is exact. Breeding picks the best milk cows, mates them, picks the best calves, mates again, over many generations, to build a dairy breed. Model breeding is the same — pick the best-scoring recipes, mix them, mix the best again, to get a model specialized for the target.

What exactly happens in one generation?#

A "generation" is one lap of four steps. Slowly:

① Make several children. Whip up 20-100 merged models with slightly different recipes. Merging is number-combining, not training, so it takes minutes — you can mass-produce children.

② Test (fitness). Score each child on a target benchmark (Korean math, GPQA, etc.). A=52, B=61, C=48…

③ Select (survival of the fittest). Keep only the high scorers. B(61), D(59) survive; the rest die.

④ Recombine + mutate. Mix survivors (B×D) and jitter them slightly to make the next generation.

Repeat these four dozens to hundreds of times. Bad recipes vanish, good combos survive and mix, and an optimal blend humans couldn't find converges on its own. The optimizer is usually CMA-ES, which you can picture as "nudging the search distribution toward where good recipes cluster."

Who scores it — and why is that the crux?#

Address the common misconception: "it scores automatically" does not mean the AI judges itself. The benchmark already has ground truth, and a grader program mechanically checks against it. Humans only pick "which test" at the start.

Here's the trap. If fitness is the benchmark score, evolution honestly optimizes toward "doing well on that specific test" — test-specialization (gaming), not genuine capability. Tell a student "just ace this test" and they learn test tricks. Especially with a fixed or leaked benchmark, evolution finds the gaps uncannily.

How do you block that trap?#

Defense is entirely in the scoring design. First, held-out — re-check at the end on a test not used in evolution. Second, diverse benchmarks — measure several abilities, not one. Third, generalization metrics — like MergeBench, also score "does it do the un-trained things (forgetting/transfer)?" Sakana guarded with a held-out set, and a 7B EvoLLM-JP beating some 70B models on a broad Japanese suite showed real generalization. So evolutionary merging is powerful, but neglect "what you score on" and it quietly optimizes only the test.

Sakana's method also has two search spaces. PS (parameter space) evolves per-layer mix ratios and sparsity, actually blending weights; DFS (data-flow space) keeps weights intact and evolves only the path tokens take through layers. Combining both worked best.

Latest trends (2026)#

The biggest shift is that evolutionary merging went mainstream. Sakana didn't release the search algorithm, but the community re-implemented it as mergekit-evolve, so anyone can evolve their own recipe. Meanwhile, work on gaming/contamination surged — MergeBench (adds a generalization axis), ArenaBencher (evolves the benchmark to dodge memorization), ConStat (measures merge-leaderboard contamination). The stronger merging/evolution gets, the more "what you score on" separates real ability from illusion.

One-line: "running generations" = evolving a merge recipe via generate→score→select→recombine, hundreds of times. Powerful like breeding, but if the score is a benchmark it evolves to game that benchmark — so held-out/generalization checks are essential.

Note: As of 2026-07-01. Cited scores are as reported by the Sakana paper (measured); re-verify table figures in the published version.

References#

• Evolutionary Optimization of Model Merging Recipes
• Sakana AI blog — Evolutionary Model Merge
• EvoLLM-JP-A-v1-7B (model card)
• mergekit-evolve docs
• What Matters for Model Merging at Scale?