A Korean small model from scratch — moche-native's honest G1 baseline

moche-native is a from-scratch Korean+English small omni-model project that designs and trains from the tokenizer's first merge rule up instead of fine-tuning a finished model, and its first baseline G1 (0.49B, zero tricks) reaches val bpb 0.910 while lifting Korean tokenizer efficiency to 5.0x versus GPT-2.

In short: the model is still a baseline; the tokenizer is already a settled win (Korean 5.0x) — this is a public build log of the from-scratch process, not a leaderboard rank.

Why build from scratch instead of using a finished model?#

It comes down to ownership and Korean efficiency. Fine-tuning a large finished model is fast, but you inherit someone else's tokenizer, vocabulary, and data rules. moche-native took the opposite path — a balanced 64k Korean+English BPE tokenizer that lifts Korean fertility (how many tokens a sentence is split into) to 5.0x versus GPT-2. Fewer splits means the same context fits more cheaply and further, so this is a settled asset regardless of model quality. The philosophy is explicit too: real-time interaction (chat, speech, vision) lives in the weights (the body), while high-quality image/video generation is called out to external tools (the workshop). The rule is 'put in every interaction, not every feature.'

Think of it as building a house from the foundation up instead of renting one and changing the wallpaper — the tokenizer is that foundation, and it is already laid straight.

How far along is the G1 baseline?#

G1 is a well-tuned vanilla: 0.49B (depth 12), our 64k tokenizer, English (ClimbMix) + Korean (fineweb-edu), bf16, the Muon optimizer, a WSD schedule, and zero tricks. Validation loss descended smoothly to val bpb 0.910 (min 0.899), with zero restarts and zero loss spikes confirming no 'silent bugs.' Late annealing let it pick up Korean facts like 'Sejong created Hangul' and 'the capital is Seoul.' The English benchmark (CORE) score is low at ~0.13, but that is structurally expected with a 2x vocabulary, and Korean fluency is early at 5.0/10. What matters is not the rank but that G1 is the origin point for every later architecture and optimizer comparison — tricks are promoted only if they hold a size trend (0.1 -> 0.3 -> 0.9B) on a proxy.

What should you watch out for? (honest scope)#

This is process transparency, not a competitive benchmark boast. G1 is only the first baseline, and its absolute scores (CORE 0.13, KO 5.0/10) are not numbers to compare against finished models — the repo states plainly that competitive ranks come when the model is done. The one settled win is the tokenizer (Korean 5.0x); the model still has G2 promotion trials (trick auditions), G3 main run (1B), and G4 sensory integration (speech, vision) ahead. The stack is reproducible too — a fork of Karpathy's nanochat, public data (ClimbMix, fineweb-2-edu-korean, HPLT 3.0), self-hosted. In short, the claim is not 'our model is the strongest' but rather: here is the real process of building a Korean small model from scratch, with the numbers.

Note: the figures above are moche-ai/moche-core's own training measurements (as of 2026-07), not competitive benchmark ranks. It is early stage, so numbers update as the gates progress.

Reference links

• moche-core (moche-native) public build log
• Tokenizer fertility measurement (FERTILITY.md)
• G1 baseline model (Hugging Face)
• Korean+English 64k tokenizer (Hugging Face)