What if AI didn't write one letter at a time? Google's block-at-once DiffusionGemma

Google's DiffusionGemma is an open AI that, instead of writing one letter at a time from left to right, takes a whole block of up to 256 tokens and refines it over several passes until it snaps into focus all at once. On a suitable dedicated GPU, and especially when you run alone on a local machine, it can be up to 4x faster than a normal model. The trade-off is that it is still a speed-oriented experimental model, weaker than standard models when you need top-tier quality.

In one line: a Gemma-family open model that generates text by diffusion (developing a whole block at once) instead of autoregression (one letter at a time). It shines locally at low concurrency, trading peak quality for speed.

What does it mean for a diffusion model to write text?#

Start with an analogy. Almost every conversational AI today (ChatGPT, Claude, and the standard Gemma included) is autoregressive. Picture a typewriter: it types one character (really one token, a word fragment) at a time, looks at what it just typed, decides the next one, then the next, moving one at a time, forward only. Once a character is typed, it cannot be undone.

DiffusionGemma borrows the diffusion method that image generators use and applies it to text. Think of developing a photo. A diffusion image AI first fills the frame with noise (random speckle), then removes that noise a little at a time over several passes (denoising) until a clear picture "develops." DiffusionGemma does the same, but instead of a picture it works on a 256-token "canvas" filled with random placeholders, refining the whole canvas over several passes until the sentence becomes clear all at once.

Clear up a common misconception here: "looking at a block at once" does not mean it finishes in a single computation. It refines the canvas many times (often dozens of passes). What is different is that each pass touches all 256 tokens at once, which uses the GPU far more efficiently than looping one character at a time.

Why is diffusion faster locally?#

The answer up front: local AI is slow not because of raw math but because of memory bandwidth, and diffusion flips exactly that bottleneck. That single sentence is most of this article. Let us unpack it.

When a normal model produces one character, the GPU must load the entire model weights from memory once just to compute that single character. The math itself is instant, but hauling tens of gigabytes of weights from memory to the compute cores takes time. This is the memory-bandwidth bottleneck. Because that heavy load is repeated for every character, the compute cores mostly sit idle, waiting for data.

The cloud solves this with batching. It groups many users' requests into one batch, so that after loading the weights once it computes dozens of users' work at the same time, keeping the GPU busy. But when you run alone locally, there is only one request (yours) to batch, so the trick does not apply.

DiffusionGemma processes 256 tokens in parallel within a single request, even with just one user. Having loaded the weights once, it fills the idle compute cores with 256 tokens of work, so the GPU stays busy even solo. That is how a single H100 reaches around 1,000 tokens per second. Note that this advantage is largest at low concurrency and locally, and shrinks in large-scale cloud serving that already packs the GPU by batching many requests.

Where is it strong, and where is it weak?#

DiffusionGemma is a free open model (Apache 2.0) built on Google's Gemma 4 26B (a mixture-of-experts, MoE, design). Quantized, it fits on an 18GB-VRAM consumer GPU, so a single card like an RTX 4090 is enough for an individual to try it locally.

Its strength comes from seeing and fixing the whole canvas in both directions. It is strong on tightly constrained problems where earlier and later slots interlock, such as Sudoku (about 80% success after fine-tuning). A normal model goes strictly left to right, so it cannot see future slots and cannot undo a slot it already filled. A diffusion model reworks the whole thing on every pass, so it can self-correct.

The weaknesses are just as clear. It does not write "everything at once" forever. For long text it fills a 256-token block, then moves to the next block sequentially, so blocks are still processed in order. And because it is still a speed-oriented experiment, Google itself recommends standard Gemma 4 when you need top quality. Local acceleration is supported by NVIDIA (RTX and DGX), and you can also fetch and run it via llama.cpp or GGUF.

What should you take away right now?#

It is a signal that the old assumption "AI must type one letter at a time" is breaking. For local, interactive work where speed matters, "parallel block generation" has arrived as a real candidate. But it is not a silver bullet today; it is a tool you use once you understand the speed-vs-quality trade-off. If local response speed has felt sluggish, it is worth an experiment; if final quality is the top priority, standard Gemma 4 is still the safe default.

Note: diffusion language models are a fast-moving, early field as of 2026. Gemma-family versions, quantization options, and throughput numbers are revised often, so re-check the latest values in Google's official developer guide and model card before adopting. This is a shift Hax will keep watching.