How ComfyUI Generates Images and Video, Explained
In short: ComfyUI is an execution engine that assembles image and video generation as a "node graph" (Lego blocks). Instead of one monolithic model drawing everything, you wire nodes (load checkpoint, encode prompt, sample, VAE-decode) into a data flow.
ComfyUI is an execution engine that assembles image and video generation as a "node graph" (Lego blocks). Instead of one monolithic model drawing everything, you wire nodes (load checkpoint, encode prompt, sample, VAE-decode) into a data flow. Its key strength is caching that re-runs only changed nodes: edit just the prompt and the earlier stages are reused from cache while only what follows re-executes. We run this engine as a pool, currently with 22 runnable (api) workflows and a queue history of 113 succeeded, 12 failed, and 21 cancelled, a roughly 90% success rate excluding cancellations (measured on our pool).
In one line: ComfyUI is a read-only execution engine that assembles image and video generation as a node graph and makes iteration fast by re-running only the nodes you changed.
In plain terms: ComfyUI is a recipe graph. Each step (prep, season, bake) is a node, and changing only the seasoning skips the prep and re-runs just what comes after, which makes iteration fast.
What nodes does one image pass through?#
Load, encode, sample, decode are the backbone. (1) A checkpoint loader brings up the model, CLIP, and VAE. (2) CLIPTextEncode turns positive and negative prompts into embeddings. (3) KSampler starts from noise and denoises for a set number of steps (this is where seed, steps, cfg, sampler, and scheduler are set). (4) VAEDecode turns the latent into actual pixels. In our pool's real parameters, turbo/lightning workflows run fast at steps 8, cfg 1, while full-quality ones (HiDream, Qwen-Image) run slow but precise at steps 40-50, cfg 3-5. Samplers include euler, dpmpp_2m, and uni_pc; schedulers include simple and karras.
The diagram below is that four-stage data flow: the loader on the left feeds the encoder above and the sampler below, and the decoder finally emits pixels.
| Node | What it does | Our pool's real values |
|---|---|---|
| Checkpoint loader | Load model, CLIP, VAE | SDXL, SD3.5, Qwen, HiDream |
| CLIPTextEncode | Prompt to embedding (pos/neg) | Positive + negative |
| KSampler | Noise to denoised generation | steps 8-50, cfg 1-5 |
| VAEDecode | Latent to pixels | Per-model dedicated VAE |
| SaveImage (output) | Save and return result | ~90% queue success |
Why is the caching so fast?#
The graph remembers each node's result by an input hash. If a node's inputs are unchanged, so is its output, so it is not recomputed and the cache just flows through. That is why editing only the prompt (the encode node) leaves the loader untouched and re-runs only from the encoder onward. In the figure, the cached nodes (dashed) are passed through, while re-execution starts at the changed node (bold border) and cascades downstream.
What is different for video?#
The same node paradigm gains "time". Text-to-video is still a graph, but the empty latent is a stack of frames rather than one image, and the model is video-specific. Two leaders stand out. WAN 2.2 uses a split-noise architecture with separate High-Noise and Low-Noise models in series, splitting coarse and fine denoising (flow matching, accelerated with 4-step lightning LoRAs). LTX-2 is a 19B DiT that generates low-resolution then 2x-upscales in latent space, follows an 8n+1 frame-count rule (e.g. 97, 121), and even emits audio in one pass. Our pool also hosts ltx23-t2v (text-to-video) and a wan22 image-to-video workflow.
Why WAN 2.2's split-noise takes two stages is clearer as a picture: the first model sets the coarse shape (high noise) and the second refines the detail (low noise).
Why build it as a graph (and why async)?#
For reproducibility, reuse, and extensibility. The graph is saved as JSON, so the same seed and nodes give the same result (reproducible), and you swap nodes to assemble ControlNet, LoRA, and upscaling. Generation takes seconds to tens of seconds, so it is received submit-then-poll async: queue the job and poll by job ID. We run several 96GB-class workstation GPUs as workers, one concurrent job per worker, keeping a minimum free VRAM headroom to avoid OOM (measured on our pool).
How do you try it yourself?#
Small, node by node.
- Build a basic t2i graph (load, encode, KSampler, VAEDecode) first and vary only steps and cfg to see the speed/quality curve.
- Queue the same graph twice and confirm the second finishes instantly from cache.
- For video, start with a small (short) frame count, and receive async jobs by polling after submission.
Reference links
- ComfyUI (node graph engine)
- ComfyUI execution engine (code)
- WAN video (model)
- LTX-Video (model)
- Latent Diffusion (paper)
Note: workflow counts, success rate, steps, and VRAM come from 2026 read-only status checks of our pool and public materials, and vary by model and config (internal addresses, paths, and accounts are private). Measure exact speed and quality on your own graph with the method above. Models and nodes update often, so this is reviewed quarterly.
Responses
No responses yet. Be the first to respond.