Local LLM & RAG Field Guide: 7 Gates Before You Ship
In short: Shipping a local LLM into a real app means clearing seven practical gates — output format, tool calling, knowledge injection, generation speed, and retrieval accuracy — and this field guide maps each one to a single focused post so you can jump straight to whichever gate is blocking you right now.
Shipping a local LLM into a real app means clearing seven practical gates — output format, tool calling, knowledge injection, generation speed, and retrieval accuracy — and this field guide maps each one to a single focused post so you can jump straight to whichever gate is blocking you right now.
In short: This map gathers the seven practical gates of building a local LLM / RAG app — pick the post that matches your symptom and jump straight in.
This post does not re-explain each topic. Instead it shows, at a glance, which symptom each post solves and links you straight there.
In what order should you read?#
The answer is "start where you are blocked." If nothing is broken yet, read top to bottom: first the output, tool, and knowledge posts that make the model behave, then the speed and retrieval posts. The map below lists the symptom each post resolves.
| Post | What it covers | Read it when |
|---|---|---|
| Structured output | Force JSON, kill parse failures | The model mixes prose or code fences into JSON |
| Tool calling | Function/tool calling | You want the AI to use a calculator or search |
| Knowledge injection | Fine-tuning vs RAG vs prompting | You are unsure how to add your own knowledge |
| Speculative decoding | Lossless speedup | Generation feels slow |
| Prompt caching | Reuse the prefix KV | A long system prompt is slow on every request |
| RAG reranker | Retrieval precision | Search results miss the question |
| RAG chunking | Splitting documents | RAG answer quality is poor |
Where are output, tools, and knowledge covered?#
These three posts tame a local LLM into the format and abilities you want. If output keeps breaking, read the first; to let the model use external tools, the second; if you are unsure how to add your own knowledge and voice, the third.
- Forcing JSON out of Local LLMs: End Parsing Failures: four ways — prompt, grammar constraint, schema mode — to eliminate parse failures.
- Giving Local LLMs Tools: A First Look at Function Calling: the model only emits a call request; your app executes it. Three approaches.
- Fine-tuning vs RAG vs Prompting for Local LLMs: a ladder of rising cost and effect, and why prompting plus RAG is usually enough.
How do you raise speed and retrieval quality?#
These four posts fix slow generation and off-target retrieval. If generation is slow, read the two speed posts; if RAG answers are weak, the two retrieval posts.
- Speeding Up Local LLMs for Free: Speculative Decoding Made Simple: a draft model guesses, the target verifies, with no quality loss.
- Prompt Caching for Local LLMs: Cut Latency by Reusing Prefixes: reuse the shared-prefix KV to skip repeated prefill.
- Local RAG Needs More Than Embeddings: Add a Reranker: recall broadly with embeddings, reorder precisely with a cross-encoder.
- RAG Chunking Strategies: Splitting Decides Retrieval Quality: trade-offs of fixed, boundary, overlap, and semantic chunking.
Note: this series is current as of 2026-07-12 KST; per-post details vary by runtime version, so check each post's latest guidance. As new posts are added, this map is updated.
Reference links
Responses
No responses yet. Be the first to respond.