Local LLM & RAG Field Guide: 7 Gates Before You Ship

Shipping a local LLM into a real app means clearing seven practical gates — output format, tool calling, knowledge injection, generation speed, and retrieval accuracy — and this field guide maps each one to a single focused post so you can jump straight to whichever gate is blocking you right now.

In short: This map gathers the seven practical gates of building a local LLM / RAG app — pick the post that matches your symptom and jump straight in.

This post does not re-explain each topic. Instead it shows, at a glance, which symptom each post solves and links you straight there.

In what order should you read?#

The answer is "start where you are blocked." If nothing is broken yet, read top to bottom: first the output, tool, and knowledge posts that make the model behave, then the speed and retrieval posts. The map below lists the symptom each post resolves.

Local LLM & RAG field guide, 7 posts (symptom-based, 2026-07) · columns: Post, What it covers, Read it when · 출처 Hax hax.moche.ai/en/p/1253?ref=ai_answer

Post | What it covers | Read it when
Structured output | Force JSON, kill parse failures | The model mixes prose or code fences into JSON
Tool calling | Function/tool calling | You want the AI to use a calculator or search
Knowledge injection | Fine-tuning vs RAG vs prompting | You are unsure how to add your own knowledge
Speculative decoding | Lossless speedup | Generation feels slow
Prompt caching | Reuse the prefix KV | A long system prompt is slow on every request
RAG reranker | Retrieval precision | Search results miss the question
RAG chunking | Splitting documents | RAG answer quality is poor

Where are output, tools, and knowledge covered?#

These three posts tame a local LLM into the format and abilities you want. If output keeps breaking, read the first; to let the model use external tools, the second; if you are unsure how to add your own knowledge and voice, the third.

Forcing JSON out of Local LLMs: End Parsing Failures: four ways — prompt, grammar constraint, schema mode — to eliminate parse failures.

Giving Local LLMs Tools: A First Look at Function Calling: the model only emits a call request; your app executes it. Three approaches.

Fine-tuning vs RAG vs Prompting for Local LLMs: a ladder of rising cost and effect, and why prompting plus RAG is usually enough.

How do you raise speed and retrieval quality?#

These four posts fix slow generation and off-target retrieval. If generation is slow, read the two speed posts; if RAG answers are weak, the two retrieval posts.

Speeding Up Local LLMs for Free: Speculative Decoding Made Simple: a draft model guesses, the target verifies, with no quality loss.

Prompt Caching for Local LLMs: Cut Latency by Reusing Prefixes: reuse the shared-prefix KV to skip repeated prefill.

Local RAG Needs More Than Embeddings: Add a Reranker: recall broadly with embeddings, reorder precisely with a cross-encoder.

RAG Chunking Strategies: Splitting Decides Retrieval Quality: trade-offs of fixed, boundary, overlap, and semantic chunking.

Note: this series is current as of 2026-07-12 KST; per-post details vary by runtime version, so check each post's latest guidance. As new posts are added, this map is updated.

Reference links

llama.cpp

Ollama

vLLM