Which Open-Weight LLM Should You Pick for a Local AI Agent? (2026)

If you are running a tool-calling AI agent locally, the realistic 2026 shortlist narrows to five open-weight families — Qwen, Llama, Mistral, Gemma, and DeepSeek — and you pick along two axes: license and size. Qwen and Mistral ship permissive (Apache-2.0) weights that are safe for commercial agents, Llama and Gemma are highly capable but carry custom community licenses, and a 7-8B instruct model is usually enough for tool-calling work.

In one line: for commercial use start with Apache-2.0/MIT weights (Qwen, Mistral, DeepSeek), default to 7-8B, and only scale to 14-32B for genuinely complex reasoning.

First, unpack "open-weight." An open-weight model is one whose trained weight files — the learned pile of numbers — are published so anyone can download and run them on their own machine, unlike a closed model such as ChatGPT that you can only call through an API. But "weights are public" does not mean "use them anywhere." A model can be released openly and still restrict how you may use it through its license, which is why you must read the terms before shipping it in a commercial product. That is exactly why this guide sorts families by license.

What makes a model good for agents?#

Answering once as a chatbot and looping through tool calls as an agent demand different things. Agent skill really comes down to four traits: (1) dependable tool and function calling — does it correctly decide when to call which function with which arguments; (2) instruction following — does it stay within the rules and format of the system prompt; (3) valid JSON output — does it emit tool arguments as syntactically clean JSON; and (4) a large enough context window — long enough to hold your tool definitions and conversation history at once. It has to satisfy all four within a size your hardware can actually run.

A common misconception: people assume the model topping a leaderboard is automatically the best agent, but the winner on general-knowledge benchmarks can be surprisingly weak at tool calling and strict JSON formatting. So after picking a family, always test it against your own tool schema.

Here is where each family is strong, at a glance.

Which size do you actually need?#

Parameter count is the model's "brain size." Bigger is smarter but eats more memory (VRAM) and runs slower, so you don't just crank it up — you match size to task difficulty. For most agents that call tools and follow a fixed loop, a 7-8B instruct model is enough, and quantized to 4-bit it runs on a consumer GPU like an RTX 3060 12GB. Step up to 14-32B only for complex reasoning that must plan and backtrack across many steps, and reserve the largest 70B-class models for offline batch work where real-time response is not needed.

Why read the license first?#

Even when weights are public, whether you may use them commercially is set by the license — and this is where teams get burned. Apache-2.0 and MIT weights (most of Qwen, Mistral's open models, DeepSeek's open models) carry almost no conditions and drop straight into a commercial product. Meta's Llama Community License and Google's Gemma terms, by contrast, are permissive for most teams but come with strings — for example an acceptable-use policy you must honor, or special clauses for very large deployments. Don't wave it off as "open, so it's free"; read the model's original license text before launch.

How does quantization help?#

Quantization compresses weights into fewer bits so the same model fits in less memory. Squeeze a number that was stored in 16-bit (FP16) down to 4-bit and its storage roughly quarters. It is like saving a photo as a reasonably compressed JPG instead of the raw original — inspect closely and there is a difference, but for most uses you won't notice. In agents, 4-bit is the common default: little quality loss on most tasks while cutting VRAM to about a quarter versus 16-bit. That is what drops a 7-8B model from about 16GB at 16-bit to about 6GB at 4-bit, making it practical on a consumer GPU.

If you pick just one this week#

1. Shortlist a permissive 7-8B instruct model (for example an Apache-2.0 Qwen variant).
2. Grab the 4-bit quantized build and load it on a consumer GPU.
3. Test function calling and JSON output against your real tool schema to confirm it follows it reliably.
4. Scale to 14-32B only when evaluation demands it — don't start big.

Note: open-weight models ship on a fast cadence. Treat this as a map of families, not a leaderboard, and verify each model's current release, license, and benchmarks before committing. This map is reviewed quarterly.