Running AI Agents Locally: What Decides Speed and Quality, Measured

When you run AI agents locally, speed and quality are decided not by the model alone but by function-calling reliability and the execution framework. By BFCL (the Berkeley Function Calling Leaderboard), 1-3B models are enough for simple single tool calls but collapse on multi-turn, parallel, and nested calls, while fine-tuned 7-20B models catch the closed ones (ToolACE-8B surpasses GPT-4 and Claude 3.5 on overall BFCL). For frameworks, LangGraph fits production and smolagents fits rapid prototyping.

In one line: the real bottleneck for a local agent is not "how smart" but whether it calls tools correctly. If function calling breaks, even a huge model cannot act.

What breaks a local agent?#

Function-calling (tool-calling) reliability. An agent loops through think to tool-call to observe, and if the model gets the tool name or JSON arguments wrong, the loop stalls. Drawn as a diagram, it is clear that a single broken box (the tool call) halts the whole thing.

BFCL scores accuracy via AST (syntax-tree) matching without executing functions, covering serial, parallel, multi-turn, and hallucination. The findings are clear: sub-1B models fail reliably on the harder shapes (multi-turn, parallel, nested), and 1-3B is the sweet spot for simple tool use on the edge. The capability floor drops every six months: tasks that needed 70B in early 2025 work at 32B in early 2026.

Drawn as steps, the model size needed for the same task falls by nearly half every six months.

How do the frameworks differ?#

By how they emit actions. LangGraph wires the flow as a graph, fitting production needs like audit and rollback; CrewAI is role-based, so you build fast without graph theory. AutoGen is strong at multi-agent conversation, and smolagents writes and executes Python code instead of selecting predefined tools, cutting reasoning/acting steps versus a classic ReAct loop. In local-model (Ollama) tests, LangGraph won on complex multi-step work and smolagents on rapid prototyping.

Local or cloud, which do you pick?#

It splits by use case and data. For privacy, offline, and cost, go local; for the hardest reasoning and long multi-hop chains, the cloud frontier still leads. But the gap is closing fast: fine-tuned 7-20B models passed closed ones on overall BFCL, and Qwen3-4B scores MMLU-Redux 83.7, beating models twice its size. So simple, repetitive agents run fine locally, and a hybrid that escalates only hard tasks to the cloud is the practical default.

How do you measure it yourself?#

Measure on your own tools.

• Use 5-10 of your real tools to measure function-call success (name and argument accuracy) and per-step latency per model.
• Give the same task to LangGraph and smolagents and compare completion rate and average step count.
• Use a small model for simple single calls and one tier up for multi-turn/parallel (BFCL's harder shapes are the guide).

Reference links

• Gorilla / BFCL (function-calling leaderboard)
• LangGraph (repo)
• smolagents (repo)
• CrewAI (repo)
• ReAct (paper)

Note: BFCL is solid (official leaderboard and an ICML 2025 paper), but framework comparison figures come from blog benchmarks and are directional. Measure exact values on your own tools and tasks with the method above. Models and frameworks change fast, so this is reviewed quarterly.