“Free, local, no API bill” is a seductive pitch. Point your favorite coding agent at Ollama, load up Qwen, and suddenly you’ve got an AI pair programmer that never sends a token outside your machine. No usage-based pricing, no data leaving your laptop, no dependency on a vendor’s uptime. I wanted this to work badly enough that I spent a weekend on it. It mostly didn’t, and the reason why is more interesting than the failure itself.
Why bother in the first place
The pitch for local models isn’t just “free.” For a lot of teams it’s three things at once: cost predictability (a flat electricity bill instead of a variable API bill that scales with how much the agent reads and rewrites your codebase), data locality (nothing about your source ever leaves the machine, which matters more the moment a client contract has a data-residency clause), and plain curiosity, open-weight models have gotten good enough that “can this replace a hosted model for day-to-day coding” is a fair question to ask, not a joke.
So the experiment: swap Claude Code’s backend for a self-hosted Qwen model served through Ollama, and see how far I could get before something broke.
The setup isn’t the hard part
Getting Claude Code to talk to a local model is trivial on paper: point the base URL at Ollama’s OpenAI-compatible endpoint, pull a Qwen checkpoint, and go. Ollama serves the model, the agent sends requests, tokens come back. Twenty minutes, done. If all you wanted was a chat window that answers coding questions, you’d stop here and call it a win.
Except Claude Code was never meant to be a chat window. And the moment the agent tries to do something, read a file, run a shell command, apply a diff, the whole thing falls apart.
The real bottleneck: tool-call formatting
Claude Code is an agent loop. The model doesn’t just answer a question; it decides which tool to call, emits a structured, machine-parseable call for that tool, waits for the result, and decides what to do next, repeatedly, without a human checking each step. That loop only survives if the model’s tool calls are syntactically correct every single time. Anthropic’s hosted models are trained hard on this exact behavior; it isn’t a prompt trick, it’s a property of the weights themselves.
Qwen via Ollama gets close. But “close” is the wrong bar for an autonomous loop, because there’s no human in it to catch a call that went sideways. A few failure modes I hit, repeatedly and reproducibly:
- Malformed JSON in tool arguments. A trailing comma, an unescaped quote inside a file path, a string that isn’t properly closed, the kind of thing a human reading the output would mentally auto-correct without noticing, but that a strict JSON parser rejects outright, killing the turn.
- Tool calls wrapped in prose. Instead of emitting a clean structured call, the model would narrate its intention in natural language around the JSON block, explaining what it was about to do, and the agent’s parser has no reliable way to strip that framing out and extract just the call.
- Schema drift under long context. This was the most interesting failure. Formatting held up fine for the first handful of tool calls in a session. As the context filled up with prior tool results and file contents, the model’s adherence to the exact schema started to slip, not randomly, but predictably worse the longer the session ran.
None of these are Ollama’s fault, it’s just a serving layer, faithfully passing through whatever the model produces. And they’re not really a knock on Qwen either. This is what smaller open-weight models generally do: they approximate a target format well, but not perfectly, and most of them aren’t specifically trained with a heavy penalty for that kind of imperfection. An agent loop has zero tolerance for “approximate.” One malformed call and the chain stalls, and unlike a human pair programmer, the model can’t just notice its own typo and fix it mid-sentence, the parser has already thrown by the time anyone could correct it.
What would actually fix this
There are three honest paths forward, and none of them are a config flag:
- Fine-tune on the exact tool schema. Train the model specifically on the tool-call format your agent expects, ideally using real transcripts from your own agent loop. This is the “do it properly” option, and also the most expensive in time and compute.
- Add a repair layer. Sit a strict validator between the model’s raw output and the agent’s parser, catch malformed JSON, re-prompt the model with the specific parse error, or pattern-match and patch common failure shapes (trailing commas, prose-wrapped calls) before they ever reach the parser.
- Scope down the agent’s autonomy. Accept a smaller feature set, read-only exploration, single-step suggestions, no unsupervised multi-step edits, where an occasional formatting slip is recoverable instead of fatal to the whole session.
I didn’t build the repair layer this time around. It’s the right next step if I revisit this project, and it’s a legitimate piece of engineering, not something you bolt on in an afternoon.
The takeaway
“Run your coding agent locally” and “run a local model” sound like the same claim, but they’re not. The gap between them is tool-call reliability, not raw model quality, Qwen is a genuinely capable model at writing and reasoning about code. It just wasn’t trained to be the deterministic half of an unsupervised agent loop the way Claude was. That training, not parameter count and not benchmark scores, turned out to be the whole ballgame.
If you’re evaluating local models for agentic coding work, don’t just ask “how good is this model.” Ask “how good is this model at never breaking format,” because in an agent loop, that second question is the one that decides whether you end up with a tool or just a demo.
You may also like : Context Engineering vs Prompt Engineering: What’s the Real Difference
