Same Model, Four Harnesses: What Actually Changes When Only the Wrapper Changes

I ran the same coding task through four AI coding tools this week - opencode, Pi, GitHub Copilot in VS Code, and Claude Code - all driven by the same local model on a 64 GB M1 Max, reached over our internal VPN. Same prompt, verbatim, every time. Not "which tool is best" - with the model held constant, what does the wrapper actually change?

The setup

The task was a small, educational CLI RAG chatbot in Python: index a folder of documents, embed and chunk them, retrieve by similarity, and answer questions in a REPL, with pytest tests covering chunking and retrieval. Simple enough for a local model to plausibly finish in one sitting, real enough to expose actual differences in tool-calling, context handling, and autonomy. The inference box itself is an M1 Max running oMLX, serving an OpenAI-compatible API over our tailnet (a VPN-internal endpoint, not something reachable from outside).

The model across all four tools: Qwen3.6-35B-A3B, the one model in this stack that reliably drives an agent loop. That qualifier matters - I also tried Qwen2.5-Coder-32B, and despite the "coder" branding it could not tool-call at all on this oMLX/MLX-4bit setup. Whether that's the quantization, the server's function-call template, or the model itself, I can't say from one stack. I wouldn't generalize "Qwen2.5-Coder can't tool-call" beyond this exact combination - it's a data point about this deployment, not a verdict on the model.

Getting the model in: a friction ladder

Before any tool can drive the model, you have to point it there. That step alone told me something:

The Claude Code row turned out to be the simplest of the four, once I knew where to look: oMLX ships a native Anthropic /v1/messages endpoint, confirmed with a plain HTTP request. Point Claude Code at it with scoped environment variables on the launch command, and nothing persists globally:

1ANTHROPIC_BASE_URL=http://<oMLX-host>:8000 \
2ANTHROPIC_AUTH_TOKEN=<token> \
3ANTHROPIC_DEFAULT_SONNET_MODEL=Qwen3.6-35B-A3B-UD-MLX-4bit \
4claude

Normal, cloud-backed Claude Code - in the terminal and in VS Code - stays completely untouched.

How each tool actually drove the model

With the model wired in everywhere, the interesting part starts.

opencode ran the task through its TUI, and separately through its own desktop UI (Chat / Build / Plan mode, mirroring the CLI). It coped with a fairly small context window, needed a handful of nudges to point it at errors it hadn't caught itself, and produced a working app afterward.

Pi is pure CLI, the leanest of the four, and its context compacted around ~28k tokens without drama. The one real defect: the model repeatedly called Pi's write tool with only {content} and no path argument, and Pi's harness rejected every one of those calls with "Validation failed for tool write." It recovered after retries and the app ran at the end, but I watched the same failure repeat several times before the model course-corrected on its own.

GitHub Copilot in VS Code had the nicest UI of the four and was pleasant to follow turn by turn. It also got stuck in a genuine loop at a 32k context window - burning most of its budget on tool definitions before it had room to reason. Once I gave it a larger window and explicitly told it to run and test the app itself, it did exactly that, autonomously, and worked the problem through to a result. Of the four, it used its own CLI tooling the most reliably - it rarely fumbled a tool call. It was also the most config-heavy to set up in the first place.

Claude Code was the most autonomous by a wide margin. It ran its own Explore and Plan sub-agents, went through plan mode with the usual approval gate, built a task list, and then executed against it end to end: wrote all 12 files, ran pytest, hit a nasty bug (re.split returning None where the code assumed a list), and self-debugged through several fixes until it reached 24/24 passing tests - then verified the app end-to-end itself, essentially without nudges from me. The cost was time: roughly 40 minutes total, more than 11 of those spent in planning alone, before it wrote a line of application code. That's the trade this stack asked me to make: a local model plus a plan-first harness is slow.

I don't have precise wall-clock numbers for the other three runs to put next to that 40 minutes - that's a gap in what I measured, not a claim I want to paper over with a made-up multiplier. What I can say directly: Claude Code was noticeably, not marginally, the slowest of the four to reach a working result.

Two bugs, and what they actually tell you

The embedding-dimension mismatch. opencode, Pi, and Copilot each hit some version of the same class of bug: a mismatch between the embedding dimension nomic-embed-text actually returns and what the ChromaDB collection was configured to expect. Same failure mode, three different harnesses, same underlying model. That's the tell - this is a model weakness surfacing through the harness, not something any of the three wrappers did wrong. Claude Code avoided it in practice, which I attribute to the amount of planning and the end-to-end verification pass it did before calling the task done, rather than to some special embedding-dimension awareness in the tool itself. One honest caveat here: Claude Code's own pytest suite mocks Ollama and the vector store, so "24/24 green" by itself doesn't prove the embeddings actually work end-to-end - the separate run where indexing actually executed and returned real 768-dimensional vectors is the part that actually demonstrates it.

Pi's missing path argument. This is a harness-strictness story, not a model story. Pi's tool-calling is minimal and, it turns out, less forgiving of a slightly malformed call - it rejected the write call outright rather than guessing at intent. Copilot's tool definitions are richer and stricter, and in this comparison Copilot was the tool that invoked its own tooling most reliably. A stricter, more heavily specified tool schema seems to catch more of a small local model's sloppiness before it becomes a wasted turn.

The context-window trap

Small local-model context windows and heavy tool-definition overhead compound in an unforgiving way. opencode and Pi both compacted gracefully around 28-32k tokens - their own tool surfaces are lean, so there wasn't much to compact away in the first place. Copilot's agent, by contrast, spends roughly 60% of its context budget on tool definitions before the model sees a single line of the actual task. At a 32k declared window, that overhead was enough to trigger a loop. The model itself supports a 256k context (max_model_len 262144 on the server side) - raising Copilot's declared window toward 128k fixed the loop immediately. The ceiling was never the model. It was what the wrapper told the model its ceiling was.

The surprise: friction and autonomy don't move together

Going in, I expected the leanest tools to also be the most robust, since less config generally means less to misconfigure. That's not what happened. opencode and Pi were trivial to point at the model - one config file each - and both needed real hand-holding to get to a working app: nudges toward errors, retried tool calls, a defect in Pi's own validation path. Copilot needed a GUI flow, a separate utility model, and manual context sizing - genuinely the most config-heavy of the four - and it paid that cost back with the most reliable tool-calling and the only tool, besides Claude Code, that tested itself autonomously once given the room to do so.

Claude Code sits at the far end of both axes at once: the most structured harness (plan mode, sub-agents, an explicit task list) and, by a wide margin, the most autonomous result - a self-debugged fix to a real regex bug, 24/24 tests, and an end-to-end verified run with essentially no nudges from me. It also took the longest by a clear margin, even without an exact number to put next to the other three.

The pattern I take from this: the real trade-off in this stack is autonomy versus speed, not lean versus heavy. The more structure a harness imposes before it lets the model act - planning, sub-agents, stricter tool schemas - the more autonomous the eventual result tends to be, and the slower it gets there. The most config-heavy wrapper in this test (Copilot) produced the most autonomous practical result for the effort it cost to set up. The most structured harness (Claude Code) produced the most autonomous result overall, at real time cost.

When I'd reach for which tool

opencode, when I want a TUI or a small desktop UI on top of a custom endpoint with near-zero setup, and I'm fine nudging a smaller local model past the occasional rough edge myself.

Pi, when I want the leanest possible CLI and I'm prototyping fast - just budget for retrying a sloppy tool call now and then, especially on a smaller or more quantized model.

GitHub Copilot, when I want the most reliable tool-calling I can get out of a local model and I'm willing to do the GUI setup and size the context window myself rather than trust the default.

Claude Code, when I want maximum autonomy and I'm willing to trade time for it - and when I have, or can point it at, a native Anthropic-compatible endpoint directly.

The honest caveats

This was one task, mostly one model, one machine, one quantization. The findings here are directional, useful for shaping what to try next in your own stack, not a definitive ranking of four tools. The Qwen2.5-Coder tool-calling failure is specific to this quant and this server - I wouldn't carry that verdict to a different deployment of the same model.

If you're evaluating local models for agentic coding work - or want to run a comparison like this against your own hardware, your own model, your own tools - that's exactly the kind of hands-on work we do at Infralovers. Get in touch if you want a hand setting it up.