Local Hybrid Search: Retrieval Without a RAG Pipeline

Two weeks ago I wrote about what changes when you keep the model fixed and swap the harness. The short version: the wrapper matters more than people assume. This post is about the layer underneath that wrapper, the one nobody puts in the architecture diagram until it breaks: where does the agent get its knowledge from?

For the last few years the reflexive answer has been RAG: chunk, embed, put it in a vector database, write a retrieval function, tune it until it stops embarrassing you. One of our internal experiments does exactly that, and I will come back to it briefly, because it is a useful contrast rather than the subject.

The subject is the other answer. A colleague runs an internal content wiki: a git repository of markdown, layered from immutable raw sources through curated wiki pages up to content drafts, with agents doing most of the maintenance. Its search layer is not a pipeline anyone wrote. It is qmd, a local search engine for markdown and code.¹ BM25, vectors and an LLM reranker, all on device, exposed as a CLI and an MCP server. One SQLite file, no service, no API key.

That repository is the reason this article has numbers in it. It is not a product, it is somebody's experiment on real material, which is exactly what makes it worth reading: a knowledge layer that has been in daily use long enough to have an operating model and a measurement, rather than a demo.

Why This Is Coming Up Now

Two things converged in July 2026.

The first is that the models this needs got small. An embedding model at 300M parameters and a reranker at 600M now fit comfortably on a laptop, which means vector search over a personal or team-sized corpus no longer requires an embedding API or a database service. Retrieval is exactly the layer where local stops being a compromise: the models are small and the corpus is yours.

The second is that the "give your LLM a wiki" pattern went mainstream. Andrej Karpathy's LLM Wiki gist landed in April and passed 5000 stars and forks within weeks, with more than twenty independent implementations appearing in the first five.² Its core argument is aimed straight at RAG: the model rediscovers knowledge from scratch on every question, and nothing accumulates. A wiki compiles knowledge once at ingestion and keeps it current.

That pattern has a scaling problem, and Karpathy says so himself. A plain index file holds up at moderate scale, somewhere around 100 sources. Past a few hundred documents you need real search. In the July update of that gist, the tool he names for it is qmd.

Which is why the wiki repository I mentioned is worth looking at as a real instance rather than a thought experiment. Its layout, immutable raw sources plus an LLM-maintained wiki plus a schema file that defines the conventions, maps onto that pattern closely enough that reading one explains the other, and it is well past the scale where an index file would still work.

So the question is not academic anymore. If you are building a knowledge layer for agents in 2026, you have to decide whether you are writing retrieval code or installing it.

The Contrast, Briefly

For calibration, here is the shape of the hand-built route. Our other experiment is a chatbot over scraped website content in ChromaDB, and its ingest and retrieval path runs to a couple of thousand lines of Python.³ Getting that far is the easy part. What follows is the recurring work, and it is the same in most bespoke pipelines: synonyms and domain vocabulary live in hand-maintained tables, so the retrieval layer needs editing whenever the content it describes gets renamed; relevance becomes a weighted sum whose weights a human chose, usually without a reranker in the loop; and chunking is a fixed character window with overlap, which cuts across sections and grows the index to compensate.

That is a normal shape for retrieval you build yourself, not a fault in any particular codebase. Ours answers questions today, and it carries guardrails and per-language filtering that qmd has no equivalent for. The only point worth carrying forward is that every piece of it is plumbing somebody owns. Now the other route.

What qmd Does Differently

qmd is deliberately small. It indexes files, it returns documents, and it stops there. No generation, no answer synthesis, no opinions about your prompt.

Three modes: search is BM25 only, vsearch is vector only, query is the hybrid path with query expansion and reranking. Three models run locally through node-llama-cpp: EmbeddingGemma-300M for vectors, Qwen3-Reranker-0.6B for relevance, and a 1.7B model for query expansion. The index is a single SQLite file at ~/.cache/qmd/index.sqlite, using FTS5 for full text and sqlite-vec for vectors. MIT licensed.

Four consequences follow from that, and the largest is maintenance. The hand-maintained vocabulary table from the previous section has no counterpart here, because query expansion is what the third model does: nobody has to teach the index that a product got renamed. Chunking is the second. qmd scores natural markdown break points so that sections, paragraphs and code blocks stay intact, instead of cutting at a fixed character count and hoping the overlap catches what got split. Collections default to a **/*.md glob, but the mask is configurable, and source files in TypeScript, JavaScript, Python, Go and Rust get AST-aware chunking when you opt into the automatic strategy.

The third is operational, and it is the one that shows up in a budget: there is nothing to run. No Chroma, no embedding API key, no network egress, which for a knowledge base full of internal notes, customer context and half-finished strategy is less a convenience than a precondition. The fourth is that it arrives agent-ready. The MCP server exposes query, get, multi_get and status over stdio or HTTP, and that is the entire integration: you do not write a retrieval tool for your agent, you point it at a server that already is one.

One wrinkle for the local-inference crowd: qmd runs GGUF models through node-llama-cpp, not MLX. On Apple silicon that leaves some performance on the table, so if you are chasing query latency rather than result quality, that is the first thing to look at.

Using It as a Knowledge Layer

A feature list is not an operating model. Three habits are what turn qmd from a search command into a knowledge layer, and none of them are qmd features as such.

Put hybrid search next to the other ways in, not instead of them. Fuzzy recall is what hybrid is for, the case where you know what a thing does but not what it is called. Existence checks with a term you already know are better served by plain grep or a deterministic index, because those are faster and exact. Questions about how things connect belong in a graph view, since no search mode answers "what links to X". And when hybrid comes back weak, re-run the query once as pure BM25 with exact keywords before concluding the corpus has nothing on the topic. Four routes, one rule each, written down. That division of labour is the part people skip, and it is why the tool does not have to be good at everything.

Put reindexing in a task, not in a habit. qmd update and qmd embed are incremental, keyed on content hashes, so unchanged files cost nothing. That makes it cheap to hang them off whatever already runs after content changes, together with anything else derived from the index. Retrieval quality that depends on somebody remembering to reindex is retrieval quality that rots quietly.

Treat the index as a substrate, not a black box. This is the argument for SQLite that I underestimated. Because the vectors sit in an ordinary file, other tooling can read them with no API and no export step. In the setup I looked at, a small script reads the embeddings straight out of qmd's database and projects them into a two-dimensional map of the corpus, a second consumer of the same index. With a hosted vector database that is a project.

Where These Numbers Come From

I did not run this measurement. It already existed in that wiki repository, dated early July 2026, qmd 2.5.3.⁴ Quoting it is more useful than staging a weaker version on a smaller corpus.

What makes it worth quoting is the design of the golden set. The 17 queries are taken verbatim from real session prompts over June and July, not written for the benchmark. They are split into two classes that are expected to have different winners:

• coverage: "do we already have X?" The exact term is known. Grep should win.
• episodic: "there was that thing about..." The term is unknown or paraphrased. Hybrid should win.

Metric is hit@5. Results:

Hybrid wins overall, and it is the only mode that finds two of the queries at all. That is the headline. The three caveats below it are more useful than the headline.

The grep column is not what you think. It measures a naive word-count baseline over the raw query, filler words included. It is not agentic grep. A human or an LLM that extracts the term opentelemetry from "do we already have opentelemetry as an entity" hits coverage 5 out of 5 without breaking a sweat. The coverage row is not evidence against grep. It shows that naive automation fails there, which is a different claim.

BM25 alone is the wrong tool for natural language. Four out of seventeen. As the exact-keyword counter-check described earlier it earns its place; as a primary mode it does not.

Rerankers can bury exact matches. Two of the four hybrid misses were found by grep and BM25 respectively, then pushed out of the top 5 during reranking. This is the failure mode people do not expect from adding a reranker, and it is the reason the BM25 fallback exists rather than being a leftover.

If you take one thing from that table, take this: hybrid is the best default for fuzzy recall, and it is not a replacement for deterministic lookup when you already know the term. Keep both.

What This Means for Agents

This is where the decision stops being about search quality and becomes an architecture question.

When you build RAG for an agent, you are building a tool the agent calls. You own the chunking, the scoring, the tuning, the eval harness you probably never wrote, and the deploy pipeline for a synonym dictionary. All of that is undifferentiated work. None of it is your product.

When you install a local search engine with an MCP server, retrieval becomes infrastructure. The agent gets a knowledge tool for the price of a config entry, and the interesting engineering moves up a level: what goes into the knowledge base, how it stays current, and what the agent is allowed to conclude from a hit.

That last one deserves emphasis, because it is where teams get burned. The rule worth adopting is that retrieval output never gets passed on as fact without reading the source first. qmd returns documents with scores; it does not return truth. An agent that summarizes search snippets into a customer-facing answer is a hallucination generator with extra steps, no matter how good the reranker is.

The other thing worth copying is the eval harness, and it is cheap. Seventeen queries in a YAML file, pulled from questions people actually asked, with the expected document path for each. That is an afternoon of work that turns every future retrieval change from a vibe into a measurement.

What struck me reading the wiki repository from the outside is that the golden set carries the same date as the search layer itself. The measurement was not bolted on after the first disappointing result, it shipped with the walking skeleton. That is the discipline I would steal, ahead of the tool choice: one of our two systems has a golden set and the other does not, and the difference in how confidently either can be changed is entirely down to that.

Where This Fits, and Where It Does Not

No table, because the decision is not multi-dimensional. It comes down to what you are actually building.

Adopt qmd when your corpus is text on disk and the consumer is you or an agent working on your behalf. Personal notes, engineering docs, meeting transcripts, an internal wiki, an ADR collection, a source tree. Markdown is the default and the best-supported case, but the glob mask is configurable and common source languages get AST-aware chunking. If you are about to write a chunking function for a directory of documents, stop and install this instead. The setup is one npm install plus an index run.

Keep your RAG pipeline when you have requirements qmd does not address and does not pretend to. Multi-tenancy and per-user access control. Guardrails and content filtering for anything user-facing. Sources that are not files on disk: PDFs, HTML scrapes, database rows, ticket systems. Horizontal scale beyond a single machine. Anything where retrieval is part of a product with an SLA rather than part of somebody's workflow. Our chatbot experiment sits in that column, and the most useful next step for it is not qmd, it is a golden set.

For DevOps and platform teams specifically, the practical move is to stop treating retrieval as a build. Two roles, two answers. If your platform serves customers, you are operating a retrieval service and you should own it properly, with evals and observability. If your platform serves your own engineers and their agents, you are consuming a commodity, and writing it yourself in 2026 is a choice worth justifying in a design review before the work starts rather than after.

The uncomfortable summary is that the interesting problem was never retrieval. It is the knowledge base itself: what gets captured, who maintains it, and what stops it from rotting. RAG code was always a detour on the way to that question. Local hybrid search is just the first tooling that makes the detour optional.