How we benchmarked the file-finder — and what we got wrong twice.

The cold open

On a Tuesday in March 2026 I realised the file-finder was lying to me. I asked it for a tenancy agreement I’d signed two years earlier, and it came back instantly with one row at the top of the results and a confidence number next to it: 92%. The row pointed to a PDF on my old laptop — the laptop I’d wiped in January.

The file didn’t exist anymore. The vector embedding for its summary chunk still did, because the edit-history sidecar hadn’t caught the deletion (the laptop was offline when I wiped it, so the watcher never fired). And the retriever was happily ranking ghosts at the top of my results.

That was the morning I stopped trusting the number on the screen and started building a real benchmark.

What we were measuring

The first decision was what to count as a win. Precision@1 is what most retrieval papers report — did the top result match the gold answer. But on a personal corpus where I’m the only user, what I actually do is scan the top three results, recognise the right filename, and tap it. So I made top-3 hit rate the north star and stopped reporting precision@1 entirely. It encouraged tuning that hurt the lived experience — bumping one knob to win the #1 slot can cost two other queries that fall off the top three.

The corpus, fixed at the time of benchmarking, was 1,547 files: PDFs, scanned images, Office docs, Markdown notes, opaque Telegram cache blobs, and the entire export of my Samsung Notes app. The query set was 95 hand-graded prompts I’d written across two weeks — each one annotated with the correct file ID and a one-line note on why I’d need it. On the version of the system I shipped in May, the score was 93 top-3 hits out of 95. The two misses were a Samsung Notes screenshot from 2023 and a PDF whose title was entirely in Marathi devanagari while my query was English transliteration. I keep both in the set on purpose.

The first thing we got wrong

The first benchmark I ran came back at 96/95. Which is an embarrassing number to type because, obviously, you can’t score higher than the denominator. What had happened was sloppier than a counting error and more interesting.

I’d been writing the queries while looking at the corpus. Two weeks of "let me grab a query for the lease PDF… right, what would I type?" My brain was constructing queries the system was already good at. I was benchmarking on the happy path. Every query that the retriever failed silently never made it into the test set, because I never thought to write down the question that didn’t have a satisfying answer.

The fix was negative-query sampling. I started a new file, queries_blind.txt, and committed to a rule: I write the query first, then run the retriever, then grade. If I can’t remember whether a file exists at all, I write the query anyway. If the system says "no match," that counts as a graded result — either a true negative (file genuinely isn’t in the corpus, retriever correctly abstained) or a miss (file is in there, retriever flinched).

On the corrected benchmark, the score dropped from 96 to 93 out of 95. And 93 was the honest number. The three queries that fell off were all "I think I have something on X…" prompts — the kind I’d never have written when I was already staring at the right file.

The second thing we got wrong

The second mistake was more technical and lasted longer because the numbers looked good the whole time.

I added a time-decay term to the ranking math. The intuition was reasonable — recently modified files are probably more relevant than ones that haven’t been touched in three years. I picked a six-term decay function with a half-life tuned to file modification time, and on the test set the top-3 hit rate ticked up by one point. Felt like a free win.

Then I used the system for a week and it got worse. Specifically: every time I went looking for an old contract, an old tax form, an old letter, the retriever buried it under newer files I didn’t want. I almost never search for files I’ve recently touched. If a file is new and in my head, I open the folder I just put it in. The reason I’m typing a query into the file-finder is precisely because the file is old enough that I’ve forgotten where it lives.

The decay function was tuned to a world that didn’t match my behaviour. The test set looked fine because I’d written the queries during the same two weeks — the “old” files in my benchmark were already biased toward whatever I’d been thinking about lately. The fix wasn’t to retune the half-life. It was to invert the curve: a small bonus for files that haven’t been touched in over six months, capped so it doesn’t dominate, and gated by whether the query contains any explicit time signal ("last year", "old", "2023"). On real usage that took the lived hit-rate from "noticeably worse" back to baseline, then half a point past it.

The 13 stages of the retrieval pipeline

The pipeline that produces the 93/95 number has 13 stages, and they earn very different amounts of the win. Cheaply summarised: tokenize the query, embed it, run a BM25 keyword prefilter against the lexical index, fetch the top 100 candidates, rerank with a cross-encoder (bge-reranker-base against query × candidate), apply the inverted time-decay term, run semantic-group dedupe so a single file can’t occupy multiple slots through different chunks, diversify the top results with a light MMR pass, and emit the final ranking.

I ran ablation studies to see where the win lived. The reranker earned about 60% of it. Pulling the cross-encoder out of the pipeline and going straight from hybrid fusion to final ranking dropped the score from 93 to 78. That’s a colossal swing for a single component. The reason is mechanical: the hybrid fusion stage gives me a top-100 that contains the right answer in about 91 of 95 queries, but ranked anywhere from position 1 to position 40. The reranker is the only stage in the pipeline that does pairwise semantic comparison, and it’s the only stage that consistently moves the correct file into the top three.

The rerank stage costs about 140ms per query on my laptop GPU and roughly nothing in API spend — bge-reranker-base runs locally, which matters because the bot is on a free tier and the whole point of the architecture is that the API bill is a constraint, not a footnote. The dedupe and MMR stages earn another 15% between them, mostly by stopping a single chunky PDF from squatting on every slot. The remaining 25% is spread thinly across the seven other stages, which is what an ablation budget looks like when you stop pretending every clever idea is load-bearing.

What this taught us about benchmarking agentic systems generally

Three things port across the work we build, not just this one bot.

Sampling is where most agentic benchmarks die. If a human wrote the test queries while looking at the corpus, the numbers will overshoot reality by 3–5 points every time. The fix isn’t complicated, it’s just disciplined — write the query first, grade after. Keep the negative cases. Keep the queries you can’t answer.

The time axis lies to you. A test set built in a two-week window has whatever recency bias the author’s recent week carried. Tune anything to that test set and you’ll ship a system that’s subtly mis-weighted on the dimension your benchmark couldn’t see. We caught it on time-decay; we’ll catch it on other axes too, because we’re now suspicious of any monotonic factor that “feels right.”

And the deeper one: "did the model say something plausible" is not the same question as "did it solve the task." The 92% confidence number on the ghost file was plausible. The retrieval was wrong. Agentic systems are particularly good at producing fluent, confident-looking failure, which is why every benchmark we run now ends with a manual spot-check of the top 10 most confident wrong answers. That’s where the embarrassing bugs hide.

Close

What’s next: I want to push the file-finder from 93/95 to 95/95 by adding a query-rewriting layer in front of the retriever. The two queries it still misses are both about vocabulary mismatch — the corpus uses one word, I type a different one, and no amount of synonym expansion is going to close that gap. A small LLM rewrite pass over the query, conditioned on a one-line description of the corpus, should do it. I’ll know in about three weeks — the ablation matrix is already in benchmarks/2026-06.yaml, the query set is frozen, and I’ve promised myself I won’t look at the corpus while writing the next ten queries.

If it works, I’ll write that up too. If it doesn’t, I’ll write it up worse.