Bill Yanelli · October 13, 2026

One from the file drawer: why I didn't build code search for agents

This summer, I prototyped a code search tool for AI agents, designed an evaluation to test it, and to my surprise, proved that it was completely ineffective. Then I discovered that my result reproduced prior research showing that: a) grep is (probably) all an agent needs; and b) search isn't really the bottleneck for agents to write better code.

I've decided to publish this negative result because I learned quite a bit along the way about building and evaluating tools for agents, and I hope it might be useful to share these insights. This post is my dispatch from the file drawer, where most failed research goes to die.

Motivation

Stop me if you've heard this one: your company wants to deploy an automated agent to triage issues and open PRs. Of course, the PRs are just a starting point, they'll still go through the review process, but how hard could it be to get some useful output out of these shiny new LLMs? I found myself working on exactly this problem at my previous job. I had paired with a coworker on developing a bug-fixing agent, but even after a few days of intensive work, the agent struggled to make anywhere close to mergeable changes to our team's large polyrepo codebase. Another coworker who reviewed the changes confirmed their quality was lacking. The project was postponed indefinitely.

The idea

Poring over the agents' output and watching them struggle to assemble the right context for the changes they were trying to make, I developed an intuition that would turn out to be utterly wrong: the problem was search! I looked at existing code search solutions, and, whether out of hubris or just an itch to build something, I took another step in the wrong direction by concluding I could do better. I set out to make my own code search tool more suited to our company's polyrepo, especially by tracking all the inter-repo references in its dense dependency graph. I would develop it on my own time as an open-source project, and eventually present it to my boss, who would be duly impressed with the results. I named it DeepTime after this album, a personal favorite.

Working with Claude over several sessions, I guided it in drafting a series of planning documents. The product (made with Laravel, what else?) would combine semantic search of code snippets with code symbol search. Exploring these areas turned out to be a great learning experience in itself: I learned about vector search and HNSW, merging it with BM25 full-text search via RRF (Reciprocal Rank Fusion), and parsing code symbols with SCIP. Claude and I did deep dives on the schema, how to ingest a repo at a certain commit and represent the file tree, how to efficiently store a table of hashed file contents while referring to them with a single bigint id via hash interning, and more. Finally, we discussed MCP tool design, how many tools to expose and arguments for each, and how to keep tool output token-efficient. I found all this genuinely exciting. It was a new type of development for a new type of "customer": an agent calling tools rather than a human interacting with a UI.

Recalling The Lean Startup, I realized I would have to put my product in front of its intended customer as soon as possible to measure its effectiveness. I would have to design an evaluation, which I earnestly hoped would vindicate the idea.

Building the eval

Having worked with Claude to implement an MVP, I switched gears to designing an eval. My initial idea was to have the agent re-implement PRs for an open source repo, with and without DeepTime, judging the quality of the results based on whether the original tests passed (and possibly having an LLM judge give an additional quality score based on simplicity of the code, etc.). Claude steered me towards a simpler design. Since we were building a search tool, why not evaluate it simply based on "localization," i.e., its ability to find the same files that were changed in the actual PR? Claude, always eager to introduce me to new jargon, called these "gold files." On the last turn of its evaluation, each agent would submit a list of files—ranked by confidence—that it considered relevant to the issue, which would be compared against the gold files.

Having narrowed down the problem, the next task was to choose a repo to evaluate against. I wanted something challenging, complex, and with a long git history. I was drawn to Gradle in part because I had used it extensively at work, and when I cloned the repo to look under the hood, I caught a glimpse of its immense complexity. With nearly 140,000 commits and 2.3 million lines of code, I thought it was a prime example of a repo that would benefit from code search.

But choosing a repo was the easiest part of designing the eval. The process forced me to continuously sharpen my idea of what I was testing. Looking at the PRs I was asking the agent to re-implement, I realized they often contained keywords (including specific references to files that should be changed) that would make search trivial. In other words, they were far more specific than the vague bug reports we were asking the agent at work to handle. The solution: before running the eval, use a smaller model like Sonnet to rewrite each PR description, keeping it detailed enough to implement while scrubbing all specific keywords (class names, file names, etc.).

Most vexing was the process of deciding how to score the PRs. I went through several rounds with Claude of scraping PRs and scoring them to find good candidates. Each time, I looked through the results and was dissatisfied—even the top candidates seemed too easy for semantic search to make a difference. These manual assessments were one of the most painstaking parts of the process, but, as is often the case, inspecting rather than trusting Claude's output was the only way to ensure quality. Finally, we clarified the criteria for a good candidate PR: the gold files should be lexically hard to find (i.e., not containing exact keywords from the rewritten PR description) but semantically easy (i.e., related by meaning or content). With that insight, I understood much better what we were testing: could semantic search help an agent take a vaguely worded problem description and use it to find obscure files in a large, messy codebase? In other words, could DeepTime be helpful under the typical conditions I faced at work?

But how to translate that intuition into a formula for scoring PRs? First of all, an agent (e.g., Sonnet) must look at each file and give it a sem and lex score: how semantically easy it is to find, and how lexically easy, respectively. Then we combine the scores by calculating max (0, sem - lex). Intuitively, you can see this formula expresses the difference that we expect semantic search to make over the agent's default search tools (grep, etc.). But Claude and I only arrived at this formula after an initial false start. Looking over the PRs Claude had scored highly, I feared many of the gold files would be either trivial or impossible to find. Nudging Claude to fix this, I learned that we had gotten the probability theory wrong. Our first formula, sem * (1-lex), had assumed semantic and lexical searchability were independent variables, when in fact they're highly correlated. As Claude put it: "a file whose name echoes the symptom described in the PR is almost always topically close to it as well. High lex practically implies high sem." For example, in this PR, AttributeContainer.java was scored as 0.19 when it should have been 0, since it would be trivially findable from even a rewritten PR description. A full table of examples makes the differences clearer:

File (PR #33690)	Lex / sem	Old formula: sem * (1-lex)	New formula: max (0, sem - lex)
AttributeContainer.java	0.75 / 0.75	0.19	0
HierarchicalMutableAttributeContainer.java	0.30 / 0.90	0.63	0.60
ResolveConfigurationResolutionBuildOperationResult.java	0.30 / 0.20	0.14	0

I emphasize the difference between the two formulas because I think it foreshadows the search tool's eventual failure. I was already scraping hundreds of PRs to find 30 that scored highly on this rubric, and in order to find PRs that scored highly on the new formula, I had to expand to scraping thousands—in the end, 3,376 were scraped, 514 were LLM-judged to be potential candidates, and 30 were chosen. Eliminating 99% of PRs from the outset should have given me a clue that the vast majority of semantically findable files are lexically findable as well, and thus that we should expect DeepTime to have a small marginal benefit.

The method, in brief

We settled on the following method to run the eval:

For each of the 3,376 PRs scraped, use Sonnet to rewrite its description, scrubbing any directly identifiable text that could be used for lexical search, including class names, file names, and other identifiers.
Use Sonnet as a judge to score each gold file in the PR on lexical and semantic searchability (based on the scrubbed description), calculating the score as max (0, sem - lex).
Aggregate those scores per PR and keep the 30 highest-scoring ones.
For each of the 30 selected PRs, check out the repo at the base commit. Run one agent with DeepTime and one without, instructing it to submit a ranked list of gold files.
Score each list against the actual gold files (percentage found in the top k, plus a rank-weighted score).

We "pre-registered" a definition of success: the agent with DeepTime had to beat the one without by at least 15 percentage points.

Running the eval

I finally confirmed through manual inspection that Claude's ranking system was good enough to select the 30 most promising PRs from the history of the Gradle repo. To run the eval, we settled on using Claude Code headless (via claude -p) with GLM 5.2 as the model. We used the --strict-mcp-config flag to avoid any unauthorized tool calls, and sandboxed the agent using macOS Seatbelt. Crucially, I insisted Claude write every agent's output to a nicely formatted HTML log for me to inspect.

The bugs I encountered during the first four attempts at running the eval were so numerous that describing them in detail would lose the attention of anyone still reading this post. But every bug can be restated as a question the developer failed to ask, so in that spirit, let me try to summarize:

How best to prevent an agent from cheating by reading future git history without affecting its ability to run commands like git log? Shallow clone at the base commit.
How many turns should an agent have to find N gold files? Not 10 as we initially guessed, but 30.
How to parallelize the eval and run it cheaply in the cloud? Eight Docker containers on a Hetzner box with 32GB RAM.
How to prevent the agents from spawning subagents? Use --disallowedTools Agent.
How to ensure, despite the otherwise strict parameters, that the agents are authorized to call the tools you're trying to test!? Yes, this invalidated an entire run in which both the control and experimental group were using grep all along. The fix: use --permission-mode default and explicitly allowlist the tools.
How to nudge the agent to call your tools even when it does have permission? Rewrite its prompt to discuss the relative merits of semantic search and grep.

During the last two runs, as the eval finally became reliable, it began to surface problems in DeepTime itself (which we knew was an unpolished, naive MVP implementation), mostly having to do with poorly structured SQL queries that fell apart against a repo of Gradle's scale. The most serious functional issue was that all keywords in the full-text search query were being joined with AND, causing the search tool to return zero results in many cases.

The result

After working through these waves of bugs, on July 3rd, I was finally able to arrive at a result that I considered valid. "Adaptive recall" means the fraction of gold files that appear in the top k files the agent chose, where k = max(10, gold_files_count). Looking at adaptive recall for all tasks combined weights each task by its gold files count, giving a higher weight to larger tasks, so we also considered a simple per-task average that weights all tasks equally. In practice, these turned out to be nearly the same. "MAP" means "mean average precision," a score based on how highly ranked the gold files are in the agent's list (which the agent was instructed to rank by confidence).

metric	with DeepTime	grep only	Δ
adaptive recall, per-task	0.423	0.435	−0.012
adaptive recall, all tasks combined	0.429	0.423	+0.006
MAP, per-task	0.326	0.363	−0.037
tokens per episode	88,249	46,471	+41.8k (≈1.9×)

My heart sank when I saw the numbers. DeepTime clearly failed to give the agent an advantage over grep. Not only that, it used almost twice as many tokens in the process, despite the iterations I had gone through with Claude to improve the search tool's token efficiency by changing its output from JSON into a grep-like format.

At this point, I hadn't implemented vector search yet. I was discouraged, but I wanted to be thorough, so I worked with Claude to combine the existing BM25 keyword search with vector search. We also broadened the PR count to 40 and introduced a simpler filter: a PR qualified if at least one of its gold files was scored lex ≤ 0.4 and sem ≥ 0.5—hard to find by keyword, but reasonably easy to find by meaning. (With the new filter, 20 of the original PRs qualified, plus 20 new ones.) After running the eval again, the results were almost identical. With DeepTime, the agent still named fewer gold files (−0.028 per task) at ~1.75× the tokens, while reading slightly more of them (+0.022).

Postmortem

To be clear, the experience of building DeepTime up to this point had already been tremendously valuable for me. It was my introduction to a new paradigm of development. If writing unit tests resembles mathematical proof, designing evals is closer to the scientific method—it adds an element of empirical messiness usually absent from the logical world of programming. The next step, then, was to embrace the mess and dig into the data generated by the final run.

The explosion in token use was perhaps the lowest-hanging fruit I could have picked if I had decided to try another run. The data here was unambiguous: the default number of search results was 25, but a gold file was never found after the 9th result. For agents that had access to it, output from the search tool took up a whopping 78% of the token budget. First lesson learned: optimizing tool output for token efficiency includes making tradeoffs between relevance and verbosity, and this decision should be data-driven.

The more interesting discovery, and one that would have implications for further research, was the discrepancy between files agents read and how they ranked those files' relevance to the PR. Agents with DeepTime read more gold files (+0.022) while ranking fewer (−0.028) within the top k, raising an issue I had never considered: improving search means more than improving recall, it means convincing the agent that the files recalled are relevant. Perhaps due to grep's directness (a lexical match is an unambiguous signal), or perhaps because it's so well-represented in the models' training data, they are better able to connect recall with relevance when using grep. Second lesson learned: you can lead an agent to gold files, but you can't make it rank them.

Further research

Acknowledging DeepTime's failure made me more curious to understand the state of existing research and prior art on the issue of code search. I wanted to understand why my hypothesis wasn't borne out. I put Claude on the case, and it came back with some fascinating results. The existence of other code search tools like Sourcegraph might have biased me too much in favor of the search idea from the outset. The first question I should have asked myself: if semantic search is effective, why isn't it already being deployed at the broadest possible level, i.e., within the AI labs' first-party harnesses? Sure enough, most of them have considered and rejected the idea. In January, Boris Cherny of Anthropic tweeted: "Early versions of Claude Code used RAG + a local vector db, but we found pretty quickly that agentic search [i.e., grep] generally works better." OpenAI's Codex uses a fuzzy matcher with no vector component. Cline has also rejected vector search, arguing that "[l]anguage models are powerful enough to work with code the way developers do." The biggest exception to this industry consensus is Cursor, which has "trained [its] own embedding model and built indexing pipelines for fast retrieval." In contrast to other AI labs' findings, as well as my own results, Cursor found that "semantic search significantly improves agent performance." On the one hand, there are reasons to doubt this result: the company's blog post on this issue is from November 2025; models have dramatically improved on agentic use cases since then, including search. On the other hand, Cursor may have achieved better than average results because its embedding model was trained on agent traces, giving it an edge over DeepTime (which used voyage-code-3, an off-the-shelf embedder).

Notably, however, a more recent paper supports the anti-embedding, pro-grep position. In May 2026, researchers at PwC published "Is Grep All You Need? How Agent Harnesses Reshape Agentic Search." With some caveats, and noting differences in performance between harnesses, they concluded that "grep generally yields higher accuracy than vector retrieval in our comparisons."

The most interesting paper Claude found helped me zoom out even further and question the utility of improving search at all, regardless of modality. Only a month before I started working on DeepTime, AI researchers at Microsoft and the Georgia Institute of Technology published "ORACLE-SWE: Quantifying the Contribution of Oracle Information Signals on SWE Agents." The authors break down the information AI agents typically lack into five categories: "Reproduction Test, Regression Test, Edit Location, Execution Context, and API Usage." By "edit location," they mean something even more specific than gold files, i.e., the precise line numbers in the gold files where edits should be made to accomplish a task. The authors test agents with an "oracle" in each of the five categories that injects perfect information into the session, allowing them to measure how each type of knowledge contributes to an agent's success. On SWE-Bench-Live and SWE-Bench-Pro, the strongest of the benchmarks they test, edit location turns out to be the second least effective factor, with by far the most effective one being a reproduction test (a failing test reproducing the bug). Reading this team's results helped place mine in context: not only was my search tool ineffective, even with perfect search, agents would only write slightly better code on average.

Conclusion

While I think the null result is meaningful, there are some caveats:

The eval only covered a single repo (rather than a polyrepo similar to my work project that motivated this endeavor in the first place).
It only scored the model on localization rather than a more holistic test of how search might enhance its coding ability.
Code was embedded using an off-the-shelf embedding model.
The proposed code symbol graph was never built (it would have been the next step).
The units of work tested were PRs rather than bug reports, slightly out of sync with the initial problem I was trying to solve, although I tried to mitigate this by using LLM-rewritten PR descriptions.

Beyond the lessons enumerated above, I would say the most important lesson I learned was to approach the issue from the top down. If you're working on a tool for agents, start with rigorous research into prior art. Specifically, if you're working on a tool to give agents information, start with an oracle test similar to the ORACLE-SWE paper. (In my case, I should have tried "grep vs. oracle" before implementing anything at all.) Understand the difference between an agent knowing something and acting on that knowledge. May your results be significant, and if not, consider taking them out of the file drawer!