Why an index of your code cannot answer the same question twice

There is a category of tool that parses your repository into a graph. Files, symbols, imports, calls, sometimes inheritance and route definitions. The agent gets operations for querying it: find this symbol, show what calls it, walk from here to there. Some of them render the graph in three dimensions and it looks genuinely impressive.

I have spent time inside one of these, reading its source and its database rather than its documentation. I want to start by conceding what it does well, because the criticism only means something after the concession.

What a graph genuinely gives you

It makes each step cheaper.

Asking “what calls this function” without a graph means a text search, then reading candidates to work out which hits are real calls rather than a comment, a string, or an unrelated method with the same name. With a call graph already resolved (worked out ahead of time, so the tool already knows which function calls which), the answer arrives directly. That is a real saving, it is not marketing, and any honest comparison has to start there.

The same applies to reverse imports, to inheritance chains, and to the general class of question where the answer is a relationship rather than a location. Text search is bad at those. A graph is good at them. Fine.

You cannot traverse a graph until you know where to enter it

What a graph makes cheaper is the individual hop. What it does not touch is how many hops you need.

Consider what actually happens when an agent is given a task in an unfamiliar repository. Almost none of it is graph-shaped. The task arrives as a sentence in a human’s vocabulary, something like “the export is timing out for large accounts.” There is no node, no single dot in the graph, called that. Before any relationship question can be asked, the agent has to work out which files this sentence is even about, and that is a discovery problem, not a traversal problem (traversal just means walking from one connection to the next once you already know where to start).

You cannot traverse a graph until you know where to enter it. Finding the entry point is the expensive part, and it is the part the graph does not help with.

When I traced what these tools do at that first step, the answer was blunter than I expected. The entry point is found by text search: a grep, essentially, against the repository, with the results then sorted using the graph. In the common case, the graph isn’t being traversed at all. It’s used only to rank the output of a text search.

The ranking never sees your question

That ranking step is worth looking at closely, because the way it usually works has a consequence people do not expect.

The natural signal a graph offers for ranking is how connected a node is. A file that many other files import is probably important. So results get ordered by something like incoming edge count, with adjustments for whether the hit is a function or a route, and penalties for vendored code and tests.

Read that scoring rule carefully and something is missing from it. The query. How well a file matches what you actually asked contributes nothing to its position. The ranking is a popularity score, computed before your question existed and identical no matter what you type.

For a lot of questions that is fine, because popular files are often relevant. It fails in a specific and predictable way: any file that is important without being widely imported is structurally unrankable. A database migration. A configuration file. A one-off definition that exactly one caller uses. A test fixture that encodes the real expected behaviour. These have almost no incoming edges by construction, so they sort to the bottom regardless of how precisely they match the question.

And a fair share of real answers live in exactly those files. The migration is where the column was actually renamed. The config file is where the timeout is actually set. If your ranking is popularity, the leaf where the answer lives loses to a widely imported file that merely mentions the same word.

Relevance has to include the query. That sounds too obvious to state until you find a scoring function that omits it.

More edges is not more correct edges

The other thing worth checking is what the edges mean.

Resolving a call from one file to another, working out for certain which function a given call site actually points to, is hard in a dynamic language. You see a method being called on something, and to know which definition that refers to you need the type, which often is not written down. So the resolver, the piece of the tool doing this matching, guesses. A common approach is to match on name: if exactly one function in the project has this name, link to it, and if several do, pick by some rule of thumb.

I pulled the edges out of one of these databases and looked at what the guesses were. A large share pointed at language builtins, which is technically true and useless: knowing that a piece of code calls a dictionary lookup tells you nothing about your repository. Another large share were cases where the resolver had many candidates with the same name and picked one; in that bucket the average was over twenty candidates per call site. I sampled eight of those at random and checked each against the repository. All eight were wrong. One linked a call in JavaScript to a Python test fixture that happened to share a name.

The edge count was several times larger than the one I maintain. Once builtins and coin flips came out, the real gap was much smaller than the headline. This isn’t an accusation of bad faith, it’s what happens when a number becomes a feature: the incentive is to count edges, and nothing in the interface distinguishes a resolved edge from a guess. How confident the tool actually was in each guess is often recorded internally and then not shown to the agent, which means the agent treats a coin flip and a certainty identically.

If you are evaluating one of these, the useful question isn’t how many edges there are. It’s what fraction of call sites (the individual places in the code where a call happens) resolved at all, and of those, how many had exactly one candidate. Those two numbers are the honest ones and they are rarely published.

The metric that would settle these arguments is unglamorous: for a fixed set of tasks, how many actions did the agent take before it had the right answer. Steps, not tokens, because tokens follow from steps. I have not seen that number published by anyone, including by me, and I am wary of any comparison that leads with something else.

The two hours that nothing recorded

Everything so far is a quality argument, and quality arguments can be answered with better engineering. Someone can improve the resolver, add the query to the ranking, expose confidence. The next objection cannot be fixed that way, because it is about what kind of object a graph is.

A graph is derived from your source code. That is its great virtue. It is never out of date, because you can regenerate it, and if the code changed the graph changes with it.

It is also the limit. Everything in the graph is already in the code. The graph is a re-description, in a form that is faster to query. It contains no information that was not sitting in the repository already.

Which means it cannot record a conclusion.

Say someone spends two hours on a timeout bug. They check the obvious place, the request handler, and it is not there. They check the client configuration, also not there. Eventually they find that a connection pool is being exhausted by a retry loop three files away, and that two files have to change together because an invariant, a rule that has to hold true between them even though nothing enforces it, lives between them and neither file states it.

Now ask what of that is in the graph. The edges between those files, maybe, if the resolver worked correctly. Nothing else: not the fact that the handler was checked and cleared, not the reason the two files are coupled, not the invariant itself, which was never written down anywhere in the code, which is precisely why the bug existed in the first place.

Six weeks later the same area breaks again. The graph is regenerated from the same source and it is byte for byte what it was before. It has learned nothing, because it is not the kind of thing that can learn. The second investigation starts exactly where the first one did, and re-derives the same conclusions at the same cost.

That is the claim in one line. A graph makes each hop cheaper. It does not reduce the number of hops, and it never makes the second occurrence of a question cheaper than the first.

Why the missing record has to be authored

If the missing thing is a conclusion, two properties follow, and neither is negotiable.

It has to be written by whoever reached it. A conclusion is not recoverable from the source afterwards: that is exactly what made it a conclusion rather than a lookup. The only party holding the invariant, the hypothesis that was checked and cleared, and the reason two files are coupled is the one that just spent the two hours.

And it has to be a separate object from the derived index, because it fails in the opposite direction. A derived index cannot go stale; regenerate it and it is correct by construction. An authored record can, and does, the moment someone edits the file it was about. Keeping both in one structure means either regenerating the notes away or serving them with the index’s confidence, and both are worse than keeping them apart.

That is the split coldstart is built around, and it is worth stating briefly because the argument does not depend on it. There is a static index, deliberately ordinary: it ranks files against the words in your question using paths, symbol names, exports, and references, and it can show you the shape of a file and who uses it. It is the cheap-hop layer and I make no larger claim for it. Separately there is a notebook. After a real task, the agent writes down what it worked out, anchored to the files it actually used, and each note carries the state of those files at the time it was written. If they have changed since, the note is not served as truth. It is served as a claim that needs re-checking, which is roughly what a colleague saying “this was true last month” gives you.

Both layers are useful, and the mistake worth avoiding is expecting the first to do the second’s job. The general form, whatever you end up building: anything derived from your source can only tell you what is in the source. It cannot tell you what was already checked and ruled out, and on the second occurrence of a hard question, that is most of what you wanted to know.