This page is the method I used to measure whether coldstart actually saves tokens, and the things that went wrong on the way. My first attempt ran on Kafka, Django and Mastodon, and the numbers it produced were wrong in a way I couldn't see until I went looking. The rebuild is now a separate tool you can point at your own repository, so you can check a skill, a rules file or an MCP server without taking my word for anything.
← back to the overviewA session has two halves. The agent works out which files matter, and then it reads, reasons and edits them. Only the first half is what a navigation tool touches. The second half is the same work whichever setup you're running, so measuring it adds the same cost to both sides and makes the difference between them look smaller than it is. So every question here asks the agent to find files, and stops there.
The first version was straightforward. I pointed an agent at a repository, let it skim the code and read commit messages, and had it write questions from what it found. Then I ran those questions on Kafka, Django and Mastodon. Large, real, well organised codebases. They are also three of the most written-about projects on the internet, and that turned out to matter more than anything about how they're built.
Claude has read all three. So the agent would take a question, already know roughly where the answer lived, and go straight to coldstart with the exact identifiers that happen to match the real filenames. It would get the right files back, read them, read the neighbours to be sure, and produce a correct answer. That looks like the tool working. It isn't. The agent was using the tool to confirm what it already remembered, and the questions themselves had been written by an agent that had read the code, so they pointed at the answer too.
You can test for this in your own results without rerunning anything. Take the arm with no tool installed, look at its very first move, before it has opened a single file, and pull out the internal class names it used. Subtract anything the prompt handed it. Whatever is left didn't come from the codebase, because the agent hadn't looked at the codebase yet.
This isn't noise that averages out. It pushes the result one way. Knowing the codebase makes the no-tool arm cheaper, because an agent that already knows where to look skips the searching it would otherwise pay for. My savings on those three ranked Mastodon 43%, Django 28%, Kafka 15%, which is the leak rates above in reverse. I had been reading that ranking as something about languages and codebase structure. Most of it was about what the model had already read.
Two things had to change. Smaller, less written-about repositories, so the model isn't recalling instead of searching. And questions written by someone who didn't know the answer.
Issue reports solve the second one on their own. An issue is filed before a solution exists, by someone describing a symptom, so it uses the project's own vocabulary without pointing at a file. Nobody had to be careful about it. It's just who was holding the pen and when. Commit messages and pull request descriptions are the opposite: written afterwards, by the person who did the work, naming classes and paths without thinking about it. That's what my first version was built on.
The answer comes from the same place the question does. Every issue I use is closed by a merged pull request, and the source files that pull request changed are the files a correct answer had to name. That's ground truth taken from the repository's own history rather than from my judgement, and it works on any repository with git history, including a private one.
A lot of candidates don't survive that, and on one repository more were thrown out than kept. Each one I drop is recorded with the reason, because a list of what was removed is the only way you can tell that what's left wasn't picked to make the result look good. The whole procedure is now its own repository, coldbench, since it has nothing to do with coldstart and shouldn't live inside it.
Two repositories, run more than once each. Arches is a Python and Django application. JMRI is Java, considerably more verbose per file and with a heavier build surface. Two arms in each: plain Claude Code with the search tools it ships with, and the same thing with coldstart installed and nothing else changed. Same questions both sides, Sonnet 5 throughout, one fresh session per question.
Both figures are the CLI, which is how coldstart is meant to be run. Those two also disagree, and I'm not going to average them. One tidier number would hide the only interesting thing here, which is that Java's verbosity and build tooling change what a single turn costs, and that changes how much a tool which cuts turns can save you.
Every question is one file of plain text with the issue number recorded beside its gold set, so you can open the original report and check the rewrite against it. Each is tagged by how far the fix spread, from one area to across subsystems, because a tool that helps on the narrow ones and not the wide ones is making a different claim than its average suggests.
None of this is specific to coldstart. The same setup measures a skill, a rules file, an MCP server, or a change you made to your own prompt. Take two copies of one repository, install the thing you're testing in exactly one of them, ask the same questions in both, and score what comes back against the commits.
Two longer pieces sit behind this method, both about the ways a benchmark misleads the person running it: why most token savings tools lie and how your own benchmark lies to you first.