Why most token savings tools lie

The JMRI sweep came back at 39% fewer tokens. That is a strong number, it was measured on a real repository with a real agent, and I nearly put it on the site.

What stopped me was wall-clock time. The baseline runs in that sweep were taking noticeably longer than the coldstart runs, by more than the token counts alone explained. The machine running them was also doing other work at the time, and the baseline’s heavier reliance on repeated full-file reads and directory walks made it more sensitive to that contention than coldstart’s smaller, targeted calls were. Some part of that 39% wasn’t the tool. It was a machine under uneven load.

I reran it in isolation and got 31%. Everything else here is downstream of those eight points, so the measurement comes before anything I draw from it.

What the two arms actually were

The standard, stated before the numbers: turns and total tokens across a whole session, coldstart wired in against a no-tool baseline doing the same task with plain file reads, greps, and directory listings. Call each of those two setups an arm, the standard term for one side of a comparison like this. Each arm is a real agent session against a real repository, tokens read off that session’s own usage records, not estimated and not summed from a printed total. Both arms ran the same fixed list of queries, once each, no retries.

I ran this on two real open-source applications rather than a synthetic benchmark repo, because a repo built to be benchmarked tends to have suspiciously clean naming and structure that a real codebase doesn’t. Arches is a Python and Django application, 27 queries. JMRI is a Java application, considerably more verbose per file and with a heavier build surface, 25 queries. Both query lists are navigation questions representative of what an agent actually asks while working a real task in that codebase: where’s the thing that handles this, who calls this function, what does this error trace back to. Recall, meaning how many of the files a correct answer actually needed were the ones the agent found, was scored against a fixed, pre-written list decided before either arm ran, not reconstructed afterward from whichever arm did better.

The number after the machine was quiet

The Arches sweep, 27 queries, came out to a 64% reduction in tokens against the no-tool baseline, with recall two points better on top of that, because the baseline sometimes ran out of turns before it found the right file at all and gave a worse answer, not just a slower one.

The JMRI sweep, 25 queries, is the one from the top of this post. Rerun in isolation with no other processes competing for CPU, it comes out at 31%, with recall at parity rather than better. Eight points lower than the run I almost published, and a weaker claim on recall as well.

Why the correction is the point

It would have been easy to keep the first number. Nobody auditing a percentage on a website reruns your benchmark. But if you can’t explain how you got a number, it doesn’t count as a number, only as a claim, and the difference matters most exactly when nobody’s checking. The methodology has to be something a skeptical reader could redo: real repository, fixed query list, controlled machine, counted from the actual transcript’s usage records rather than eyeballed from a printed summary, same as the process in the tokens post.

Both numbers also stayed inside the standard from the top of this post: whole session, both arms, same task, counted from the transcript. A tool that only ever shows you a percentage without saying what it was measured against, whole session or single call, is hiding the one fact that would let you check it.

What these numbers don’t cover

Both sweeps measure navigation: an agent asking where something lives, who calls it, what it depends on. Coldstart also keeps a notebook, and after a real task it writes down what it worked out, which costs its own tokens on top of whatever the navigation calls cost. That write is not in either number above, because a pure navigation query doesn’t trigger it, and I haven’t yet run a sweep built to isolate what a full task, edits and note-writing included, costs with coldstart against the same task without it. I’d rather say that plainly than fold in a number from a different kind of run just to make this one sound more complete. When that sweep exists, it gets its own post, not a quiet edit to this one.

Same caveat on repeats: each arm here ran once per query, not several times averaged. Token counts on a coding agent vary run to run even with nothing else changed, so a single run tells you what happened, not the full spread of what could happen. Wider than that, I haven’t measured.

The shortcut most published numbers take

Most token-savings numbers you’ll see for agent tooling are measuring the wrong thing, and the reason is mechanical, not a matter of anyone rounding generously. I covered the mechanism in Where the tokens go in an agent session: on a real trace I decomposed turn by turn, about 90% of total tokens were cache reads, the accumulated conversation getting re-billed on every single turn. Coldstart’s own output, the part a tool can shrink directly, was roughly 2% of the bill.

That ratio is why compressing an individual call doesn’t tell you what you think it tells you. If a tool takes a grep result, a directory listing, or a batch of retrieved context and returns a smaller version of it, that’s a real, measurable reduction of one artifact. It says nothing about the session, because it doesn’t touch turn count, and turn count is what the other 98% is riding on. Worse, it can go the other way: give an agent cheaper access to more context and it often just asks for more of it, or issues more calls to compensate for a thinner answer, and the total goes up even though the one number that got measured, the size of a single response, went down.

So the only claim worth calling a savings number is total tokens (or dollars) across an entire session, one full task done with the tool against the same task done without it, counted from the session’s own usage records afterward. Anything narrower, a single call’s output, a compressed context batch, a shorter prompt, is a real measurement of something, just not of that.

When smaller costs more

There’s a version of this that doesn’t just measure the wrong thing, it can point in the wrong direction entirely, and the mechanism is worth understanding because it isn’t obvious from outside.

Prompt caching is what makes a long agent session affordable at all, and every provider that offers it works the same way underneath: the cache is keyed to an exact prefix, the literal, unchanged beginning of the conversation. Send that same prefix again on the next turn and you pay a fraction of the price for it. Change one byte anywhere before the cache boundary, summarize an earlier exchange, drop a tool call judged no longer relevant, reorder something to save space, and the match breaks from that point forward. Every token after the edit gets billed fresh on the very next turn, at full price, regardless of how much smaller the edited version is.

That means a tool can genuinely shrink what’s sitting in the context window and still raise the bill for that session, because it converted tokens that were about to be cheap cache reads back into expensive cache writes. A dashboard reading “40% smaller context” isn’t lying about the context. It’s just not the same claim as “40% cheaper session,” and on the same transcript, the two numbers can point in opposite directions.

The ratio this rides on makes the bet a bad one by default. On the session I decomposed in the tokens post, the fixed base alone, system prompt plus tool schemas, came to roughly half the entire session’s bill, and every one of those re-reads was a cheap cache hit precisely because nothing before that point in the conversation ever moved. Touch anything earlier in the transcript to save space and you’re betting against that ratio, not for it: you’re risking the cheap half of the bill to shrink a number that was never the expensive part.

So the question worth asking of anything that compresses or rewrites context already in the conversation, as opposed to a single new call’s output, is whether its savings number survives being measured after the edit, from the resulting usage record, on the same real task, rather than compared against the size of the thing it rewrote. If the number only holds at the instant the edit is applied, it isn’t a session cost number yet. It’s a description of the edit.

Two questions to ask of any savings number

None of this needs a benchmark to check. Take any published savings number and ask two questions. First, is it measured across the whole session, from the final usage record, or against the size of one call, one context batch, or one edit? Second, if it involves rewriting or summarizing context that was already in the conversation, does the number still hold after accounting for the cache writes that rewrite forces on every token that comes after it? A number that survives both is measuring a session. A number that only survives the first is measuring a call. A number that fails the second was measuring the wrong direction the whole time.

What a savings number is actually a property of

Arches at 64% and JMRI at 31% came out of the same method, on the same day, from the same tool. Averaging them would have produced one tidier figure and destroyed the only interesting thing about them, which is that they disagree. Java’s verbosity and build tooling change what a single turn costs in that codebase, which changes how much a tool that reduces turns can save there.

So the honest form of a savings number is not a percentage. It’s a percentage with a codebase attached, and a task attached, and a note about how many times it was run. A saving is a property of a tool interacting with a particular repository on a particular kind of work, never a property of the tool by itself. If you run this on your own codebase and get something different, that isn’t a contradiction. It’s the number behaving the way the quantity actually behaves.