Using AI to Save on AI: A Cache Experiment at 6.53 Million Calls

Developer Thomson Nguy has described an experiment in which his FinOps agent Tycho, running on Opus 5.5, cut the projected cost of a single scoring job from $6,541 to about $600. That happened four days after Tycho was put in charge of AI spending. The job projected roughly 6.53 million calls, each sending about 2,800 tokens of identical prompting instructions.

The first probe: the cache was not working

Haiku had been the original choice for the task, but Tycho tested a DeepSeek route. His first probe sent four requests with an identical prefix, and all four reported zero cached tokens. Nothing errored; every call was simply billed at the ordinary rate. The cause lay in the routing. Fireworks' prompt cache is replica-local, so repeating the same instructions does not reuse a cache entry when the next request lands on a different replica.

Tycho changed the routing so subsequent requests returned to the same replica, then ran the probe again. Across five warm calls, roughly 2,812 of 2,960 input tokens per call were cached. An eight-request batch reported 19,688 cached tokens out of 23,677 input tokens, a result consistent with one cold prefix and seven warm hits. It also answered the question that mattered: cache reuse survived the batch route in the small test.

What it did to the projected bill

Four projections for the same DeepSeek scoring job: ordinary calls with no cache, $6,541; batch with no cache, $3,271; ordinary calls with the cache working, $1,105; and batch with the cache working, about $600. The model and the 6.53-million-call workload stayed the same across all four estimates. Only the way the calls were served changed.

The Haiku line of roughly $2,800 comes from an earlier setup with a longer rubric and is a reference point rather than a controlled four-way comparison. In his notes the author adds that Haiku 4.5 was the original scoring model before production moved to Gemini in June, that a 300-atom probe recorded 98.6% cache hits, and that 43,747 Haiku-scored atoms averaged $0.000426 per atom.

What the numbers do not say

The author is explicit that the dollar figures are full-job projections extrapolated from small probes, not a completed 6.53-million-call invoice: the job has not been run or billed at that size. The check he recommends before scaling a repeated-prompt workload is to send identical prefixes, inspect the cached-token count, repeat with session affinity, and then test the batch route separately. A price card tells you the nominal token rate; these probes changed his full-job projection under each routing choice.

He also frames the result with some irony. An expensive frontier model was put in charge of finding out where the company could spend less on AI. Anthropic and OpenAI are offering frontier intelligence, but that very intelligence can now be used to figure out how to spend less money on the frontier labs.