About this video
Every step an AI agent takes, it reads its whole history again. In one test from this new paper, re-reading cached history was 82% of the agent's cost. The usual fix, trimming or summarizing old messages, can make the bill bigger and the agent less accurate. A team at Tsinghua University and CUHK found another way: keep the text, and switch off parts of the model's memory instead.
Would you let your agent decide what to forget?
We read "Cache Engineering for Agents" page by page: how agents loop, what the KV cache is and why prefix caching makes cached input cheap; why context engineering (trim, summarize) forces a costly re-prefill (trimming to the latest 12 tool results cost 1.85x full context; a 48k summary saved ~7% but lost 5 points); the three cache operations, append, drop and reposition (RoPE keys turned back, values untouched); the tool-response rolling drop (keep all reasoning and the latest 12 tool results); history dependence (same tokens, different cache states, so prefix matching is insufficient) and the AKV serving engine (a stateless API with a cache field, an extended radix tree, drop-aware eviction, segment-batch attention); the results across MiniMax-M2.7, GPT-OSS-120B and MiroThinker-1.7-Mini on BrowseComp-Plus, GAIA-Text, SWE-bench Verified and xbench-DeepSearch (cost -27.7% vs threshold summarization, -21.9% vs a low threshold, +1.7 points; SWE-bench Verified 40.0% at 20.0c to 43.2% at 9.7c; 24% / 32% / 37% on histories over 96k tokens; 1.36x mean and 1.51x P95 latency); agents that improve their own cache policies (7.1% cheaper than the expert design) and cache control as a tool; what the paper does not show; and four things to do if you build agents.
Chapters
0:00 Your agent re-reads its whole past
0:58 What an agent carries
2:10 Why forgetting backfires
3:31 Idea 1: Drop the notes, keep the words
4:56 Idea 2: Reposition, so the count never runs out
6:13 Idea 3: Same words, different memory
7:47 Idea 4: Cheaper, and more accurate
9:27 Idea 5: The agent designs its own forgetting
10:26 What this paper does not show
11:24 What to do about it
12:28 Recap
Paper: Cache Engineering for Agents. Tianyu Fu, Ruoxi Wang, Zihan Min, Zhongqi Ren, Zhihao Li, Tengxuan Liu, Shengxu Mu, Yu Wang (Tsinghua University), Hanling Zhang (The Chinese University of Hong Kong). arXiv:2610.28577, October 2026 (preprint, not yet peer reviewed). https://arxiv.org/abs/2610.28577
Code: https://github.com/thu-nics/AKV
The paper has no open licence: its charts are redrawn as our own animations from the published numbers, and the paper appears only as short credited excerpts (title block, one abstract sentence, the Limitations sentence) for commentary. Costs in the paper are API-equivalent estimates from public token prices.
Music: Voice Over Under by Kevin MacLeod, CC BY 3.0, via Wikimedia Commons (https://commons.wikimedia.org/wiki/File:Voice_Over_Under_(ISRC_USUAN1600001).mp3)
Pranav Innovates: AI research, explained. Subscribe to stay up to date.
YouTube: https://www.youtube.com/@PranavInnovates
Instagram: https://www.instagram.com/pranavinnovates
What the video says
Every time an AI agent takes a step, it reads its entire past again. In one test from this paper, eighty two percent of the agent's bill was just that: re-reading its own history. The usual fix is to make the agent forget. Cut old messages, or squash them into a summary. But here's the twist. Forgetting that way can make the bill bigger, and the agent less accurate. A team at Tsinghua University and the Chinese University of Hong Kong found another way to forget. Don't touch the text at all. Reach into the model's memory, and switch off the parts it doesn't need. Across three models and four benchmarks, their agents got about twenty eight percent cheaper, and a little more accurate. And stay to the end, where the agent writes its own forgetting rules, and finds cheaper ones than the researchers did. The paper is called Cache Engineering for Agents. Let's read it, page by page. First, how an agent works. A language model sits in a loop. It thinks, calls a tool, like a web search or a terminal, and reads what comes back. Then it goes again. Hard tasks can run for hundreds of rounds. Every round, the whole history goes back into the model. The task, every thought, every tool result. And the history only grows. So why isn't that impossibly slow? Because of the KV cache. As the model reads each word, it writes a small note for it, a key and a value. Later words look up those notes, instead of reading the old words again. Servers keep these notes between rounds. If the start of the history hasn't changed, its notes are reused. That's prefix caching, and it's why API providers charge less for cached input. But reuse isn't free. Every new word still has to look through all those notes. In the paper's cost breakdown, with the open model GPT OSS one twenty B on a deep research benchmark called BrowseComp Plus, reading cached notes was eleven of the thirteen cents per task. Eighty two percent. And the history keeps growing, past what the model handles well, and eventually past its limit. So agents use context engineering. They edit the history: trim old messages, or replace them with a summary. Here's the catch. Each note depends on every word before it. Edit something early, and every note after that edit is now wrong. The model has to read everything after it again, from scratch. That's called re-prefill. So the earlier you cut, the more you pay. That's why common agent tools, the paper cites Pi and OpenAI's Codex, wait for a size limit, then summarize almost everything at once. The paper measured both, with GPT OSS on a hundred BrowseComp tasks. Trimming, keeping only the latest twelve tool results, cost one point eight five times as much as never forgetting at all. Summarizing at forty eight thousand tokens saved about seven percent, but lost five points of accuracy. Cut early, and you pay. Cut late, and you lose information. So here's a question. How could an agent forget something, without changing a single word of its history? Think about it. The answer: forget the notes, not the words. The paper calls it cache engineering. The text history stays exactly as it is. Instead, the agent works on its cache directly, with three operations. Append is the normal one: add notes for new words. Drop switches off chosen notes, so later words can't look at them. Nothing is recomputed. Every note that stays is reused as it is. The third, reposition, is idea two. Picture an agent whose history is full of café hours, ticket prices and bus routes. Now you ask: when did the museum open? With drop, the café and bus notes are switched off. The useful search results stay, and nothing is read again. There's a bonus. The notes you keep were written while the dropped words were still visible. So they've already soaked up some of what you dropped. When you trim text instead, the re-read notes never see the trimmed part at all. So which notes should go? The paper looked at where the model's attention actually goes. Older words get less of it. And tool results, like raw web pages, get less than the agent's own reasoning. That gives a simple policy, a rolling drop. Keep all the reasoning. Keep the latest twelve tool results. Drop older tool results as new ones arrive. In that same test, it was more accurate and cheaper than both kinds of editing. Dropping has one problem. Every word has a position number: first, second, three thousandth. Dropping notes doesn't change those numbers. New words keep counting up, and can hit the highest position the model handles, even with a small cache. Models like these mark position with RoPE, rotary position embedding. Each key is turned by an angle that depends on its position. Word five is turned a little. Word five thousand, a lot. So reposition simply turns the kept keys back. The notes that remain are renumbered zero, one, two, with no gaps. The values aren't touched, and nothing is recomputed. It's a little cheap math on the keys. Does it matter? In the paper's ablation, rolling drop without reposition scored fifty five percent on BrowseComp. With reposition, sixty two. Without it, runs could hit the position limit, and stop. They also tried dropping the reasoning instead of the tool results. Worse, on both benchmarks. On SWE-bench Verified, a coding test, dropping old tool results scored forty one. Dropping reasoning, thirty six and a half. The agent's own thoughts are worth keeping. Now the hard part: serving it. Today's servers reuse notes by matching text. Same start of history, same notes. Cache engineering breaks that rule. Three agents. Each ends with the same two messages still in its cache, B and C. The first wrote C, then dropped A. The second dropped A, then wrote C. The third never had A. Is C's note the same in all three? Take a guess. No. All three are different. In the first, C was written while A was still visible. In the second, A was gone, but B had already soaked some of it up. In the third, no trace at all. Identical words, three different memories. A server that matches text alone would hand over the wrong notes. In the paper's words, prefix matching is insufficient. So the team built a serving engine called AKV, on top of SGLang. To the agent, it looks like a normal API. You send your messages as usual, plus a small cache field: drop these messages here, reposition there. Inside, it keeps a tree of histories where drops and repositions are branches too, so notes are reused only when their whole history matches. When memory is tight, dropped notes are the first to go. And if notes were cleared, it rebuilds them all in a single pass, with a trick called segment batch attention. So the API stays simple, like today's, and the server does the bookkeeping. The test. Three open models: MiniMax M two point seven, GPT OSS one twenty B, and MiroThinker. Four benchmarks: three for deep research, and SWE-bench Verified for coding. The rivals: summarizing at a high limit, as most agent tools do, and at a low one. Averaged over everything, cost fell twenty eight percent against the usual setup, and twenty two percent against the low limit. And the score went up, by one point seven points average. on Here's my favourite row. GPT OSS on SWE-bench Verified. The usual setup solved forty percent of tasks, at twenty cents a task. Cache engineering solved forty three point two percent, at nine point seven cents. Better, for half the price. Where does the accuracy come from? Long tasks. On short ones, under forty eight thousand tokens, all three methods scored eighty eight percent. Past ninety six thousand, the usual setup fell to twenty four percent, the low limit to thirty two, and drop held rolling seven. thirty That fits the story. A summary is a sudden cut, and things get lost. Rolling drop forgets a little at a time. And on real hardware, eight GPUs replaying long coding sessions, each method at its best setting: requests came back one point three six times faster on average, and the slowest five percent came back one point five times faster. Push to sixteen sessions at once, and the usual setup's slow requests took two hundred and thirty seconds. AKV's took thirty one. Because these operations are just a few fields in a request, an agent can program them itself. So the team asked: can an agent design a better forgetting policy than they did? They let MiniMax refine cache policies on twenty BrowseComp tasks, then tested them on a hundred tasks it had never seen. Within six rounds, it found policies that kept the usual setup's accuracy at fifteen to eighteen percent lower cost. That's about seven percent cheaper than the researchers' own design. Then they went further, and gave the agent cache control as a tool, to call whenever it decides. GPT OSS jumped from fifty two to sixty one percent on a BrowseComp set, and got cheaper. MiniMax went from fifty six to sixty four, but its cost rose a little, because it kept working for more rounds. That's a quiet shift. Memory used to be the serving system's job. Here, the agent manages its own. Now the catches. One. The costs are estimates, from each model's public token prices, not real bills. Reposition is charged like a cached read, which the authors say overstates its cost. Two. You need your own serving engine. As far as I know, the big closed APIs don't let you drop cache entries today. AKV's code is public, and built on SGLang. Three. The gains are modest, and not everywhere. On BrowseComp, MiniMax scored higher with the low summary limit, and on coding it dipped slightly. A smaller model, GPT OSS twenty B, gained less, and there the low limit was slightly cheaper. Four. The speed tests replay recorded sessions, with tool results played back rather than run live. It's a preprint, not yet peer reviewed. And training agents to work this way is left for future work. So what do you do if you build agents? Four things, then my take. One. Look at your bill. If cached input is most of your agent's cost, that's history being re-read every round, and that's where the savings are. That one's my suggestion, not the paper's. Two. Treat your agent's history as append only. Every edit near the start throws away the cache after it. If you must compact, do it rarely, and in one go. Three. When you cut, cut old tool results first, and keep the reasoning. In this paper, that beat dropping reasoning on both benchmarks they tried. Four. If you serve your own open models, try AKV. Rolling drop, keeping the last twelve tool results, is a strong default to start from. And my take. We spent two years teaching agents to remember more. This paper suggests the next win is forgetting better, and the agent itself may be the best judge of what to forget. Quick recap. Agents re-read their whole history every round, and in one test, cached reads were eighty two percent of the bill. Editing the text forces a costly re-read. Cache engineering drops notes instead, renumbers what stays, and needs a server that tracks history, not just text. The result: about twenty eight percent cheaper, a little more accurate, and an agent that can learn its own forgetting rules. You've just seen an agent design its own forgetting. So here's my question for you. Would you let your agent decide what to forget? Tell me in the comments. The paper is linked below.