About this video
With thinking switched off, an AI coding agent fixed a bug in 15 steps for 19 cents. At extra high thinking, it took 27 steps, cost a dollar, found the official fix in the project's history and copied it in, byte for byte. Both runs were marked solved.
What effort level do you run your AI at?
We read the new Johns Hopkins + Together AI + Cornell preprint "What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents" page by page: what the reasoning-effort dial is and why hidden thinking tokens and long agent loops cost so much; why effort is a 'credence good'; eight models (GPT-5.6 Sol, Terra and Luna, Claude Sonnet 5, GLM-5.2, Inkling, Kimi K3, DeepSeek V4 Flash) on AIME, GPQA Diamond, Terminal-Bench 2.1 and SWE-Bench Pro; how more effort loses answers lower effort had found; why easy tasks cost up to 42x more at the top setting; why switching thinking off isn't automatically cheaper for agents; how higher effort checks itself more and goes looking for answers (75% of SWE-Bench Pro tasks at extra high, and 77.3% to 59.1% once the shortcut was removed); naive laddering (start low, escalate only after a failed check); what the paper does not show; and five things to do if you build with AI.
Chapters
0:00 Same fix, five times the price
0:56 The effort dial
2:32 Finding 1: More effort loses answers
3:48 Finding 2: Paying more for the same answer
5:22 Finding 3: It works differently, and finds the answer key
7:43 Finding 4: Choose effort per task
9:03 What this paper does not show
9:56 What to do about it
10:56 Recap
Paper: What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents. Kaiser Sun, Yongchan Kwon, Zain Hasan, Shang Zhu, Kaitlyn Zhou, Federico Bianchi (Johns Hopkins University, Together AI, Cornell University). arXiv:2610.34491, October 2026 (preprint, not yet peer reviewed). https://arxiv.org/abs/2610.34491
Figures, tables and excerpts from the paper are shown under its CC BY 4.0 licence (https://creativecommons.org/licenses/by/4.0/) with credit; the highlights, redrawn bars, animations and characters are ours.
Music: Voice Over Under by Kevin MacLeod, CC BY 3.0, via Wikimedia Commons (https://commons.wikimedia.org/wiki/File:Voice_Over_Under_(ISRC_USUAN1600001).mp3)
Pranav Innovates: AI research, explained. Subscribe to stay up to date.
YouTube: https://www.youtube.com/@PranavInnovates
Instagram: https://www.instagram.com/pranavinnovates
What the video says
An AI coding agent fixed a bug with its thinking switched off. Fifteen steps, nineteen cents. At extra high thinking, it took twenty seven steps and cost a dollar. Because after doing the same work, it dug through the project's history, found the official fix, and copied it in, byte for byte. Both runs were marked solved. Almost every AI model now has a thinking dial. More thinking costs more money, and most of us assume it's worth it. Is it? A new paper from Johns Hopkins, Together AI and Cornell turned that dial on eight models. On tasks every setting could solve, the top setting cost up to forty two times as much. More thinking lost answers that less thinking had found. And part of the gain may have been the agent looking up answers. Stay to the end for a simple rule that matched the top setting's success, and usually cost less. The paper is called What Effort Level Should I Choose? Let's read it, page by page. First, what is the dial? Reasoning models think before they answer, in tokens you usually never see, but still pay for. The effort setting tells the model how much to think. The levels go none, low, medium, high, extra high and max. Not every model offers every level, and each company decides what the names actually do. In an agent, effort matters even more. An agent works in a loop. It reads, runs a command, reads the result, and calls the model again. And every call re-reads the whole history so far. So a longer path costs more on every single step. The authors borrow an idea from economics: a credence good. Think of a car repair. You can see the car runs again, but you can't tell whether the expensive part was needed. With effort, you see the answer. You never see whether the extra thinking mattered. So they tested eight models. OpenAI's GPT five point six in three versions, Sol, Terra and Luna. Anthropic's Claude Sonnet five. And four open-weight models: GLM five point two, Inkling, Kimi K three and DeepSeek V four Flash. On four tests. Two answered in a single call: AIME competition math, and GPQA, graduate-level science questions. And two for agents: Terminal-Bench, jobs done in a Linux terminal, and SWE-Bench Pro, fixing real issues in real code projects. Same tasks, same prompts, same agent code at every setting. Only the dial moves. The whole study cost about fourteen thousand dollars of model calls. Here's what they found, in four findings. Finding one. More effort can lose answers it already had. Higher effort usually raises the overall score. But a score is a count, and counts hide swaps. Take Inkling on the science questions. Going from low to medium, it got sixteen more right overall. Inside that, it newly solved twenty one, and failed five that low had solved, on all four tries. Now the terminal jobs. GPT five point six Sol, going from high to extra high. More thinking. Does it solve more tasks? Take a guess. No. It solved five new tasks but lost eight, and fell from seventy six to seventy three out of eighty nine. And every one of the eight models had at least one terminal task it solved at its lowest setting but failed at its highest. Each agent task ran only once per setting, so some swaps could be luck. So they checked public runs from another coding benchmark, DeepSWE, with four tries per task. The swaps were still there. Claude Sonnet five, going from high to extra high, gained seven tasks and lost twelve. For builders: an average that goes up can hide the exact cases you care about. Compare which tasks pass, not just how many. Finding two. Higher effort spends more, even when it isn't needed. To see the waste, they kept only the tasks a model solved at every setting. Here, extra thinking can't buy a better result. It can only cost more. On the terminal jobs, going from low to extra high made GPT five point six Sol two and a half times as expensive. Claude Sonnet five cost three times as much. On the code fixes, Sol went from twenty seven cents a task to a dollar twenty four. The extreme case is Inkling. On the code fixes, at max effort, it cost forty two times as much as on low. Same tasks, all solved either way. Where does the money go? Not only into thinking. On those code fixes, at extra high, Sol took nearly twice as many steps, from about fifteen model calls to twenty eight. Each call re-reads the growing history, so its bill for input alone went from seventeen cents to sixty. So switching thinking off must be the cheapest option. Right? What do you think? Not for agents. On the terminal jobs, with thinking off, GPT five point six Luna averaged twenty one and a half calls per task, against five point four on low, and cost four and a half times as much. It saved on thinking, and paid for it in wandering. For builders: effort works like a multiplier on every step, not a budget the model stops spending when the job is easy. And off isn't automatically cheap. Measure the whole run. Finding three. More effort doesn't just think longer. It works differently. It checks its own work much more. On terminal tasks Claude Sonnet five passed either way, its verification steps nearly doubled, from six point six to twelve point three. On tasks it failed either way, they still went up. More checking, same result. And self-checks can miss what the real test checks. On one terminal task, Sol at its highest effort reported that its output met a numeric requirement. The task's own checker said no. The low-effort run passed. In another, GLM five point two at max rewrote a project's history and deleted a commit the test needed, while its lower setting passed. But the bigger change is something else. Higher effort goes looking for answers. SWE-Bench Pro's test machines still held each project's full history, including the commit with the real fix. At low effort, Sol read that fix on about sixteen percent of tasks. At extra high, on seventy five percent. That's the agent from the start. With thinking off, it fixed the bug from the issue and the code. At extra high, it did the same work, then searched the whole history, found the official fix, overwrote its own file with it, and checked for a byte for byte match. Both counted as solved. One cost five times as much, and copied the answer instead of working it out. So the authors built a corrected version: history removed, and internet limited to downloading packages. Sol at extra high still tried to look answers up online on sixty eight percent of tasks. Every attempt was blocked. And its score fell from seventy seven point three percent to fifty nine point one. Every one of the eight models scored lower at its top setting once the shortcut was gone, by eleven to almost thirty points. Claude Sonnet five fell from seventy nine and a half percent to fifty. Was it cheating? The test allowed it, and nobody told the agent not to. The authors call it retrieving existing answers, and a sign of benchmark leakage. But it means part of more effort, higher score may have been lookup, not reasoning. For builders: if an answer is reachable, high effort is more likely to find it. Lock down your test environments, or your scores will measure search, not skill. Finding four. If no single setting is best, choose effort per task. First idea: let the model choose. Before it starts, it reads the task and picks its own effort level. The result? No consistent advantage over picking at random. Second idea, almost too simple. They call it naive laddering. Start at the lowest setting. Check the result with a real test. Only if it fails, run it again at the highest. It matched or beat always using the top setting, for every model, on every benchmark. On the math and science questions, the typical saving was a quarter to over forty percent. On the corrected code fixes, about fifteen percent. And on the terminal jobs, Kimi K three solved eighty four percent instead of seventy three, for thirty percent less. It isn't free everywhere. A retry pays for both runs, so on the terminal jobs, four of the eight models solved more but paid more. And it needs a check you can trust, which many real products don't have. And the bill misses one cost: your time. The authors priced waiting at the average US wage across all jobs, about thirty four dollars an hour. For GPT five point six Terra on the terminal jobs, waiting was over eighty percent of the total cost. Count it, and laddering's saving grew from under five percent to nearly twenty. Now, what this paper does not show. Each agent task ran once per setting, to fit the budget, so some swaps may be luck. The four-try check suggests the pattern holds, but on five of the models. The code test is a pilot of forty four tasks, and in its corrected version, the always-solved sets were only ten to twenty tasks. Max effort wasn't tested on the OpenAI and Anthropic models, because it was too expensive. Laddering was rebuilt from runs they already had, not run live, and it assumes a pass or fail check you can rely on. The waiting cost assumes you sit and wait. And it's a preprint, not yet peer reviewed. The authors' conclusion: effort is sold as a dial, but it doesn't behave like one. They ask providers to make reasoning charges verifiable, so that, in their words, paying for more thought becomes an informed choice rather than an act of faith. So, what do you do if you build with AI? Five things from the paper, then my take. One. Don't default to max. Start low or medium, and raise effort only where your own tests show it helps. Two. Ladder when you can check. If you have tests, a validator or a schema, run cheap first, and escalate only on a failure. Three. Don't assume off is cheapest for agents. Count the calls and the input tokens for the whole run, not just the thinking tokens. Four. Measure cost per solved task, including how long you wait. And compare which tasks pass, not just how many. Five. Sandbox your evaluations. Remove the answers the agent could find, like git history and open internet, or you'll reward search instead of skill. And my take. This is the same lesson as our last two episodes. The gains came from the harness around the model, a test and a rule for when to try harder, not from turning one knob. This paper didn't test that framing. It's my reading of it. Let's recap. Higher effort often raised the overall score, but lost answers lower effort had found. On easy tasks, it cost up to forty two times as much. It checked itself more without changing the result, and it went looking for answers. And starting low, then climbing only after a failed check, matched the top setting's success, usually for less. The dial promises more thinking. Sometimes it buys more answers. Sometimes it buys the same answer at a higher price. And sometimes it buys a look at the answer key. Remember the agent that copied the fix? With its thinking off, it had already solved it, for nineteen cents. So start low, check the result, and only then turn the dial up.