About this video
On a coding benchmark, a router that picks a model for each task stayed within half a point of always calling Claude Opus 4.6, at about 1/25 of the cost, and it never called Opus once. A new paper from NUS's HPC-AI Lab, with UC Berkeley and HKUST, ran 452 model setups and 374,401 scored runs across text, images, video, speech, code and robots to answer one question: when does picking a model per request actually pay off? The surprise: smart routing barely made answers better. The real win was price.
Which model do you send everything to, and which half of that traffic could a cheaper one handle?
We read "Route Anything" page by page: the floor (best fixed model) and the ceiling (the oracle, a magic router that picks the best candidate per request), and the 5.4-point gap between them; finding 1, the quality myth (a 107-type task tree closes only 8% of the gap, part of it is pure luck, and a trained 4B Qwen router beats the best fixed model by just 0.3 points across 16 benchmarks; robots cut policy time by 24%); finding 2, the real win is price (8 models and 9,999 coding tasks, one category clue and a no-quality-loss rule: 25.5x quality per dollar, Opus never called; 8.82x on average over code, text, images and robots); finding 3, prove your quality bar (the fitted rule held on test in only 12.5% of 200 splits, a certified check held in all 200 at 19.4x; what retries, compiler checks and AI judges are worth; and why cheaper is not faster); finding 4, the traps that fake a result (winner labels 42.8% vs outcome predictions 71.6% on robots, and 11 missing Opus 5.5 scores that turned a small loss into a fake +2.4-point win); what the paper does not show; and four things to do if you build with AI.
Chapters
0:00 It never called the expensive model
1:06 The floor and the ceiling of routing
2:25 Finding 1: The quality myth
4:05 Finding 2: The real win is price
6:25 Finding 3: Prove your quality bar
7:46 Finding 4: The traps that fake a result
9:13 What this paper does not show
10:16 What to do about it
11:39 Recap
Paper: Route Anything: A Study of Model and Policy Routing for General-Purpose AI Systems. Pengfei Zhou, Yixing Ma, Hexin Wang, Weidong Tang, Zhenglin Wan, Wangbo Zhao, Yang You (National University of Singapore HPC-AI Lab; InfRec, Cardinal AI Lab; UC Berkeley; HKUST). arXiv:2611.05275, November 2026 (preprint, not yet peer reviewed). https://arxiv.org/abs/2611.05275
The paper is licensed CC BY-NC-SA 4.0 (non-commercial): its tables and figures are redrawn as our own animations from the published numbers, and the paper appears only as short credited excerpts (title block, one line from the key findings, the conclusion's three rules) for commentary.
Music: Voice Over Under by Kevin MacLeod, CC BY 3.0, via Wikimedia Commons (https://commons.wikimedia.org/wiki/File:Voice_Over_Under_(ISRC_USUAN1600001).mp3)
Pranav Innovates: AI research, explained. Subscribe to stay up to date.
YouTube: https://www.youtube.com/@PranavInnovates
Instagram: https://www.instagram.com/pranavinnovates
What the video says
A router matched Claude Opus on a coding benchmark, within half a point, at about one twenty-fifth of the cost. And it never called Opus once. Most AI apps send every request to one big, expensive model. This paper picked a model for each task instead. The surprise? Smart routing barely made answers better. Where it shined was price, and the price gaps between models haven't gone anywhere. And stay for the trap where eleven missing scores turned a loss into a fake win. Four hundred fifty two model setups, three hundred seventy four thousand scored runs, across text, images, video, speech, code, and robots, all to answer one question. When does picking a model per request actually pay off? It's called Route Anything, from the National University of Singapore, with Berkeley and HKUST. Here's what it found. First, a router is a mailroom. Requests come in, and it sends each one to a candidate: a model, a tool, or a robot policy. In episodes seven and ten, we asked how long a model should think. This one asks which model should do the thinking. Every router is measured against two lines. The floor is the best fixed model: the single candidate that scores highest, used for everything. The ceiling is the oracle: a magic router that, for each request, picks whichever candidate actually did best. So here's a question. On average, how many points better is the magic router than the best single model? Think about it. About five points. Eighty two point six for the best fixed model, eighty eight for the oracle. In speech, under one point, because every voice model is nearly perfect. In video and robots, around nineteen and eighteen. That gap is the prize. Four findings: quality first, then in finding two, how a router matched Opus without ever calling it. Finding one, the quality myth: smart routing barely beats one model. And that's exactly why the price trick works. The obvious router sorts by task type: math to this model, images to that one. So the team built a tree of over a hundred task types, and picked the best candidate for each, even with hindsight. It closed eight percent of the gap. The other ninety two percent lives inside each type. Two requests that look alike can need different models. And some of that is pure luck: on one test where every question was answered three times, the oracle's edge vanished once each model was scored by its average. No router can predict a coin flip. So they trained a real router: a small, four billion parameter Qwen model that reads each request and picks a candidate. Across sixteen benchmarks, it beat the best fixed model by just three tenths of a point, because within a benchmark one model wins almost every type. It only shone where models really complement each other, like Terminal Bench: eight points on thirty six tasks. On image generation, none of thirty one router designs clearly beat the single best generator. Robots did a little better: matching each scene to the right policy cut running time by a quarter, though success barely moved. For builders: part of the gap you see is noise, and the rest is hard to catch. Routing's real win isn't quality. It's money. Finding two, the big one. Price is where routing shines. On a coding benchmark with eight models and nearly ten thousand tasks, always calling Claude Opus four point six costs about one point two cents per task. The router gets one small clue, which of twenty one categories a task belongs to, and one rule: don't lose quality compared with always calling Opus. Then, in each category, it picks the best quality per dollar. So here's my question. To keep up with Opus, how often did this router need to call Opus? Take a guess. Never. Not once. It sent every task to a cheaper model and stayed within half a point of Opus's quality, at about twenty five times the quality per dollar. And that clue doesn't need hand labels. Guessing the category from the prompt's own text worked almost as well. And yes, that's the previous Opus, so the exact numbers will age. The pattern won't. Compare that with the lazy version: always use the cheapest model. That's twenty times the efficiency too, but it loses over four points. Routing by category wins those points back. And with no clue about the task at all, the router had to keep calling Opus, and the gain fell to about three and a half times. Across four testbeds, code, text, images and robots, routing averaged almost nine times the efficiency, with seven times on text questions. Images saved less: twenty six percent, within half a point, because the best image model costs twenty one cents an image, several times the rest. Speech wasn't worth routing on price at all, because the best voice is already among the cheapest. For builders: routing pays when models are close in quality and far apart in price. But don't copy it yet. On fresh tasks, that no-loss rule broke most of the time. And in finding four, eleven missing scores fake a win. Finding three: how it broke, and the fix. The rule was fitted on training tasks. The team refit the router on half the tasks, two hundred times, then checked it on held-out test tasks. The quality bar held only twelve and a half percent of the time. So they added a check. Before trusting a cheaper router, test it on separate tasks, going from safest to cheapest, and stop at the first one that fails to prove it's as good. With that check, the bar held in all two hundred splits, at about nineteen times the efficiency. Extra steps have to pay their way too. When one model fails a task, the next model succeeds only about half as often as usual, because what's left is harder. A free compiler check gave four times the efficiency. An AI judge at its list price cost more than it saved. And cheaper is not faster. Opus answered in about four seconds, the cheaper model in about six. Measured in seconds instead of dollars, always calling Opus beat the cheaper routes. For builders: decide what you're paying for, dollars or waiting, before you route. One finding left: the traps that fake a result. Finding four, and the trap I promised. Before you trust any routing result, including finding two, check these. It starts with how you train a router. The natural label is the winner: for each training example, which candidate did best? Then train a classifier to name it. On robots, that classifier reached forty three percent success. A router trained the other way, predicting each policy's own chance of success, reached seventy two. Same data, same model. The reason: on most scenes, several policies succeed. Ties were broken by speed, so the label just said, the fastest one. Ninety four percent of the labels were set by that tie-break. The classifier learned to pick fast policies that often fail. The second trap is in the comparison itself. One baseline only kept models that had a score on every training example. Opus five point five was missing eleven out of more than a thousand, so that baseline quietly dropped it. That turned the router's small loss into a two point four point win that wasn't real. For builders: predict every model's outcome, not just the winner. And compare models on exactly the same inputs, or a few missing rows can fake your result. Now the catches. One. The biggest savings come from the code and text testbeds, which replay recorded answers with estimated token prices. And their reference models are older: Opus four point six and GPT four. Against a tougher yardstick, the best model per category, the code router's gain shrinks from twenty five times to about three. Two. The guarantee is on average. Even the checked router lost some quality in about eight of the twenty one categories. Three. Several quality gains, including the robot one, are within the error bars. And the check itself costs money: you need the reference model's answer on every test task. Four. Routing here is one choice per request. Long agent tasks, where each step could use a different model, are left for future work. And it's a preprint, not yet peer reviewed. So what do you do if you build with AI? Four things, then my take. One. Measure the prize first. Run your top few models on the same few hundred real requests, and compare the best per request with the best single model. If the gap is tiny, like speech here, skip the router. That one's my suggestion, built on the paper's method. Two. Route on price. Find where models are close in quality but far apart in cost, and give your router at least the task's category. Here, that alone gave most of the savings. Three. Set your quality bar before you look, and check it on fresh tasks. Then price every extra step, a judge, a retry, a forecast, against what it saves. And four: train on outcomes, not winners, and compare every model on exactly the same inputs. And my take. Everyone asks which model is the best. For most requests, this paper suggests a better question: what's the cheapest model that's good enough, and can you prove it? The smartest system isn't the one that always calls the biggest brain. It's the one that knows when it doesn't need to. Quick recap. A perfect router would beat the best single model by about five points, but most of that prize hides inside each task, and real routers won only a sliver of it. The real win is price. On code, within half a point of Opus at about one twenty-fifth of the cost, without calling Opus once. Prove your quality bar on fresh data, train on outcomes, not winners, and compare on the same inputs. So here's my question for you. Which model do you send everything to right now, and which half of that traffic could a cheaper one handle? Tell me in the comments. The paper is linked below.