Speaker A
Seven seconds. That is how long a big model spends reading a long conversation before it writes one new word. Nvidia measured that. Then they did the same job in 278 milliseconds, 25 times faster. And here is the part that should not work. The second model never read the conversation. It borrowed the first model's reading, the notes, the memory, whatever you want to call the KV cache, which was supposed to be impossible. A cache belongs to the model that built it. Nothing else can read it. Nvidia's new paper says that wall is thinner than it looks. On the good pairs, one matrix multiply walks through it. Four of the six model pairs kept between 73 and 98% of the receiver's own accuracy. Two of them fell apart completely. And the reason those two failed is the most useful thing in the paper. So, what is actually inside a KV cache? Why can no other model read it? And what did Nvidia strip out to move it? Three tricks, six pairs, two failures. Start with the seven seconds because that number is why this work exists. When you send a prompt, the model reads all of it at once in a single pass before it generates anything. That pass is called prefill, and its only product is the KV cache. For every token you sent at every layer, the model writes down two vectors. A key, which is a label it will search for later, and a value, which is the content it pulls when that label matches. Generation then runs off those notes. Each new word looks back across the keys, decides what deserves attention, and pulls the matching values. Without the cache, the model would reread your whole conversation for every single word it writes. So, prefill is the expensive half. And it gets worse in two directions at once. Longer prompt, more work; bigger model, more work. An agent 20 turns into a session is dragging an enormous prompt behind it every time it speaks. Which is why prompt caching exists. Anthropic's own pricing page builds a cache read at one-tenth of the base input rate. Send the same prefix again, pay 10%. That is the largest standing discount in this business, and teams do strange things to earn it, freezing prefixes bite for bite, so one cache keeps matching. But that discount has a hard edge. The cache only works on the model that produced it. Those keys and values came out of that model's weights, at that model's layer count, in that model's geometry. To any other model, they are noise, and that edge sits exactly where production is heading. Routers now move traffic between models mid-conversation. A small model for the easy turns, a bigger one when the question gets hard. Cheaper waits again when it softens. The moment traffic moves, the cache is dead. The receiver rereads everything from scratch at full price. Nvidia measured that penalty on their own hardware. Seven seconds for the 32B model at 32,000 tokens. Eleven to twelve seconds for the 70B. Here's where the paper turns. If prefill's only output is the cache, then skipping prefill is not a computation problem. It is a translation problem. Take one model's cache, rewrite it in the format the other one expects. So the first question was whether that translation has any structure worth exploiting or whether it is noise. Take Young Ho's team at Nvidia tested it in the dullest way available. Plain linear regression, one source layer, one target layer. Going from QN 314B up to 32B, a single layer's linear fit reconstructed 56% of the variance in the big model's keys, which means more than half of what a 32 billion parameter model would have written down was already sitting in the 14 billion one's notes, reachable by multiplying by a matrix. The two models do not have the same number of layers, so there is no natural pairing between them. The fix: for each target layer, rank every source layer by how well it predicts that layer. Then feed the best eight in together. That takes keys to 79% and values to 65%. The team's own ablation says this cross-layer selection carries more weight than either of the other two tricks. Second trick, and this one is elegant: keys carry a position rotation baked into them. RoPE, the thing that tells the model where in the sequence a token sat. The same word at position 10 and position 10,000 comes out looking different. So they unwind the rotation before fitting. Learn the map in position-free space, then reapply the receiver's own rotation at inference. The fit stops caring about length, which is why a map learned on thousand-token samples still works at 32,000. Third, the map itself is ridge regression fitted independently for every layer in every head, closed form, a solve, not a training run. The calibration set is 500 web documents of 1,000 tokens each. That is the whole input. No gradient descent, no back propagation, about an hour on a single 8-GPU node. And the map for that pair is finished. At inference, the mapper replaces the receiver's entire transformer body with one batched matrix multiply per layer. That is the whole reason it is fast. There is almost nothing to it. They tested six pairs across three families: Qin 3, Llama 3.1, and Mistral's Mistral 3. Every pair stays inside one family with the small model handing off to the big one and the big one handing back. Best case QN 314B into 32B, 97.6% of the receiver's own accuracy averaged across five benchmarks on HellaSwag. It came in at 101%, marginally better than the model doing its own reading. The most extreme jump they tried was Llama 3.1 8B handing off to 70B, nearly nine times the parameters. That one still held 73%. The latency table is the part I would pin to the wall. 32,000 tokens small to large, 278 milliseconds for the mapper against 6,975 for reading it again. Across 70 measured cells, seven pairs by 10 context lengths, the mapper won everyone. Modest at short prompts where a fixed 14 millisecond floor dominates. Enormous at long ones because the mapper barely notices length and prefill grows with it. They also check the case. This is actually for 100 CoQA conversations, roughly 15 turns each. Models alternating turn by turn drift over 10 turns stayed under two points going up and about a third of a point per turn coming back down. So here's a question worth answering before the paper answers it for you. If your router could skip a seven-second reread but the receiving model kept 97% of its accuracy instead of all of it, would you take that trade? Keep your answer because two of the six pairs did not get 97%. Both Mistral pairs feeding the 14B model landed at 44 and 42% average retention. Grade school math shows what that means. 95.6% of the arithmetic retained on the best pair, 1.6% on the worst. It did not degrade. It was gone. And the obvious explanation is wrong, which is the best thing in this paper. You would expect the broken pairs to be the ones where the regression fit badly. They are not. Llama 8B into 70B fits with an R squared of 0.84, 84, and keeps 94% going up and 37% coming back down. Mistral 3B into 8B fits at exactly the same 0.84 and keeps 93% both ways. Same fit quality, opposite outcomes, which means the size of the mapper's error is not what decides this. Where the error lands is. Attention does not weigh every direction equally. It scores keys against the model's queries, and most directions in that space are ones the queries barely look at. Error that lands there is invisible. Error that lands where attention is reading corrupts the answer. Ho's team measured that directly. Cosine similarity between the attention output computed from the mapped cache and from the real one. That predicts retention at a correlation of plus 0.57. The calibration R squared, the number you would naturally trust, lands at us 0.2 backwards. The metric that looks like it should screen your model pairs is the one that misleads you. The proof is what happens when they swap the linear map for a small neural network. Same data, same pipeline, only the shape of the mapper changes. The worst pair goes from 59% on HellaSwag to 95 and a half, plus 37 points. And on the pair
