Speaker A
Okay, I need to talk about a mistake I made because it cost me real money in real time and honestly a little bit of my dignity. A few months ago, I was deep in the rabbit hole. The one where it's
A creator explains why single-stream tokens-per-second benchmarks mislead local AI buyers, and shows how batching and agent workloads change GPU value.
Key Takeaways
- Single-stream tokens per second measures only one-user chat speed and can badly misrepresent agent or multi-user performance.
- Batching exploits idle compute and can multiply throughput several times on the same GPU at little extra cost.
- Evaluate hardware by aggregate throughput at your realistic concurrency, cost per token-per-second under load, and memory capacity for concurrent contexts.
- Check inference server configuration (parallel slots, batching settings) before concluding hardware is too slow.
What the video covers
- The creator bought local AI hardware based on single-stream tokens-per-second benchmarks, then found agentic multi-task workloads ran painfully slow.
- Single-stream token speed is memory-bandwidth-bound: each token requires reading all model weights, so tensor cores sit mostly idle (about 3.5% utilization on an RTX 3090 at 45 tok/s).
- Batching multiple requests reuses the weights already read from memory, so aggregate throughput scales dramatically; one RTX 3090 measured 45 tok/s with one request and 298 tok/s with eight.
- Cost per token-per-second drops sharply under batching, changing which hardware looks like the best value.
- Agent workflows with parallel sub-agents naturally create concurrent requests, making single-stream benchmarks almost irrelevant for that use case.
- Head-to-head comparisons show the rankings can flip: a Mac that wins at one stream may lose badly to an NVIDIA card under concurrent load.
- Software defaults matter: some inference servers run only one parallel slot unless configured, wasting available hardware.
- Memory capacity is reframed as buying concurrent 'seats' with working context, not just bigger models.
- The creator suggests changing default settings, counting actual concurrent streams, and benchmarking real workflows instead of single prompts.
Chapters
- 00:00The costly hardware mistake
- 01:39Why single-stream tokens per second is seductive
- 03:20Memory bandwidth and the math behind single-stream speed
- 05:00Compute sitting idle: the 96% problem
- 06:30Batching and measured throughput on an RTX 3090
- 09:00Agents, concurrency, and cost per token-per-second
- 11:00Mac vs NVIDIA under concurrent load
- 13:00Software defaults and what to do next
Full Transcript — Download SRT & Markdown
Speaker A
11:00 a.m. and there are 17 browser tabs open with one GPU benchmark video after another. I watched Tech with Tim Locally and Manolo Remedi and I had my notepad and my spreadsheet and every single guide pointed me to the same number
Speaker A
which was tokens per second. The RTX 5090 does 75 tokens per second and gets called the king. The RTX3090 does 40 and gets called respectable. And the DJX Spark does 12 to 17, which comes across as disappointingly slow. So, I did what
Speaker A
any reasonable person with a dangerous amount of curiosity and a slightly unreasonable budget would do. And I made my decision based on that number, fully confident and with zero doubts. Then I started using my local AI setup for real
Speaker A
agentic work, running code with sub agents and kicking off several coding tasks at once. And I sat there watching it crawl. I'd bought the right specs and watched every video. And the benchmark said this was fast. So what was
Speaker A
happening? And that's when it hit me that the problem wasn't the GPU or the benchmark because I had been measuring the wrong thing. And it wasn't just me.
Speaker A
Since pretty much every buying guide, every YouTube video, and every Reddit thread I'd read was measuring the wrong thing, too. Tokens per second is a one chat number, which tells us how fast the model talks to one person at a time. And
Speaker A
if that's all we're doing, casual chatting with one prompt and one response, it's completely fine advice.
Speaker A
But the moment we start running agents in cloud code or any setup where several tasks fire at once, that number becomes almost meaningless. And the boxes that look slow at one stream can absolutely smoke the ones that look fast. So, I'm
Speaker A
going to show you exactly what I mean and walk through the math without turning this into a lecture. And by the end of this, I think we're going to look at GPU benchmarks completely differently. Now, why is the single
Speaker A
stream number so seductive? It isn't a stupid metric, and it actually makes a lot of intuitive sense. So, I want to give credit where it's due before I tear it apart. When we type a prompt, the model generates one token at a time. So,
Speaker A
we're literally watching the words appear on screen, and there's a real human experience to that. Tech with Tim has nearly 200,000 views on his hardware guide. And his rule of thumb is that 50 or 60 tokens a second feels almost
Speaker A
instant. 35 to 40 is still totally usable. And somewhere around 10, it starts getting frustrating. And I felt that frustration personally, watching a model think at the pace of a tired toddler sounding out words. So the single stream metric does capture
Speaker A
something real. The math behind it is also elegant. For every single token the model generates, the chip has to read the entire model's weights out of memory, the whole thing, every time. So if we divide the memory bandwidth by the
Speaker A
size of the model, we get a pretty accurate ceiling on how fast it can run.
Speaker A
A used RTX3090 moves about 936 GB of data per second. And Quinn 3.827B, 827B, a dense model that most of these guides test, weighs in at about 17 GB at 4-bit precision. So 936 / 17 gives us a
Speaker A
ceiling of around 55 tokens per second. And in practice with real inference engines, we land around 40 to 46. That's a really useful formula where you check the bandwidth and the model size and know roughly what you're getting. Tim
Speaker A
calls bandwidth and capacity the two specs that decide everything. And for a single chat window, he's not wrong. The problem is what that same formula quietly tells us about the rest of the hardware. And most people, myself included, never followed that thread far
Speaker A
enough. While the memory bandwidth is busy being the bottleneck, something else is sitting there doing almost nothing. And once we understand what that is, the whole picture flips. Now, this is the part I wish someone had explained to me before I bought
Speaker A
anything. An RTX3090 can do about 71 trillion math operations per second, which is the official Nvidia figure for its tensor cores at 16 bit. And that's a really insane number. So why does it only push out 45 tokens a second on a
Speaker A
single chat? You'd assume all that computation is what's taking so long. But the bottleneck is really the memory.
Speaker A
So for each token, the car drags those 17 GB of model weights from memory into the cores. The cores do their roughly 54 billion operations on those weights and finish almost instantly and then they just sit there waiting for the next 17
Speaker A
GB to show up. At 45 tokens a second, that works out to about 2.5 trillion operations per second being used out of 71 available, which is approximately 3.5% of the available math doing actual work, while the other 96.5% sits idle
Speaker A
waiting on the memory bus. When I first read this, I laughed out loud because I'd been worrying about whether I bought a fast enough GPU, and the card is basically sitting in a beach chair sipping a drink 96 times out of 100.
Speaker A
It's like hiring a worldass chef and keeping him waiting because you can only walk one tomato at a time from the fridge. So, what if instead of making the chip read the same 17 GB again for the next conversation, we stacked
Speaker A
several conversations on the same trip? If a second user needs a token at the exact same moment, the card multiplies the weights it already hold out against both conversations at once and then four conversations and then 64 because the
Speaker A
weights cross from memory once per step no matter how many conversations are writing along. Engineers call this batching and the version that runs on real servers where requests can join and leave mid answer is called continuous batching. One stream is a bus with one
Speaker A
passenger and batching fills the bus which was already going anyway. Now, I'm not just theorizing here because someone actually measured this on a card most of the local AI community already owns, the RTX3090.
Speaker A
There's a GitHub repo called Hyper Quen from a user called SYVAI where they ran Quen 3.827B on VLM, a proper batching inference engine on a single RTX3090 capped at 250 W. It was the same card, model, and engine, and they only changed how many
Speaker A
simultaneous requests it was handling. With one request, it did 45.4 tokens per second, which is exactly what the buying guides say. And with two requests, it did 81.8, already almost double. And with four, it did 153.8, which is three
Speaker A
times the output on the same card. And with eight requests, it did 298.4, 6 1/2 times. And at 64 requests, it hit 1 35 tokens per second. That's on a used RTX3090, a card you can buy right now
Speaker A
for around 1,500 bucks. Now, to be fair, each stream does slow down as we add more. And at 8 requests, each one gets a little under 40 tokens per second, but that's still faster than anyone actually reads. And what really changed is that
Speaker A
the total output of the card exploded because it stopped wasting 96% of its compute. And on cost per token per second, which is how the buying guides measure value, a 3090 at one stream works out to about $33 at current
Speaker A
prices. At eight streams, it's about $5. So, it's the same card at the same price with 6 1/2 times the value just waiting to be unlocked. And this is where I had a small personal crisis. I'd been evaluating hardware by the one stream
Speaker A
number like everyone else. And I was essentially leaving most of what I owned on the table. It wasn't because I had the wrong hardware, but because I was using it wrong. Or rather, because the tools I'd set up were using it wrong by
Speaker A
default. We'll get to that in a minute. But first, why does this matter so much right now? Because one specific thing changed about how most of us actually use AI, and that thing is agents. Not long ago, most people use local AI the
Speaker A
way they use chat GPT, where you type a question, get an answer, and read it one at a time. And if that's you, nothing I'm saying today really changes your buying decision. But the way serious developers use AI has fundamentally
Speaker A
shifted. Artificial analysis, one of the most respected AI benchmarking organizations out there, actually stopped ranking models by tokens per second in November. They announced it publicly. And their reason was that as work gets more agentic, the time to
Speaker A
finish a task matters more than speed per token. And that's the real change because we don't sit there watching the token stream by anymore. We tell an agent to handle something and go do other work while it runs. So the measure
Speaker A
we care about is how long it takes until the task is done. Now Claude Code Enthropic's coding agent can run up to 20 sub aents simultaneously in a single session before it refuses to spawn another. And that's the documented
Speaker A
default. Its own documentation shows prompts that explicitly split research work across separate sub aents running in parallel. And it has background sub aents that keep going while you're doing something else. And one developer I read about on Local Lama had two RTX3090s
Speaker A
running VLLM. And he moved his Open Code setup from running sub agents one after another to five at once. He reported 2.43 times the real throughput on the same two cards with the same models. And the only thing he changed was the
Speaker A
concurrency. And Manolo Remedi, whose video I watched maybe four times when I was buying, plans to run two or three 200,000 token contexts simultaneously, one for Hermes and one for Open Code.
Speaker A
And Revolute, the fintech company, made self-hosted AI the default for about 2,000 engineers. Their infrastructure lead said people run tasks in parallel and check in when the task is done because they've stopped watching the token stream by. His exact line was that
Speaker A
to the server, one developer running four sub agents is four users. That sentence reframed everything for me.
Speaker A
When I kick off a research task with four parallel agents, I'm four people using one GPU. And if that GPU is set up to serve one at a time, three of my agents are waiting in line while one
Speaker A
runs. Now, let's make this concrete in the way that actually matters, which is time. So, say an agent splits a job into eight subtasks of about 2,000 tokens each. A fairly normal chunk of code, analysis, or research. So, that's 16,000
Speaker A
tokens to generate. On a 3090, running them one at a time at 45.4 tokens per second, that takes 352 seconds, or about 6 minutes. While on the same car, running all eight at once at 298 combined tokens per second, it takes 54
Speaker A
seconds. So, with the same hardware, model, and tokens, it finishes 6 and 1/2 times faster. The spec sheet didn't change, and the only thing that changed was whether the card was allowed to do its actual job. Now, you might be
Speaker A
wondering whether that only holds for the 3090. And what about the DGX Spark? The little box Nvidia sells, which every guide puts near the bottom because of its low single stream speed. Tim's guide showed the Spark at 12 to 17 tokens per
Speaker A
second, near the bottom of the chart. And on the single stream metric, that's accurate. But MIAI Lab actually tested it properly with batching, setting up SG lang for Quen 3.827B on a DGX Spark, and they got 56.6 tokens per second for one
Speaker A
stream, which is already way better than the stock engine. And at eight streams, each one still gets 30.8 tokens per second. So, all eight finish in about 65 seconds, which against the RTX3090's 54 seconds at eight streams makes the Spark
Speaker A
actually competitive even though it was supposedly the slow one. So, the guy had it near the bottom and the real task finished in about a minute on the same box because we asked it a different question. And then there's the benchmark
Speaker A
Alex this kind RAM between the Mac Studio and the DJX Sparks. That's where things get really interesting and a little uncomfortable for me because I'd already formed some strong opinions about those Mac Studio numbers. And Zeiss did something most reviewers
Speaker A
don't, which is that he actually tested multi-user performance and not just single stream speed. And what he found should be required reading for anyone spending this kind of money. He put a 256 GB M5 Ultraax Studio against two DGX
Speaker A
Sparks cabled together with the same total memory and the same models on both sides. And he started at one user and then scaled up with one user and a short prompt on DeepSeek V4 Flash. It was 38 tokens a second on the Mac and 38 on the
Speaker A
Sparks, a tie. On Quinn, the Mac actually won 45 to 38. So, at this point, every buying guide is right because the Mac is faster per dollar and faster per stream. And then he added users. On Deepseek, the MAC peaked at
Speaker A
four users with 66 tokens per second combined and then fell to 46 at 8 while the Sparks kept climbing to 70. And on Quinn, at 8 users, the Sparks put out 120 tokens per second combined, while the Mac did 66 to 70. So, the box that
Speaker A
was faster at one user got outrun at 8. But it gets worse. He gave each of those eight users 8,000 tokens of chat history, which by the way is a pretty modest context for an agent. And the max
Speaker A
combined output fell to about 11 tokens per second in total, where each person waited over a minute just for the first word. And the sparks took about 24 seconds to the first word and then ran at seven tokens per second per user. And
Speaker A
his verdict was that the Mac is a machine for one person, maybe two. And I think that's exactly right. And it hit me because I've been evaluating this the same wrong way. Under his video, one viewer literally filed a gent
Speaker A
development under single user, single stream, and said it doesn't apply. And that's the whole mistake in one comment.
Speaker A
Now, let's put this in price terms. The M5 Ultra Mac, as configured with 256 gigs and 8 tab, is $14,299.
Speaker A
The two Sparks were around $5,000 each when he filmed and have since gone to around $6,950 for the 128 gig version. So, a pair is about $13,900.
Speaker A
Practically the same money with one user on Quinn. $14,299 for 45 tokens per second is $318 per token per second and $13,900 for 38 is $366. So, the MAC is cheaper per token, and every single stream chart tells you to buy the MAC. At eight
Speaker A
users $14,299 for 70 tokens per second is $24 per token per second, and $13,900 for 120 is $116. So, the Mac went from the better buy to nearly $1.8 time the price per unit of work. And nothing about either machine changed except how
Speaker A
many conversations it was handling. Now the reason comes down to the other half of every request which nobody talks about. Before a model writes a single word back, it has to read our entire prompt. And reading the prompt and
Speaker A
writing the response lean on completely different parts of the hardware. Reading the prompt is math heavy because the model processes the entire input at once with all its compute. And writing the response is the bandwidth heavy part we
Speaker A
talked about earlier. one token at a time, dragging weights out of memory over and over. Now, the M5 Ultra's secret weapon is memory bandwidth, about 1.2 terabytes per second, which is more than four sparks combined. That's why it
Speaker A
ties or beats everything at a single stream. But compute is its weaker side and every new user brings a new prompt to read. And this can found that past about four users, the Mac was spending more time reading prompts than writing
Speaker A
responses. And every new conversation had to wait behind all the others being processed. So everyone's share of the output shrank. And batching magnifies compute work because more simultaneous streams means more math per memory trip, which is the whole point. So a box with
Speaker A
enormous bandwidth and modest compute is perfect for one stream and runs out of room at 8. So the spec that decides an agent speed turns out not to be gigabytes of memory, which is what the capacity charts show, or gigabytes per
Speaker A
second of bandwidth, which is what the speed charts show. It's compute, meaning tensor cores. And almost none of the buying guides even mention it. I really didn't understand this until I dug into it because I was buying bandwidth and
Speaker A
calling it speed when what my agentic workload actually needed was more compute to feed the batching engine. And the worst part is that even with a beefy GPU, the default settings on the most popular local AI tool actively prevent
Speaker A
you from using it. This one is completely free to fix and it might be the most important thing in this video for people who already run local AI.
Speaker A
Now, Olama is the most popular way to run local models because it's simple and has a great model library. It's what probably 70% of people running local AI are using, which included me until recently. Ola has a setting called Olama
Speaker A
num parallel which controls how many requests the server handles at the same time and the documentation lists the default as one one single request. So if you've got an agentic setup like cloud code with four sub agents or any agent
Speaker A
harness that fires several requests and you point it at a default lama install, your server isn't patching anything.
Speaker A
It's queuing. Your 3090 is doing the 6minute version of the job and never getting asked for the 54cond one.
Speaker A
There's a tutorial for running local models under cloud code that makes setting this to four literally the second step. And that's the whole fix.
Speaker A
One line in a config file. And I can't believe I went so long without knowing it. Now, to be fair, the other engines handle it better by default. Llama CPP server picks four slots when you leave it on auto, and LM Studio defaults to
Speaker A
four parallel predictions on its Llama CPP back end. But if you're on Olama and haven't touched that setting, you're leaving an enormous amount of performance behind. The real irony is that cloud code is configured to spawn up to 20 sub aents by default. So that's
Speaker A
20 agents asking and one request served at a time. We could wire up the most sophisticated agentic setup in the world with thoughtful parallel sub aents and all the right prompting and the inference server would just quietly serve it like a single lane road at rush
Speaker A
hour. So change the setting first before you buy or benchmark anything. Set noom parallel to at least four or move to Llama CPP server or VLM and then time a real task because you might be shocked at what you already own. But there's one
Speaker A
more thing to understand about batching because it isn't free and the cost lands on memory. Now every simultaneous conversation has to keep its own working state, a cache of everything the model has already processed in that conversation which is called the KV
Speaker A
cache. For Quinn 3.827B, this is actually pretty lean since only 16 of its 64 layers maintain a cache, which works out to 64 kilobytes per token. So a 32,000 token agent context, a decent working memory for a coding
Speaker A
task, takes about 2 GB per session. and a 64,000 token context takes four. That Hyper Quinn run at 1,035 tokens per second used 640 token conversations. And if we gave each of those 64 seats of 32K working context instead, we'd need 128
Speaker A
GB of cache before even loading the model, which is more than five full RTX3090s just for cash. Now, Olama's own FAQ puts it plainly where a 2K context with four parallel requests becomes the equivalent of an 8K context in memory demand. And
Speaker A
this reframes what memory actually means for agents. More memory doesn't mainly buy a bigger model. It buys more seats, meaning more simultaneous conversations, each with a reasonable working context.
Speaker A
So, a 24 GB card has the compute for dozens of streams and the memory for only a few long ones. A 128 GB Spark has room for many long contexts, but the compute to serve them all at once
Speaker A
becomes the constraint. So, the real question before buying isn't what the biggest model is that this will fit.
Speaker A
It's how many conversations we need running at once and how long each one needs to be. Manolono's video says you don't need 512 gigs because a model that big runs too slowly. And I see where he's coming from since single stream
Speaker A
speed on a giant model is rough. But if you're running six agents with 50k context each, those 512 gigs aren't buying one huge slow model. They're buying six seats with real working memory, which is a completely different value proposition. and Revolute running
Speaker A
2,000 engineers on self-hosted AI found the ceiling from the enterprise direction and their infrastructure lead said in a quote I found pretty funny that long context is a scam. Their real usage showed that 256,000 tokens is what you actually need and everything beyond
Speaker A
that was just cost for no benefit. It's the same lesson, which is to size the box by how many seats we need and how long each one has to be, not by the biggest model we could load. Now, I want
Speaker A
to briefly address something that might be in the back of your head because Sam Alman made it public at the end of October. He said on a podcast roughly that it's quite terrible from a global allocation perspective for everyone to
Speaker A
have chips in their personal computer getting used maybe 5% of the time. And his argument is that the cloud keeps chips fully utilized, so we should pay for what we use rather than own hardware that's mostly idle. And he's not wrong
Speaker A
about the 5% if we measure hours in the day, since most chips owned by individuals aren't running 24/7. But there's a second smaller inefficiency inside every second the chip is running, which is the 3.5% of its math doing work
Speaker A
during a single stream. And that one we can fix ourselves. The cloud wins because it keeps chips saturated with work. But saturating a chip is now something we can do at home, especially with aentic workloads where several tasks fire at once. And Revolute is the
Speaker A
proof at scale since they moved to self-hosted models as the default for 2,000 engineers running on 56 Nvidia B200s. And their cost per million tokens landed at about 30 cents, roughly a tenth of what cloud providers like Codeex or Claude Code charge. and their
Speaker A
total coding inference bill dropped about 90%. And when we look at their utilization, interactive speed during working hours is about 200 tokens per second. After 5:00 p.m., when fewer engineers are online, it climbs to about 800. So, it's the same chips with fewer
Speaker A
concurrent users and four times the throughput per session. So, the argument for self-hosting doesn't require arguing with Altman about global chip allocation. It just requires keeping our own chips busy, and Agent workflows do exactly that. Now, I want to be fair
Speaker A
here because I've been pretty bullish on batching for the last 15 minutes, and there are real limits. The first is that each stream slows down as we add more.
Speaker A
And on Mi Spark test, going from one stream to 16 drops each stream from 56 tokens per second to about 28. And the time to first token goes from about a tenth of a second to over 4 seconds. So,
Speaker A
if you read every answer as it streams, more sessions make each one feel noticeably worse. The second is that speculative decoding fades under load.
Speaker A
Some engines use speculative decoding which essentially makes educated guesses about the next few tokens and then verifies them. And it can make a single stream significantly faster by using spare compute. But batching eats that spare compute. And the Hyperquen repo
Speaker A
actually ships separate modes. One for single users with speculative decoding and one for batch mode without it. and the crossover on a 3090 is apparently around eight concurrent users, beyond which speculation hurts more than it helps. The third is that the numbers
Speaker A
don't transfer between models because the batch multipliers I've been quoting are specific to Quen 3.827B, a dense model and mixture of experts models behave differently. So when comparing two boxes, we should only compare their own scaling curves and not
Speaker A
raw speeds from different benchmarks. And the fourth, the most practically important one is that the tasks have to actually split. The person with 23090 said his 2.43 times only worked after he restructured his workflow. So each sub aent wrote against a shared contract
Speaker A
file instead of depending on each other's half-finished code. Work that has to happen in sequence stays in sequence no matter how fast the GPU is.
Speaker A
None of that makes single stream speed the right number. It just makes it one row in a larger table. Before I wrap up, I want to be real about who this advice applies to because it doesn't apply to
Speaker A
everyone. If your local AI is basically a chat window where you ask one question, get one answer, and respond, then you're literally one stream and every buying guide I criticized today is giving you correct advice, which is to
Speaker A
buy bandwidth and capacity. And the Max Studios 45 token per second lead over the Sparks at one stream is real for you. The same goes if you mostly feed it one large document at a time or if your
Speaker A
main goal is running the biggest possible model at any usable speed. Capacity first is exactly right there and the 512 gig Mac studio exists for you. And if your agents only run while you're at the keyboard and you don't run
Speaker A
things in the background overnight, then Ultman has a point about your setup too. A hosted model keeps someone else's chips full and you only pay for the tokens you use, which for light agent work might be the better deal. But if
Speaker A
you're running an agent harness with sub agents, review passes, overnight task cues, or background work, you're buying for several streams, and almost every guide out there is pricing one. So, what do we actually do with all of this? Step
Speaker A
one before buying anything is to count your streams. And this is the step I skipped. Run your agent harness through a normal working session and look at how many requests are genuinely in flight at the same time. Llama CPP server reports
Speaker A
its active slots and VLM reports running requests. So actually check if the answer is one. Buy for one stream where bandwidth per dollar wins and a used 3090 at around $1,500 still leads the list. So maximize tokens per second and
Speaker A
buy the MAC if you want speed and have the budget. If the answer is four or more because you run agents and parallel tasks, then buy for compute and for seats. And you'd want a GPU class box, an inference engine that actually
Speaker A
batches, and a performance chart measured at your stream count instead of at one. Step two, whatever you already own, is to change the defaults by setting Lama num parallel to at least four or switching to Lama CPP server on
Speaker A
auto or to VLM, which costs $0 and takes about 5 minutes. and you might find you already have what you need. Step three is to measure the right thing. Instead of benchmarking tokens per second on a single prompt, run your actual workflow,
Speaker A
your agent task or your code review, and time it from start to finish. That's what artificial analysis moved to. And the box that finishes your task fastest is the right box for you. This is how Revolute ended up on 56B200's instead of
Speaker A
a bigger Mac cluster. and how the 23090 owner discovered his hardware was fine and his inference server just wasn't using it right. The question was never how fast this GPU can talk, but how fast it can finish the job. And what I keep
Speaker A
coming back to is that I didn't waste my money on bad hardware, but I might have wasted some of its potential because I was asking the wrong questions. But the buying guides aren't lying to you.
Speaker A
They're just measuring single stream speed. And that matters less every month as the way people actually use AI shifts toward agentic parallel taskdriven work the benchmarks haven't caught up to. An artificial analysis said they stopped ranking by tokens per second because as
Speaker A
work gets more agentic the time to finish a task matters more than speed per token which is how the faster streamer ends up losing. GPT6 Astra at 51 tokens per second finishes a task in 17 minutes while Claude Opus 5.5 at 111
Speaker A
tokens per second takes 26 minutes because it generates more than twice the tokens. So the model that looks twice as fast takes half again as long on the actual task. Speed feels like tokens per second because that's what we watch
Speaker A
streaming by. But the work is measured in how long the task takes. And those two can point in opposite directions.
Speaker A
So, count your streams, change your defaults, and time your actual tasks. And if any of this was useful or you caught yourself nodding along because you've made some version of this mistake, too, I'd really appreciate a like or a subscribe. I'm making more
Speaker A
videos that dig into the stuff most guides skip over, and it genuinely helps to know it's worth making. Thanks for watching. Go check that out parallel setting and I'll see you in the next
Topics:local AIGPU benchmarkstokens per secondbatchingcontinuous batchingRTX 3090AI agentsinferencememory bandwidthhardware buying guide
