Speaker A
On the 27th of July, a Chinese lab called Moonshot uploaded a file to HuggingFace. The file was 1 and a half terabytes. Inside it sat 2.78 trillion parameters. Not a demo, not an API key.
Demonstrates running the massive Kimi K3 model on 8GB RAM without GPU by streaming weights from NVMe, overcoming memory limits with storage engineering.
Ask about this video. Answers come from its transcript only — with the timestamp, so you can check them.
Generated from the transcript and can be wrong — check the timestamp.
Key Takeaways
- Large-scale models can run on modest hardware by streaming weights from fast storage.
- Mixture of experts architecture enables sparsity, reducing active parameters per token.
- Quantization-aware training allows efficient 4-bit weight representation without accuracy loss.
- Memory bottlenecks can be overcome by trading for bandwidth, a more scalable resource.
- Open release of large models accelerates innovation and independent implementations.
What the video covers
- Moonshot released a 1.5TB Kimi K3 model with 2.78 trillion parameters openly on HuggingFace.
- Initial runs required massive GPU clusters, but independent engineers optimized it to run on much smaller hardware.
- A researcher in Karachi ran the full model on 8GB RAM, one CPU, no GPU, using a custom C engine of 6000 lines.
- K3 uses a mixture of experts architecture activating only 16 out of 896 experts per token, enabling sparsity.
- The model weights are quantization-aware 4-bit, trained with this precision rather than compressed later.
- Key innovation: streaming most of the model weights from NVMe storage instead of loading all into RAM.
- This trades the memory wall for a bandwidth wall, allowing operation on ordinary hardware but with slow speed (~32 seconds per token).
- Other projects like Deltafin and Waste independently implemented similar streaming approaches with different tradeoffs.
- Despite slow inference, the output is byte-identical across memory budgets, proving correctness.
- The breakthrough shows memory limits are a loading assumption, not a physical barrier, opening new possibilities for large model deployment.
Chapters
- 00:00Moonshot releases massive Kimi K3 model
- 00:45Running K3 on 8GB RAM without GPU
- 01:26Model performance and ranking
- 01:59Cost analysis and loading challenges
- 02:38Mixture of experts architecture explained
- 03:10Four key reductions enabling low RAM usage
- 04:42Verification of output correctness
- 05:20Memory vs bandwidth tradeoff
- 07:02Independent projects replicating the approach
- 07:36Summary and implications for AI deployment
Full Transcript — Download SRT & Markdown
Speaker A
The weights themselves were published under Moonshot's own license for anyone to download. No fee, no waiting list, nothing to sign, assuming you happen to own a rack of Blackwell GPUs. Within nine hours, somebody had it running on one machine at home written in Rust from
Speaker A
scratch. A day later, a second engine did it in 29 GB of RAM. That one hit the Hacker News front page. Then on the 1st of August, a researcher in Karachi got it down to 8.24, 8 GB of RAM, one CPU, no
Speaker A
GPU, no deep learning framework, not even a BLAS library. About 6,000 lines of C compiling to a binary of 176 kilobytes. Roughly 1,400 stars in under three days from one person in 27 commits. There is a catch here. It is an
Speaker A
enormous catch and it is the most interesting part of the story. Start with why anyone bothered. Kimmy K3 is not a curiosity. It currently sits first in the front-end code arena at 1679 ahead of Claude Fable 5. On one
Speaker A
independent index, it ranks fourth out of 189 models, though the source of each number matters. Moonshot's own model card reports 88.3 on terminal bench 2.1.
Speaker A
An independent run by Val scores the same model at 80.9. Both can be true.
Speaker A
Different harnesses give different results, but only one of those numbers was chosen by the people selling the model. It reads a million tokens of context. And it was quantization aware, trained from the supervised stage onward. So the 4-bit weights it ships
Speaker A
with are the weights it learned with, not a compression applied afterwards. And Moonshot gave it away. Nearly a million downloads in the first week. The catch was not the license. Giving it away also makes commercial sense. One engineer ran the numbers and put the
Speaker A
marginal cost of serving K3 on rented B300s at about $137 per million tokens against roughly $15 a million charged for output through the API. The catch was the loading. A single node of eight Nvidia B300s gives you 2,300 gigabytes
Speaker A
of high bandwidth memory. The weights alone take about 196 per GPU. Rented, that node runs around $60 an hour.
Speaker A
Therefore, the interesting question is not whether the model is good. It is arithmetic. How do you get 1.5 terabytes of weights through 8 GB of RAM and still get the right answer out? The answer starts with a fact about how K3 is
Speaker A
built. It is a mixture of experts and in each layer it carries 896 of them. But a router picks 16. For any single token, 16 experts fire and the other 880 do nothing at all counted in parameters.
Speaker A
104 billion are active for a given token out of 2.78 trillion. That is 3.7%.
Speaker A
Which means that at any instant more than 96% of that 1 and a half terabyte file is asleep. That gap is the crack the whole story goes through. But asleep is not the same as gone. The router only
Speaker A
tells you which experts it wants once you are already computing the layer. So all of them have to stay reachable fast right now. Sparsity alone gets you nothing. It took four separate reductions. The first is format. The routed experts ship MXFP4bit
Speaker A
nibbles sharing one 8-bit exponent across every 32 weights. That is a hair over half a byte per parameter. And the engine multiplies straight from the packed nibbles without ever unpacking them. The second is the attention. 69 of the 93 layers use Kimmy delta attention
Speaker A
which keeps a recurrent state of a fixed size 9 and a quarter megabytes per layer, 626 megabytes in total, and that number does not move. In the author's words, it is the same whether you feed it 10 tokens or a million. The usual reason long
Speaker A
context eats your memory simply is not there. The third is the other 24 layers which cache a single 576-dimensional latent per position instead of separate keys and values for every head. That is 53 times less cache, about 2 and a third
Speaker A
megabytes per position. The fourth is the part that sounds like cheating and is not. The dense trunk, 109 GB of it, does not fully load. It streams off the NVMe with direct IO, pinning the first and last layers and rotating the rest through
Speaker A
one slot, which gives a hit rate you can predict exactly, n over 93. Add the four together. In BF16, holding this model would want 5 and a half terabytes of RAM. He runs it in 8.24 GB. That is a factor of
Speaker A
675, but that number only counts if the output is right. And a handwritten engine is where silent corruption hides.
Speaker A
So he checked the tokenizer matches Python's tick token on 45 cases out of 45. The 4-bit multiply was verified exact against the release checkpoint bytes. And the detail that turns a stunt into a result is this. The output is
Speaker A
byte identical at every budget. 8 GB, 32, 95, 224, same tokens, same bytes. The memory budget changes the speed and nothing else. One writer summed it up better than the readme did. The memory limit we had all taken for granted was
Speaker A
not a property of the model. It was a property of how we were loading it.
Speaker A
Which brings us to the catch. You cannot delete work. You can only move it somewhere else. And he moved it onto the disc. 32.69 seconds per token. Read the units in the right order. Seconds per token, not tokens per second. A 500-word
Speaker A
answer is roughly 650 tokens. At that rate, you are waiting nearly 6 hours for it. And the benchmark host was not a laptop. It was a dual socket EPYC with 124 cores and a 3.2 TB NVMe. Laptop is
Speaker A
the name of a memory budget, not a measurement on a laptop. Before any of that, there is the first token about 167 seconds of loading pin trunk layers before a single character appears. And during generation, the processor is
Speaker A
mostly idle. Disk IO accounts for somewhere between 41 and 71% of wall clock time. The CPU is not the bottleneck. It is waiting. You also need 1.7 terabytes of free NVMe just to hold the thing. The checkpoint plus a repacked
Speaker A
trunk. 93% of it stays on the disc out of RAM entirely. So the engineering that matters here is storage engineering.
Speaker A
Direct IO reads cold at 3.2 GB a second where buffered reads manage 2.3 and during inference it sustains 5 to 6.
Speaker A
Therefore, the accurate description is not that the memory wall came down. The memory wall was traded for a bandwidth wall, and bandwidth is a wall you can buy your way through, which memory capacity at these sizes is not. That
Speaker A
trade is exactly what the argument on Hacker News was about. One commenter put the case for the prosecution. By the time your tokens arrived 30 hours later, you might want your laptop back. But the reason to take this seriously is that it
Speaker A
is not one person's hobby. In the week after those weights landed, at least three independent teams built the same idea from scratch. Eight and a half hours after the upload, a project called Deltafin appeared, written in Rust, running the full model on a single
Speaker A
device with an OpenAI compatible server bolted on. 665 stars 28 hours after Waste arrived from the team behind East Kolite Cloud. The name is an acronym.
Speaker A
Weight-aware streaming tensor engine. Same core move. Hold the trunk in memory. Stream the experts off NVMe. But Waste spends its memory differently.
Speaker A
Experts get three-bit residual vector quantization while the more sensitive shared weights stay at four or eight and the matrix multipliers run directly against the codebooks. A look-ahead router guesses which experts the next layer will want and starts reading them
Speaker A
early. The result is a different point on the same curve. 29 GB as the hard floor, 64 recommended, and about half a token per second on a MacBook Pro. Every cold token drags roughly 17 GB of experts off the disc, and it stays
Speaker A
faithful doing it. Waste's final logits agree with the reference to within about 3 million, though Bambini's own run came in slower than his readme's range at roughly a third of a token per second.
Speaker A
Streaming speed depends on what the cache happens to be holding. That project drew 338 points and 166 comments in a day.
Speaker A
The defense from a commenter called Ruler 88 was one sentence. This is not about practicality today. It is...
Speaker A
a little over 1,400 by the 4th, around 480 a day with no front page thread carrying it. Line the numbers up and the shape is obvious. 8 GB buys you a token every 32 seconds. 29 GB buys you half a
Speaker A
token a second, 16 times faster for 3 and 1/2 times the memory. 128 GB gets you to 20 seconds a token. 2,300 gets you tens of tokens a second. So, memory was never a wall with the model on one
Speaker A
side and you on the other. It is a dial. Every notch you decline to buy, you pay for in time. And the exchange rate is brutal at the bottom. Which leaves the question of what any of this is for. And
Speaker A
both authors answer it the same way. Bambini's own assessment of waste is that at its current speed, it is a technical result, not a generalpurpose interactive deployment. The C engine's readme is blunter, slow, and answering correctly in 8.24 GB. That is a familiar
Speaker A
kind of useless. Back in 2023, Andre Carpathy published Llama 2 inference in one file of Pure C. It was not built to serve traffic. It was built to make the architecture legible. It has just under 20,000 stars and it taught a generation
Speaker A
how a transformer actually works. The comparison the C engine invites is with Llama CPP.
Speaker A
122,000 stars, 400 megabytes of repository and the tool people actually run. This is seven source files and 176 kilobytes. Those are not competitors.
Speaker A
One is infrastructure. The other is an explanation you can compile. And the explanation came from one person. Farad Khan works out of Karach. His profile says he does research on AI and is looking for a PhD place or a job. 27
Speaker A
commits pushed inside a 2-hour window on a Saturday and he has not touched the repository since. One caveat since it gets repeated wrongly. These weights are not MIT licensed. They ship under Moonshot's own Kimmy K3 license. Open weights and open source are not the same
Speaker A
thing. And the difference sits in a document most people downloading a terabyte and a half will not read. He did not make a 2.78 trillion parameter model practical. Nobody has. What he did was make its shape visible. Proving that
Speaker A
the thing standing between a frontier model and ordinary hardware was an assumption about loading, not a law of physics. That is the useful part.
Speaker A
Someone draws the line.
Topics:Kimi K3large language modelmixture of expertsquantization-aware trainingmodel streamingNVMe storagelow RAM inferenceRustC languageMoonshot AI
