**Qwen 3.8 27b x NInfer = 1.5x Faster@256K Context(on one 24GB 4090) โ€” Transcript & Summary | SozAI** Source: https://sozai.app/transcript/qwen-38-27b-ninfer-faster-256k-context-24gb-4090/ Qwen 3.8 27B with NInfer framework achieves 1.5x faster inference at 256K context on a single 24GB RTX 4090 GPU. ## Key Takeaways - NInfer delivers significant speed improvements for Qwen 3.8 27B on RTX 4090 GPUs. - Extreme optimization and fixed hardware/model path enable efficient long-context inference up to 256K tokens. - Prefix caching and RadixAttention reduce redundant computations and VRAM fragmentation. - Cold start latency is higher but overall throughput and generation speed are superior to llama.cpp. - NInfer sacrifices flexibility for maximum performance on a single model and GPU configuration. ## What the video covers - The video discusses deploying Qwen 3.8 27B model using the new NInfer inference engine optimized for long context and MTP setups. - NInfer claims up to double the speed compared to llama.cpp and vLLM by fixing the model and GPU path for extreme optimization. - The presenter tested NInfer on an RTX 4090 with 24GB VRAM, achieving stable 256K token context and generation speeds around 100 tokens per second. - NInfer uses prefix caching and RadixAttention to efficiently reuse computations and reduce VRAM fragmentation. - Unlike llama.cpp, which supports many models and devices with compromises, NInfer is a specialized engine locked to Qwen 3.8 27B and RTX 4090 hardware. - Initial cold start latency (TTFT) is higher with NInfer but throughput and speed improve significantly during runtime. - The framework is compared metaphorically to a Formula One car optimized for one track, whereas llama.cpp is likened to a versatile Toyota SUV. - NInfer's quantization uses RK4V4-E8, a complex compression method not fully explained in the video. - The presenter highlights the advantages of longer context windows, less VRAM usage, and faster speed with NInfer. - The video includes detailed benchmarking, configuration testing, and a comparison to other inference engines. ## Chapters 1. 00:00 Introduction to Qwen 3.8 27B and NInfer framework 2. 00:48 Discovery of NInfer and speed claims 3. 02:14 NInfer framework overview and optimization 4. 02:56 4090 optimization fork and deployment 5. 03:56 Comparison with llama.cpp and vLLM 6. 04:44 Prefix caching and RadixAttention explained 7. 05:29 Quantization details and model specifics 8. 06:02 Performance benchmarks and speed analysis 9. 06:46 Summary of advantages and trade-offs 10. 08:46 Final thoughts on NInfer and Qwen 3.8 27B Answers ## Questions about this video What is NInfer and how does it differ from llama.cpp? NInfer is a new inference engine built from scratch, optimized for a fixed model (Qwen 3.8 27B) and GPU (RTX 4090). Unlike llama.cpp, which supports many models and devices with compromises, NInfer maximizes speed and efficiency by locking the hardware and model path. What performance improvements does NInfer provide? NInfer achieves about 1.5 to 2 times faster inference speeds compared to llama.cpp, supporting up to 256K token context with generation speeds around 100 tokens per second on a single 24GB RTX 4090 GPU. What are the trade-offs when using NInfer? NInfer has higher cold start latency (TTFT) compared to llama.cpp and sacrifices flexibility by only supporting a specific model and GPU configuration, but it gains significant speed and VRAM efficiency for that setup. ## Full Transcript โ€” Download SRT & Markdown 00:01 Speaker A Okay, today I'm gonna talk about Qwen 3.8 27B and a new framework I found recently in specific setups: long context plus MTP. The official and community benchmarks show it getting close to twice the speed. So, how fast would my own 4090 ninety actually go? 00:18 Speaker A Stick around till the end and you'll see so how do we start this last week I just finished deploying Qwen 3.8 back then I was on llamacpp and I went pretty hard on the optimization Dozens of hours of metrics testing 00:33 Speaker A Stick around till the end and you'll see. So, how do we start? This last week, I just finished deploying Qwen 3.8. Back then, I was on llamacpp and I went pretty hard on the optimization. Dozens of hours of metrics testing. 00:48 Speaker A But just one week later I found out all of that testing was for nothing Because while I was digging around I found a new framework called NInfer The readme is absolutely unhinged It says straight up double the speed 01:01 Speaker A All to find that one best set of parameters. And I finally got where I wanted it. So, I locked it in: 200k context, seventy to eighty tokens per second. At that point, llamacpp was already optimized to the absolute limit for my hardware. 01:15 Speaker A Then I threw that plan at DeepSeek V4 Flash on my server V4 Flash grinded away on its own for five hours configuration metrics testing for every NIFER parameter Then metrics comparison against Llama I went out came back And it had delivered a working best config and a comparison result 01:34 Speaker A But just one week later, I found out all of that testing was for nothing. Because while I was digging around, I found a new framework called NInfer. The readme is absolutely unhinged. It says straight up double the speed. 01:48 Speaker A basically the same category of thing as a llama.cpp or V vLLM but it's not like those at all It's an inference engine written from zero And the special part is that other frameworks have to figure out how to support different models 02:02 Speaker A And if you round up generously, that's basically someone handing me a free fourteen ninety. Obviously, I have to deploy it and try it. So, I had GPT 56.6 write me a detailed plan covering deployment and testing the whole thing. 02:16 Speaker A And then someone else built a 4090 optimization on top of that 3090 work and the one I'm running is that fourteen ninety fork Which basically locks in Quad Three Point Eight Twenty Seven B plus the RTX Four Thousand Ninety 02:29 Speaker A Then I threw that plan at DeepSeek V4 Flash on my server. V4 Flash grinded away on its own for five hours: configuration, metrics, testing for every NInfer parameter. Then metrics comparison against Llama. I went out, came back, 02:41 Speaker A parse it pick a backend for your device and then run That whole chain exists because llama.cpp is trying to support a huge range of models and devices so it inevitably carries a lots of compromises and bugs NInfer the opposite 02:56 Speaker A and it had delivered a working best config and a comparison result. And keep in mind, four or five months ago this kind of thing wasn't possible at all. Back then, I was sitting there arguing with ChatGPT myself, hand-building a VLM config. And it took me several days. For anyone who doesn't know, NInfer is 03:13 Speaker A Snow sand mountain roads It'll handle all of it And the speed is well fine vLLM On the other hand It's more like a subway system It doesn't care how fast one passenger moves It cares about the total throughput per hour 03:27 Speaker A basically the same category of thing as llama.cpp or vLLM, but it's not like those at all. It's an inference engine written from zero. And the special part is that other frameworks have to figure out how to support different models 03:44 Speaker A It also has prefix caching now that have already been computed can be reused SGLang on the other hand it puts a lot of emphasis on reusing cache across complex workflows thinks about which cargo can share a route What can be cached 03:59 Speaker A and different hardware. Well, NInfer just locks the model and the hardware down to one single path. The original targets the RTX 5090 on the GPU side and mostly the Qwen 3.8 family on the model side. Somebody ported it to the 3090, 04:15 Speaker A since so much of what happens in there is repetition and then NInfer the one I tried is a Formula One car one track only that track one engine and only that engine the tires are fixed Transmission is fixed 04:29 Speaker A and then someone else built a 4090 optimization on top of that 3090 work. And the one I'm running is that fourteen ninety fork, which basically locks in Qwen 3.8 27B plus the RTX 4090, 04:43 Speaker A The forty ninety with twenty four gigs of RAM Ninfa can fit the entire two hundred and fifty six k native context of the model And generation speed averages around one 100 tokens per second Qwen 3.8 27B is also a bit unusual as a model 04:59 Speaker A and then optimizes that one path to death. So, it's basically a fixed GPU and a fixed model. What it doesn't do, what llama.cpp does, where you hand it to GGUF and it has to look at the architecture, 05:14 Speaker A It's quantized to and not the original K4V4 either it's RK4V4-E8 What that actually means I'm not going to get into wouldn't understand it anyway Mostly because I don't understand it either well it's some kind of a rotating jumping compression 05:31 Speaker A parse it, pick a backend for your device, and then run. That whole chain exists because llama.cpp is trying to support a huge range of models and devices, so it inevitably carries a lot of compromises and bugs. NInfer, the opposite. 05:44 Speaker A much as possible at the smallest quality cost So the full 262K context fits inside 24GB other frameworks make the model and the hardware compromise with the framework in exchange for compatibility NInfer rebuilt itself into the shape of Qwen 3.8 06:02 Speaker A It's extreme optimization and extreme fixation. This one path works and it's tuned to the limit. Llama.cpp is a really excellent Toyota SUV. You can drive it anywhere. There are parts for it all over the world. You can get it repaired anywhere. 06:16 Speaker A that should be an unmatched experience if any of you whales have tried it come and show off in the comments once NInfer was fully deployed the first thing I gave it was a small task Modify in my local AI console 06:28 Speaker A Snow, sand, mountain roads, it'll handle all of it. And the speed is, well, fine. vLLM, on the other hand, is more like a subway system. It doesn't care how fast one passenger moves. It cares about the total throughput per hour, 06:39 Speaker A the first thing I noticed was the first token got slower llamacpp to CPP is around five seconds This was taking twelve thirteen but to be clear that's the TTFT I see on the first cold request when I'm actually driving it with an agent 06:53 Speaker A how many people it can move in parallel. Its classic technique is a PagedAttention, which manages the KV cache in pages, kind of like virtual memory. That cuts down the VRAM fragmentation, and together with continuous batching, it's a great fit for high concurrency serving. 07:06 Speaker A after that it keeps getting faster because cache hits keep piling up It starts It's at a fifteen seconds and eventually it drops to a few seconds The other thing is speed In real use small tasks hit around one hundred and thirty tokens per second 07:21 Speaker A It also has prefix caching. Now, that which has already been computed can be reused. SGLang, on the other hand, puts a lot of emphasis on reusing cache across complex workflows, thinks about which cargo can share a route, what can be cached, 07:37 Speaker A One thing that's really impressive about this is inside NInfer I can push the context all the way to the top which is 256K And at that point I still have around one point two five gigs of RAM left 07:50 Speaker A which requests share a common prefix, and how to reuse those states. Its signature technique is RadixAttention, which roughly builds all the request prefixes into a tree and skips recomputing the duplicated ones, which makes it very effective for agent loops, 08:02 Speaker A so to sum up less VRAM usage longer context and faster speed Those are the three advantages The only downside is that cold prefill speed drops by twenty to thirty percent and one more thing that really matters when my machine is decoding 08:18 Speaker A since so much of what happens in there is repetition. And then NInfer, the one I tried, is a Formula One car: one track only, that track, one engine and only that engine, the tires are fixed, transmission is fixed, 08:34 Speaker A it's not a rule for every machine So altogether based on the sample I have so far it's roughly a twenty five to thirty percent power saving you can see the real wall power measurement here The days I was on llama.cpp 08:46 Speaker A the suspension is fixed, even the weather is fixed. Everything is a known quantity. And then he just goes out and hammers, slaps as hard as he can. And the result of that extreme optimization is on a card like mine, 09:01 Speaker A And just on the electricity cost and my current power rates that annualizes to somewhere around there 140 bucks a year The sample size is still small though I can't say for sure but from how it feels to me 09:13 Speaker A the 4090 with 24 gigs of RAM, NInfer can fit the entire 256k native context of the model. And generation speed averages around 110 tokens per second. Qwen 3.8 27B is also a bit unusual as a model. 09:25 Speaker A compared to an online API it still felt a little slow a little constipated Because the speed was only sixty or seventy And with the long text it drops to fifty or sixty which is a little annoying but once I switch to NInfer it gets 09:39 Speaker A It doesn't have a giant KV cache on every layer. Its hybrid architecture, which is why it can open up a full 256k context on a 24-gig card. A while back, that was unthinkable. That cache isn't running at original precision. 09:53 Speaker A There's no contest but anyway I'm extremely happy with how this optimization turned out Right now I've moved my whole production setup over to NInfer And it's been very stable No issues triggered at all and I don't feel any drop in quality 10:07 Speaker A It's quantized to, and not the original K4V4 either, it's RK4V4-E8. What that actually means, I'm not going to get into. You wouldn't understand it anyway. Mostly because I don't understand it either. Well, it's some kind of a rotating jumping compression 10:20 Speaker A so it doesn't do what my previous llama.cpp config did Where I'd run into VRAM and slot management issues on long sessions because with Llama if you don't manually pin the KV slots to one the default is four and it uses unified KV by default 10:35 Speaker A that's a bit better than the normal compression. So, with very little precision loss, it saves more VRAM than a Q8 or an FP16 KV cache. Its main purpose isn't speed at all. It's to shrink the KV cache as 10:50 Speaker A on a single 4090 running the context at the full 256K I personally think that's great Especially when you are using an agent for a long chain development Some people might say one 128K and twentyeight k of context is enough 11:04 Speaker A much as possible at the smallest quality cost. So, the full 262K context fits inside 24GB. Other frameworks make the model and the hardware compromise with the framework in exchange for compatibility. NInfer rebuilt itself into the shape of Qwen 3.8. 11:20 Speaker A the Qwen 3.6 27b The best I could get was forty or fifty tokens per second with MTP on a few months later stronger model faster speed forty to fifty native and over a hundred with MTP and the best parameters I locked in last week 11:35 Speaker A It serves Qwen 3.8 and nothing else. It's pure hardware juicing. So, if you have the matching hardware, it's genuinely worth trying. For running it on a 4090 48GB version, I know if it's been optimized well, but if it has, 11:49 Speaker A If not watch another one See you next time Topics: Qwen 3.8 NInfer RTX 4090 long context MTP inference engine llama.cpp vLLM prefix caching RadixAttention --- This is the markdown twin of https://sozai.app/transcript/qwen-38-27b-ninfer-faster-256k-context-24gb-4090/ โ€” the same content, without the markup. Published by SozAI (https://sozai.app). Reuse and quotation are allowed with attribution and a link back. Machine-readable index: https://sozai.app/llms.txt ยท data API: https://sozai.app/api/