Speaker A
Hello everyone. This is the team from Serverflow. Today, we tested 10 quantizations of Qwen 2.5 7B, ranging in size from approximately 11 to 14 GB.
ServerFlow AI Lab tests 10 quantizations of Qwen 2.5 7B models on 16 GB GPUs, evaluating performance, code quality, and interface capabilities.
Key Takeaways
- Qwen 2.5 7B quantizations fit well on 16 GB GPUs and run efficiently with Llama.cpp.
- Deeps Harness is effective for managing autonomous agent workflows and debugging.
- Most quantizations can generate functional complex projects like Minecraft-like games in one attempt.
- Quantization impacts code quality and task completion, with some models missing critical instructions.
- UD Q3 KXL quantization offers the best trade-off between performance and resource usage.
What the video covers
- Tested 10 quantizations of Qwen 2.5 7B models sized 11-14 GB, all runnable on GPUs with 16 GB VRAM like RTX 4060 Ti and Tesla V100.
- Used Llama.cpp with identical parameters and Deeps Harness for long-term autonomous work and agent management.
- Performed recursive tree growth and wind animation tests to evaluate model understanding of recursion and animation quality.
- Challenged models to create a Minecraft-like game, assessing world rendering, controls, interaction, game loop, and UI completeness.
- Found two quantizations (IQ3_S and IQ4_XS) failed core building mechanics; others produced functional games with varying interface quality.
- Tested full interface creation with features like search, filter, theme switching, interactive charts, and model comparison blocks.
- All 10 models successfully linked UI components with shared state, showing diverse design approaches and additional features.
- Compared quantizations to FP8 reference over 16,000 tokens by prediction confidence correlation, identifying IQ4S Pure and UD Q3 KXL as top performers.
- Recommended UD Q3 KXL as the best balance for 16 GB use, IQ4 XS Pure for quality, and UD IQ3 XS for memory efficiency.
- Overall, compact quantizations deliver respectable performance and functionality for complex AI and UI tasks.
Chapters
- 00:00Introduction and Overview of Qwen 2.5 7B Quantizations
- 00:25Running Q3_K_XL Quantization and Deeps Harness Setup
- 00:46Deeps Harness Modes and Features
- 01:09Recursive Tree Growth Test Results
- 01:37Wind Animation Test and Increasing Task Complexity
- 02:01Minecraft-like Game Creation and Evaluation
- 02:29Game Logic Failures and Successes Among Models
- 02:55Full Interface Creation and Feature Testing
- 03:50Interface Design Variations and Additional Features
- 04:39Quantization Quality Comparison and Recommendations
Full Transcript — Download SRT & Markdown
Speaker A
This means that all models fit onto a single 4060 Ti, Tesla V100, and many other GPUs with 16 GB of VRAM. All models were run via Llama.cpp with identical parameters. On a single RTX 4060 Ti, we were able to comfortably
Speaker A
run the Q3_K_XL quantization. With the launch parameters shown on the screen, the inference speed was around 36 tokens per second. For the harness, I chose the Deeps Harness. It is designed for long-term autonomous work, saves the entire agent trajectory, and
Speaker A
manages context, command execution, and retries. It features four modes. Standard provides a full set of tools.
Speaker A
PTC allows you to combine a sequence of actions into a single TypeScript script. Minimal leaves only the terminal and file editor, while Creator is needed for creating and debugging your own agents and plugins. I have often seen
Speaker A
the combination of Qwen 2.5 7B and Deeps Harness referred to as a "Killer Combo." So, let's see how good it actually is in practice. The first task is to recursively grow a tree and show the construction process. This is a
Speaker A
relatively simple test, but it checks several things at once. Whether the model understood the recursive structure, created a working animation, and met the visual requirements. All 10 results are shown simultaneously. Some models build a normal branching crown, some result in an almost bare trunk,
Speaker A
and for certain variants, the tree starts growing in a completely unexpected direction. The next stage uses the same tree, but now the model must add a wind animation. The base should remain relatively stable, while the thin branches should deflect more.
Speaker A
The movement must also be smooth and cohesive. After this, the task becomes significantly more difficult. Each model receives the same prompt to create their own Minecraft-like game.
Speaker A
Now it is necessary to simultaneously implement world rendering, controls, interaction with blocks, the game loop, and the user interface. One forgotten link between components can render the entire project useless. The results turned out to be very different. Some
Speaker A
models created a full first-person view. Others interpreted the task as an isometric world editor, and some focused on a beautiful interface, sacrificing game logic. While evaluating each model, I check more than just the visuals. It is important
Speaker A
whether the game launches, if the controls work, if you can interact with the world, and how fully the initial prompt requirements were met. This is exactly where quantization degradation becomes most noticeable. A model might write a large amount of outwardly clean
Speaker A
code but miss one critical instruction, and the project turns from a game into a pretty demo. As a result of this test, only two out of the ten models failed to implement the core building mechanism. These are IQ3_S and IQ4_XS.
Speaker A
Their games looked fine externally, but did not respond to the left and right mouse buttons, so placing and deleting blocks was impossible. And this is a good example of why such projects cannot be evaluated by video alone. The
Speaker A
remaining eight models handled the main task quite well. Moreover, some paid attention not just to the game itself, but to the page around it. They added panels, instructions, decorative elements, and tried to capture the vibe of old browser game sites. As a result,
Speaker A
even in such a powerful quant, the code quality degrades in some places. But the overall result still looks respectable. Most models managed to create a truly functional Minecraft-like game in just one try.
Speaker A
Now let's see how the models handle a full-fledged interface for all ten quants. The technical requirements included a name search, a filter for models under 12 GB, light and dark theme switching, an interactive chart with tooltips, a results table, and a
Speaker A
dedicated model comparison block. For each site, I performed the same fourteen-second test. First, I switched the theme, enabled the size limit, entered Q3 into the search bar, and reset the filters. After that, I hovered the cursor over a point on the
Speaker A
chart and changed one of the models in the comparison block. All 10 variants handled the main scenario successfully.
Speaker A
The search and filter actually updated the model list. The chart points responded to hovering, and the comparison block updated after selecting another quant. At the same time, the models approached the design differently. Some created a strict light analytical interface, while
Speaker A
others started with a dark theme and used blue or purple accents. Most added four cards with key results at the top, but the layout varied thereafter. In some, the chart and comparison are side-by-side, while in others, the table is the central part and the
Speaker A
comparison is below. Some models went beyond the minimum requirements, adding table sorting, reset and clear buttons, model icons, and A/B selection directly from the table or by clicking a chart point. There are both Russian and fully English versions of the interface. The
Speaker A
main thing is that all 10 models didn't just draw a similar dashboard, but linked the search, filter, chart, and comparison with a shared state. For such compact quants, this is a truly worthy result. And so as not to
Speaker A
evaluate quants just by eye, I compared them with an FP8 version on a sequence of 16,000 tokens. For each token, I measured how confidently the model predicted the correct next token, and then these values were compared with FP8. The higher the correlation, the
Speaker A
more accurately the quant mimics the reference's behavior. The best result was shown by IQ4S Pure. Almost on par with it was the more compact UD Q3 KXL.
Speaker A
NWFP4 lagged noticeably behind, and the last place went to Rich 3.7 BP. Based on the test results, if I were told to keep one quant for 16 GB, I would choose UD Q3 KXL. It is the golden mean.
Speaker A
IQ4 XS Pure is worth getting for quality, and UD IQ3 XS for memory overhead. That's all for now. You've been watching the specialists from Serverflow. Like the video and subscribe to the channel. Goodbye everyone. M.
Topics:Qwen 2.5 7BquantizationAI models16 GB GPULlama.cppDeeps HarnessMinecraft-like gamemodel evaluationFP8 comparisonServerFlow AI Lab
