A 16GB card and a gaming PC. With 32GB of system RAM or less, run Qwen 3.8 27B, the Swift 1.5 fine-tune at GSQ-RCO IQ3_S, everything on the card. With 64GB or more, run Qwen 3.8 Flash Next on UnbiasedStrata and let the experts and the n-gram table live in RAM. Why the picture changes every week, the hardware, how Bob-Bench Wide decides, and what Q2 does to a model.
Want a copy without ads, an account, or tracking? Grab the torrent (214.8 MB).
It includes a webseed, so it downloads even when nobody else is seeding. Please seed it if you can.
Magnet link
86 private software-engineering tasks, one run per configuration, same harness throughout. Score is the mean task score; Full = tasks scored 10/10; TO = timeouts (5-point deduction each). The card is held to 16,384 MiB for every row.
| Configuration | System RAM | Score | Full | Hours | Hardware |
|---|---|---|---|---|---|
| Qwen 3.8 27B Swift 1.5 GSQ-RCO IQ3_S +MTP (llama.cpp) | none used | 87.3% | 71 | 5.5 | RTX 4090, 16 GiB frame |
| Qwen 3.8 Flash Next GSQ-RCO IQ2_XS +MTP (UnbiasedStrata) | 32 GB | 89.4% | 71 | 6.9 | RTX 4090, 16 GiB frame |
| Qwen 3.8 Flash Next GSQ-RCO IQ3_XXS +MTP (UnbiasedStrata) | 64 GB, table on NVMe | 89.0% | 68 | 7.7 | RTX 4090, desktop i9 |
| Qwen 3.8 Flash Next GSQ-RCO IQ3_XXS +MTP (UnbiasedStrata) | 128 GB, table in RAM | 92.4% | 76 | 6.7 | RTX 4090, 24-core EPYC |
| Qwen 3.6 27B UD-IQ3_XXS +MTP (llama.cpp) | none used | 87.3% | 66 | 5.8 | RTX 4090, 16 GiB frame |
Qwen 3.8 27B, Swift 1.5 fine-tune, GSQ-RCO IQ3_S with the MTP head. One 12.1 GB GGUF, all on the card. Reasoning effort medium, not the xhigh default.
llama-server \
--host 0.0.0.0 --port 8081 \
-hf ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S-mtp \
-c 131072 -ngl 99 -fa on --jinja \
--cache-type-k q4_0 --cache-type-v q4_0 \
--spec-type draft-mtp --spec-draft-n-max 2 \
--chat-template-kwargs '{"reasoning_effort": "medium"}' \
--reasoning on --reasoning-budget 8192 \
--temp 0.6 --top-k 20 --top-p 0.95 --min-p 0.0 \
--presence-penalty 0.0 --repeat-penalty 1.0 \
--n-predict 32768
Qwen 3.8 Flash Next, GSQ-RCO IQ3_XXS, on UnbiasedStrata. The 40 GB expert set and the 29 GB n-gram
table sit in page-locked system RAM; the CPU serves the cache misses, so headless Linux and a many-core CPU.
Set --pool-workers to your core count minus a couple. On a 24 GB card add
--vram-reserve-mib 8850 to hold it to the 16 GiB frame.
M=IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS
unbiased-strata --engine strata -m $M-00001-of-00002.gguf \
--host 0.0.0.0 --port 18090 --alias flash-next \
-- --pack pack-iq3xxs --expert-profile pack-iq3xxs/expert-profile.bin \
--native $M-00001-of-00002.gguf --ple-gguf $M-00002-of-00002.gguf \
--expert-cache auto --prefill auto \
--spec 4 --spec-min-p 0.5 --mtp mtp/ \
--max-context 131072 --kv q4_0 --kv-resident 32768 \
--ple-io ram --pool-workers 24 \
--prompt-cache 6 --prompt-cache-root 0
With 64 GB the table stays on NVMe and the run is CPU-bound: same expert hit rate as the 128 GB run, a third slower, timeouts on the long tasks. That rung is the trap, not the recommendation.
From the final-cut captions. Names and technical terms have been corrected; timestamp links open the corresponding point on YouTube.
0:00Say you want to turn your 16 gigabyte VRAM gaming PC into an inference platform so you can code at home using intelligence that's not that far off from what the frontier was 6 months ago, powered by the hardware that you already have with no data leaving the house.
0:14Here are the models and inference frameworks that I recommend. It depends on how much system RAM you have available. We're considering between 16 gigabytes and 128 gigs of system RAM here, alongside your GPU. If you have 32 gigabytes or less, I'm going to recommend Qwen 27B.
0:313.6 or 3.8. The best performing variants I tested were the Swift 1.5 at the GSQ-RCO quant, expect over 70 tokens per second here, and this is a pretty smart coding agent. And with the GSQ-RCO quant. It performs really well at Q3.
0:49So if you have an immense amount of RAM available, say 64 gigabytes dedicated to the task, or even better, 128 gigs, you can get better performance by downloading China's new hot model, Qwen 3.8 Flash Next. This model and the framework that's largely popularizing it, is taking everything by storm, is much larger, and it takes a lot more resources to run.
1:12So, With 128 gigs, Grab the IQ3_XXS quant, it's going to take longer to run. It'll use all the cores on your machine. GPU will be at full utilization too. You can even bump up to the IQ3_S quant. Use Strata or a fork of Strata.
1:29I have a fork that removes all the Python and bullshit to make it much like a compiled C++ binary is possible. This model is going to be significantly smarter, and it will take about 30% longer than the 27B dense to complete tasks.
1:43So expect around 90 tokens per second and a higher level of peak intelligence.
1:49Table of contents. Are we going to go over in this video? That was the intro and the configs. We showed them on screen. We're going to talk about developments in the space. Why is this stuff moving so fast and what are the developments that are impacting us most?
2:02In terms of our ability to run this stuff at home? Second is a little bit more detail on the hardware and configurations that I'm thinking about, making this recommendation. There it is Strata. This new framework is largely synonymous with running QFN, and we need to talk about it in a little bit more detail, even though I have a dedicated video on the subject on the channel.
2:23How I tested these configurations, we'll talk about Bob-Bench, how I measure this, and what my priorities are, when I'm evaluating a model, and why I think it might be generally relevant to look at C/C++ coding. And then I've got a couple issues with lower bit quants that we'll look at.
2:40And basically they have damage at Q2. And then further work, additional knobs. that we can turn an additional performance, tweaks and measurements that we should be doing.
2:55So why is this stuff changing so fast? What the hell is going on? 2026, you wake up, every day there's a new framework, a new thing.
3:03I think the biggest change in what has been the largest shift that I've seen in terms of a step function up in what you can run at home for local intelligence for coding is Qwen Flash Next. This is a preview of the Qwen 4 architecture.
3:18It has a large n-gram table that effectively increases the amount of state that the model has to work with without increasing the amount of VRAM that you need or the amount of RAM that you need, which is rather expensive.
3:31So this new MoE architecture, and others like it, are making it much easier to run much larger models at home. Uh, with great performance. So we're seeing specialization.
3:45winning. And that brings us to the second thing that's developing really fast and it's the inference frameworks, okay? So Strata, ninfer, freetoken. Specialization seems to be winning. Dwarf Star, which runs DS4. Um, llama.cpp is not the best performance.
4:04I think it was generally assumed that llama.cpp CPP had excellent performance for most models. This is a general purpose framework. It's largely C, C++ CUDA. It's definitely not true today. And they are so far behind what Strata is capable of doing in terms of running larger MoE models at higher tokens per second on consumer hardware.
4:28So, that's the second thing I'm seeing that's making it so, it's difficult to track what's best to run on your, on your, on whatever GPU you happen to have. So, and then third, and these are all hitting so hard.
4:43The third new quantization techniques. Okay, so GSQ-RCO, Gumbel softmax quantization, and Riemannian constrained optimization, these are ways to quantize the weights, and then optimally pick which weights to quantize how hard, given a budget, in the end resulting model.
5:02And these are making it so that Q3 is an excellent high quality quant, which I don't think was true before. So it's bringing up the size of the model that can be run, and it's bringing down the amount of VRAM that you need to run it.
5:15And so DASLab, the Austrian dudes who make this, doing an absolutely fantastic job and releasing these GSQ-RCO quants, which work with the existing GGUF format. Um, absolutely banger quants and they work in llama.cpp or whatever other framework uses a GGUF.
5:34So, additionally, number 4th development that keeps happening, new fine-tunes. And so Swift 1.5 is the first Qwen fine-tune that I have tested that absolutely dominates the base model, base Qwen. It's excellent. I have no idea what they fed this thing, but so the fine-tune.
5:56So we've got the architecture, the inference frameworks, new quantizations and new fine-tunes. All 4 of these developments have been in the last couple weeks and are influencing the recommendations. And the benchmarking that I'm doing at a given hardware budget today.
6:10So, and there's another thing that's happening, which is new speculative decode methods, and new quants of those are coming out every single day. Also, there was a day last week where we had uh, 2 new speculative decode.
6:26Technologies drop on the same day. I think it was Eagle and something else. So, yeah. That's going to offer more trade-offs of VRAM for performance, or straight up just be improvements over the existing defaults.
6:39So, let's take a look at the hardware. And we're going to kick this off by looking at this team hardware survey. You got a gaming PC, DDR 4, DDR 5, probably Ryzen 7, or Intel i9, or newer, anything from like the last 10 years will do.
6:55Eight cores is nice. My main inference box is Ryzen 7. is definitely not new. My other box is an i9. And the i9 can be DDR 4 or DDR 5, minus the older DDR 4 variant. And this is a very, very common configuration and popular configuration on steam.
7:14And this episode should help you understand what you can expect. So, 32 gigs of RAM, 16 gig GPU. This is like the median gaming PC. So for 5060 Ti owners. This is for the folks for the 5080, 4080, 4070 Ti, 5070 Ti. You get the picture.
7:33There's a lot of variants like this. I have a 5080 myself, and additionally, uh, these 2 configuration should work with no issues at all on AMD or Intel cards like the AMD 9070 XT and the Intel B770. Now, these are just absolutely non-existent among the steam hardware survey and counterstrike players, but, um, as far as I know, support for inference frameworks is a little bit more experimental on Intel and AMD graphics cards, but I've heard that it works absolutely fine.
8:05So ideally, the box that we're talking about, you're running Linux on it, headless, and it's a dedicated server for this task. And Strata, if you decide to go that route and serve Qwen Flash Next, we'll do its absolute best, use every single resource on the device, CPU, GPU, PCIe bandwidth, SSD.
8:24Um, so I hope you're running Linux headless. If you plan on going that route because say you're running Windows on it and you want to use Chrome or Edge or whatever you Windows Savages use to search Bing for the latest clippy update, you're going to want the 27B dense variant because your system is going to be busy doing whatever else you're doing with the system memory.
8:49So, Strata. Again, this is the biggest step change I've seen in the past year, 10 months looking at local inference, in terms of boosting, what can I run on my own hardware? And there's 2 parts to Strata. One is this incredible, optimized inference engine and C, C ++ CUDA Python.
9:09It borrows heavily from llama. It's not a llama fork, keeps the GPU busy at all times, keeps the CPU. Busy, like we said. It's amazing. The second part of Strata is a bunch of bullshit, bundled website, bundled harness. It auto downloads binaries from GitHub during startup without asking.
9:27Guys, auto update on startup without consent. It's one compromised credential from a massive synchronized supply chain attack similar to what's happened to LiteLLM twice. So it recommends that you install it through curling a bunch of markdown in your agent of choice.
9:44A bunch of this stuff is just non-starters for me. I want a Swiss army knife. I want the good parts and the performant parts in a C++ binary that I can poke at, attach a debugger to, have my agents run a bunch of different tests on without all the annoyance of the packaging stuff within, within Strata.
10:05So, ain't nobody got time for that.
10:07I have forked Strata. I'm calling it UnbiasedStrata. I know, I know, I'm a genius at naming things. The goal of the fork is to be distributed as source, and then you copy the source to your server, which is completely offline, and then you run make.
10:20So, you're going to need to get a few things from the internet, which is going to be CUDA, your Nvidia drivers, you're going to need the model, and then it creates like a pack. It creates a new version of the speculative head and quantizes it.
10:38Uh, and then once you have those binaries, which is the model, the pack, the spec head, and the, uh, you know, the developer libraries on your machine, you can just run make, and then it will never reach out to the internet again.
10:55This is a much more palatable approach to an inference framework for me. So, I, like I said, I want the performance benefits of Strata without the packaging and distribution annoyances. Now, I feel a little bit conflicted, recommending something that I have created, and when we're going out there and we're trying to find something to run for inference at home, I think we should be really careful about running something that's being recommended by a content creator or a YouTuber.
11:19In this case, I don't feel too bad about this particular instance because it's MIT licensed. It's not a binary. It respects the upstream source. And if you use it properly and the way it's intended, I will never know because you will clone the source code.
11:34And then you will download your model, and then you'll copy the source code to a server that's completely cut off from the broader internet, and you'll run inference there. And there won't be any telemetry or auto updating or further reaching out to GitHub or any other server.
11:50So, and then that binary will sit there until the end of time and just serve requests. from, you know, we'll just serve requests from your machine.
12:02So, as of today, I want to reiterate this. I looked at the source code for Strata with an agent. And as of today, every time you start Strata. It looks for a new version of the binary and download it. In addition, additionally with DeepSeek, every time you run it with DeepSeek, it looks for a new version of that engine and downloads it.
12:27and runs that instead of whatever you had on disk. So this is an absolute non-starter for me, and it conflicts with the SAFETY.md notes in that repo where it says that it asks you first, it does not. So I recommend UnbiasedStrata report, or if you're going to run Strata, do it in a docker container and then wipe it when you're done using it for testing.
12:48So, I recommend UnbiasedStrata. Report bugs to me. I try to fix them. I don't intend to fix every single or merge every single one percent performance gain, but I do plan to maintain this as a Swiss Army knife offline first, no auto update bullshit way to run, Qwen Flash Next.
13:05Okay, so testing with Bob-Bench. I am mostly focused on this channel on how to do coding with LLMs, so we're not talking about chat bots or things like that. General purpose coding agents. And to me, the agents should be able to work in code bases that are common for humans and agents today.
13:22So for me, that means bash, C, C++, Python, Rust. And if you think about an agent that's allowed to call tools, they need to be able to write code to call the tools. This is a basic requirement. So my benchmark set reflects this.
13:36Agents are typically dumped into a fairly large code base, something like memcached has a lot of lines of C, it's a public framework. I do tests on memcached. And uh, so yeah, the agents get bug reports, they get performance issues, they get black box debugging, they get feature requests, and a corpus of 86 tasks.
13:56Everything is run on more or less the same hardware and the same harness version and everything that I can hold constant. I hold constant to keep as much of these arms comparable as possible. And I like to organize it by VRAM budget, okay, 8 gigabytes, 16 gigabytes, and 24 gigabytes are the arbitrary cutoff points for VRAM that I have chosen that I've been running the tests on.
14:21So agents are rewarded for completing tasks on time in my benchmark set also, 15 to 30 minute timeouts. If I can run DeepSeek V4 Flash at 4 tokens per second. It's going to get a 0 on my benchmark set. Actually, I think it'll get a 10.
14:34If you don't do anything, I think you get a 10%. So it doesn't mean it's a good trade-off. If it's possible to run the thing, but it's too slow. Um, so the timeouts are critical for evaluating models within that framework of actually using it.
14:50in real life. So, one thing I noticed is that the 27B and the QFN variant that I'm recommending in this video, both aced the same number of problems, and my task set that was 71. So they both got a 100% on 71 problems. However, QFN lost a bunch of points by timing out after it had solved the problem.
15:14So the model goes off, it analyzes, it looks at the source code, it writes a solution, it comes to the correct conclusion and writes code that passes the test, but then keeps cranking on and on and on, fuzzing, or writing more unit tests or whatever, and lost points by being too thorough.
15:34And so this is a new behavior in Qwen 3.8 and Qwen 4 and I think we saw this in Glimmer, where they do a lot of fuzzing on these more recent models. And so runtime takes a lot longer. And so honestly, we may need to reconsider these sort of timeouts and the penalty for timing out in this manner within Bob-Bench, but as of today, the policy, that is standard for every one of these runs is that if you get to the right answer, you get a 10 out of 10, and then if you time out because you're still working and you haven't yielded, at the 30 minute cutoff, you get -5 points.
16:15So Qwen got half points off on a significant number of problems by acing them and then working for too long. So if you watch this video, and you decide you're going to try to deploy Strata, consider keeping an eye on it, and adjusting your prompting to have it fuzz less, or do less testing, or just check in with you when it thinks that the work is complete before it goes off on a long verification journey.
16:39Okay, one problem with lower bit quants, and then we'll wrap up. I've seen quant damage and a small number of runs on coder and Q2 and IQ2 runs. And funnily enough, the Austrian Lingua Franca is German, but almost everyone speaks English.
16:52So how many countries is involved there? So the funniest run on that I've seen is when the model is doing a bunch of reasoning about numbers and gets stuck emitting nine. Now, I think this is also what the engineers at DASLab were yelling as they were examining the traces of the quantize model fucking up.
17:10999. So I'm serious. It has issues with run on nines. In my testing, I recommend IQ3 or greater to avoid this problem. And the, uh, DASLab guys reached out and commented on one of my live streams saying that they had updated the coder quant.
17:28And the coder quant is very small, REAP variant. And so apparently that's doing better.
17:35And we need to we need to look into that. So, I'm releasing this video without 100% of the data that we need, and man, in 2026, it is hard to evaluate everything fully. We haven't talked about new developments and speculative decoding, really.
17:50DASLab, you know, the coder quant needs to be reevaluated. We need to do further tuning of Strata. I've mainly been using it with the default parameters and then varying the different memory modes between arena and the other options.
18:04So, there is a bunch of further work to do. Even for me to understand how to run Strata optimally on the hardware that I have, not
18:16to mention a general purpose recommendation for everybody. But if I were to draw a line in the sand, and I think we should do this from time to time, I think we should hold ourselves to a bar of saying, here's what we understand about the space, and and say, here's a concrete recommendation.
18:32And guys, sometimes it's wrong. I got this completely wrong with the 12 gigabyte recommendation video on the channel from July, August. And so, but I think that that is an important part of being an evaluator in the public discourse is to put your hypothesis and your journey out there to be validated or disproved.
18:56So we're doing that today, and I've been delaying on it because I want to gather every single piece of information and test every variant, and you just can't. And so I said, all right, yeah, it's Tuesday, Qwen 4 coming out on Thursday.
19:11Let's let's get this thing out here and say what we think the best is right now.
19:19So, I'll wrap up with a little bit of update about the channel. Qwen 4 coming out. is rumored on Thursday, this is from an anonymous Twitter account. I don't have any contacts at Qwen. I have no information from the inside about it, but I'm planning on streaming Thursday as the weights and inference frameworks come out.
19:36Plan on being at the computer from 8 AM Pacific until whenever we get the full benchmark set in and can put it on the Bob-Bench Wide plot. So, join me on the live stream. If you would like to support the channel, it's enough to watch the video.
19:49I also post the full videos to my website also. Do consider subscribing, sharing the video with a friend. This actually helps the video and YouTube's
19:57algorithm reach more people quite a bit if you share the video with others. And a huge thank you goes out to our channel members, Kruz, zircon, soupwizard, Mitchell, dinferno, Eeee10, Dennis, psjoshooaj, Heroian, and Josh G.
20:11So, thank you for watching. See you in the next one.