The model to run on a 16 GB card, entirely resident in VRAM — nothing offloaded, no system memory in play. Plus the abliterated configs, how the models were tested, and live iOS coding on a Q3 quant.
Want a copy without ads, an account, or tracking? Grab the torrent (118.1 MB).
It carries a webseed,
so it downloads even when nobody else is online — and every person who
seeds makes it cheaper for me to keep serving. Please seed it if you can.
magnet link
Qwen3.6-27B at UD-IQ3_XXS, fully resident — no MoE offload, nothing in system RAM. Measured 14,990 MiB at the full 128k context, which leaves roughly 500 MiB of headroom on a 16 GB card. Drop to 98k for 14,254 MiB, or 64k for 13,518 MiB.
llama-server \ --host 0.0.0.0 --port 8091 \ -m Qwen3.6-27B-UD-IQ3_XXS.gguf \ -c 131072 \ -ngl 99 \ --cache-type-k q4_0 --cache-type-v q4_0 \ --cache-ram 0 \ --parallel 1 \ --jinja \ --reasoning on --reasoning-budget 8192 \ --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0.0 \ --presence-penalty 0.0 --repeat-penalty 1.0 \ --n-predict 32768
Auto-captions, lightly corrected for proper nouns. Timestamps jump to that point on YouTube.
0:00Hello everyone, unbiased Bob here bringing you local AI news and memes. Today I've got something I'm pretty excited about. It's a 16 gigabyte VRAM budget recommended configuration. If you want a smart coding assistant. You want to run it locally, entirely resident within the VRAM on your graphics card.
0:15I've tested a bunch of different configurations today. We've got like 2000 tokens per second prompt processing and 50 to 60 or even higher decode speed tokens per second output. Get yourself a Qwen, 27B. It's the IQ3 quant. We're going for MTP off, 128K context, and this is a great configuration.
0:37If you want the 3.8 model, you really like Qwen, 27B, 3.8, Got the GSQ-RCO from Austria, that's an excellent configuration, very similar performance. It outperforms the base Qwen Q3 in my experience on the set of benchmarks today.
0:56I've also got some bonus configs. If you want to do red team or, you know, reverse engineering work, you're going to want one of the abliterated versions. I tested out two of them. They have very similar names. It's the Huihui and the Hauhau. One of them gets an 8% score.
1:11on my benchmark set at the IQ2 heavily quantized 35B-A3B. And the other one gets a very reasonable score, but still degraded performance. I think when you're talking about obliteration, the more quantized they are, the worse these perform compared to their, you know, fuller counterparts.
1:30So I ran each model through a grueling task set of 86 different software engineering programming challenges. Glimmer failed. It was too verbose. Ridge failed, didn't make the cut, dropping down to a much smaller model at a higher quant and a higher KV cache quant.
1:47That didn't work either, very poor performance at the 9B, um, at the 9B size here. So there's three real options on the frontier and the one that we haven't really talked about yet is the most interesting one and it's the Qwen 35B-A3B base at the IQ3_XXS quant.
2:05Okay, this thing fits at 256K context.
2:09Pull up the graph. So we need to take a look at this. This is how much RAM you can cram into 16 gigabytes for each model. This is measured. It's not interpolated, extrapolated, isolated, or any other kind of pollated. This is the nice, gentle slope on Glimmer and Qwen 35B-A3B.
2:26This is probably where we want to be. Now, unfortunately, the Qwen 35B. It is dumber than the 27B and the scores do reflect that. So, but regardless, today, we're covering this 16 gigabyte VRAM, zero gigabytes of RAM config. So there's bound to be a better config for some use cases, if you can offload a bunch of stuff, especially with the mixture of experts model into system RAM.
2:51You know that stuff that you can't buy anymore. I actually looked into upgrading my gaming PC, putting 128 gigs of RAM in that thing, and it's actually cheaper to buy a home in Seattle than to do that right now. The market's really incredible out there.
3:03Okay, so what kind of tasks are involved? What kind of capabilities can you expect here? This eval set has the model reading between 20 and 30,000 lines of code. It's performing about 2000 tool calls on average per model. It's reading a bunch of legacy code, writing a ton of code, emitting 700K to about 3 million on the top end tokens for these models on this task set.
3:28These are gnarly programming problems. Okay, taking a working virtual machine, given to the model in Python and having them converted into C. This requires writing memory, safe code and pointers. The evaluation suite checks if the models write memory safe code by testing it with Val- grind or something like that, and these top configs ace it, web dev, C, C++, game development, particle simulation.
3:54This is all in the automated task set. There is not anything right now related to Windows development, iOS or Android. These tasks run in Docker. So I don't have that set up right now.
4:05I did not believe that a Q3 quant could be this intelligent and win the best score and the best time on this benchmark set. I fired it up and did some live development on YouTube, and we did a graphing calculator run in two sessions, about an hour's worth of time interacting with the model, and I was very impressed.
4:27And this is, you know, to complement this automated benchmark set that I like to present to you guys. I like to do qualitative work with these models and check it out. if I'm gonna run it. And so this was the first time I had actually done it live on stream, and I thought it was really fun to kind of go through it interactively, and it gives you a sense of what it's really like to sit down, because a lot of people know what it's like to program with Claude, and I think, and I'm not certain about this, but I think that a lot of viewers on my channel are kind of curious about what it's like to do this stuff locally.
4:55In addition to a lot of you guys who I know are experts and are testing out these models and know how it works and use it frequently. So, um, But yeah, I do like to add in that qualitative bit. So we'll show a little bit of that footage from what it was like to prompt it into creating and modifying an Xcode project and all the stuff that comes along with making an iOS app with the native tools on macOS.
5:20So one more thing. I previously did a video like this for the 12 gigabyte regime and it was wrong. Now, it had a significant omission. We didn't look at the heavily quantized 35B-A3B because it fits in a 12 gigabyte card. Now, if you go back and watch that video, it looks at a bunch of 9B configurations, and then I tried one larger quant, and I think it was a 27B model and it just didn't do well on the 12 gigabyte card, but the 35B-A3B does.
5:54And so shout out to folks like Gabe Hanson, Sonic Snail 71, Brigandier, Lewis D 101, you guys were in the comments correcting me on this, saying, I use the 35B-A3B on my 12 gigabyte card, and it works well for me. So thank you. We're going to have to do an update on that.
6:10And yeah, I just totally missed this. So I think the moral that I'm going to take away is to use the spirit of Kaizen and continuous improvement in trying not to make the same mistake twice.
6:21To try these extreme quants, I sort of counted out that an IQ2 or an IQ3 uh, XXS variant could do well. Uh, and and they do, they do perform very well. And then the other thing I want to get explicit about is showing these configs with a VRAM and a RAM budget. Okay, so this is a 16 gigabyte VRAM, zero gigabyte RAM variant that we're going through today.
6:47But realistically, if you're deploying this thing, you're going to have a bunch of requirements and constraints and you're probably going to have the ability to dedicate some system RAM to inference if you're deploying on a 16 gigabyte card. So, I'd like to cover, for example, a 16 gigabyte VRAM, 8 gigabyte RAM setup, which you could offload the vision projection.
7:11You can offload a bunch of the experts and degrade your performance a little bit, but run a larger quant and get better results. So anyways, thanks for watching. Subscribe if you want to, and I will see you in the next one.