I overlooked the power of offloading the experts on the Qwen 3.6 35B-A3B. Here: the 12 GB correction, 8 GB VRAM configs that lean on system memory, and Bob-Bench Wide.
Want a copy without ads, an account, or tracking? Grab the torrent (76.2 MB).
It carries a webseed,
so it downloads even when nobody else is online — and every person who
seeds makes it cheaper for me to keep serving. Please seed it if you can.
magnet link
Qwen3.6 35B-A3B at IQ3_XXS, 25 of 40 layers' experts in host RAM. Measured 7,962 MiB peak on a 24 GB card held to an 8 GB budget — 52.7 tok/s decode, 2,946 tok/s prefill.
llama-server \ --host 0.0.0.0 --port 8091 \ -m Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf \ -c 131072 \ -ngl 99 \ --n-cpu-moe 25 \ -b 2048 -ub 2048 \ --cache-type-k q4_0 --cache-type-v q4_0 \ --cache-ram 0 \ --parallel 1 \ --jinja \ --reasoning on --reasoning-budget 8192 \ --temp 0.7 --top-k 20 --top-p 0.8 --min-p 0.0 \ --presence-penalty 0.0 --repeat-penalty 1.0 \ --n-predict 32768
--n-cpu-moe is the flag that makes this work: it keeps the expert FFN weights of the first N layers in system RAM while everything else stays on the GPU.Auto-captions, lightly corrected for proper nouns. Timestamps jump to that point on YouTube.
0:00You can't swing a cat out there without hitting somebody who has an 8 gigabyte GPU sitting around. Hell on my desk right now. I've got a 2080, a 1070 TI, and a 1080 TI that I pulled out of my inference boxes for a 3090. So when I reviewed a 12 gigabyte configurations, I overlook something that fans of the channel, commenters on YouTube, and people who have the 8 gigabyte cards already knew, you can run awesome shit on these $50 devices.
0:26Let's round it up. Call it $70 including shipping.
0:30Here's the configuration I've been having a great time with and is scoring very well on the benchmark. You take the Qwen 35B-A3B. You crush it down to IQ3, and you put most of the experts into system memory. It slides right in there at under 8 gigs of memory used.
0:44Now, I didn't like discover this. Shout out to some people who had commented about this. Before and on the live streams. The point is this. If we take the 35B-A3B model from Qwen 3.6, most of it can be offloaded to system memory really gracefully. Okay, you lose some performance, but the intelligence is there, and you can crush it down to IQ3, and it doesn't it doesn't lose that much in terms of capability on these coding tasks that I give it.
1:11So the MTP is optional. llama.cpp is already sticking the token embedding into system memory for you, and then the bulk of the weights. I think it's 83% gets used at like a 3% rate on each token on each forward pass. So there's 256 of them, and you only need 8 experts are activated at a time.
1:33So stuff them into main memory.
1:36works great. Back to the 12 gigabyte video. I didn't think that a 35B-A3B Q2 could beat the 9B boys at a healthier quant. I tried a 27B squashing it down into VRAM entirely. It did terribly, largely because it's really slow to execute. So we'll do a full video on 8 gigabyte, 12 gigabyte RAM regime, but here's the baseline.
1:59This is the guy to beat. Okay, if you're into rock. This is the Roger Waters. If you like baseball, this is Roger Maris. It's Roger Federer, okay? I'm going to nickname this model Roger because it's so fucking good. If you're collecting sweaters, this is fucking Mr. Rogers, okay?
2:14Why are there so many good Rogers? So 35B-A3B Qwen 3.6 is the best, and we just went over Y. Large percentage of experts that can be offloaded to system memory. graceful degradation in performance as you go down that curve, and yeah, it's just, it's awesome. And I under appreciated how well it performs at small quants, which makes it an excellent fit for low VRAM setups.
2:40So we're going to test out some more, we're going to get the abliterated, the uncensored versions going, just not the Huihui again, the thing's terrible. So I'll try the HauhauCS, or however you pronounce that. Well, maybe we'll try a DavidAU model. He's the Fable Fusion guy.
2:54We're going to try a REAP, and there will be a full models new review of the 8 gigabyte VRAM 12 gigabyte RAM envelope. But I do have partial results now, and this thing absolutely destroys the 9B configs in the 12 gigabyte video. It finishes in half the time with a higher score.
3:11And then, you know, we could look into how performance degrades as we offload each individual layer, and we could like step through, um, you know, what happens as one, two, three, up, you know, and all the way up to 40 layers gets offloaded to system memory, but it would be tough to automate and it would take too long to run, so I didn't do it.
3:33I'm just kidding, guys.
3:34So here's the graph. And this is a sweep of each layer, as you offload the experts for that layer into system memory, and the amount of VRAM in the speed, and you can see it really drops off a clip as you offload the 1st one or 2 layers, but, you know, the resulting place you're at, you're still getting one or 2000 prompt processing per 2nd, and the decode speed is a completely reasonable, like 50 to 70.
3:58It's a fully interactive. It's it's awesome. So I had to test this out myself and the speeds feel good in addition to looking good on paper. Um, and I had this guy write a graphing calculator. I had tried this test before with the IQ3 version of the 27B model to...
4:17But, you know, before we close out, we got to talk about the BBW in the room. So you guys have been demanding it. Check the comments. You guys have been asking for this. Wait, nobody asked for this? Anyways, you're going to get it. Bob-Bench Wide Alpha, the full version, all the tasks, 86 of them, ready to 86 any models that aren't going to make the cut.
4:44So I'm releasing Bob-Bench Wide Alpha. Now, this is an apples to apples comparison of all the models that we've talked about on the channel, and I'm drawing a line in the sand and saying, I'm not going to add any more tasks for now, and this will be the evaluation point for the local models that we have talked about.
5:02And so it's going to have the same time limits and the same hardware for each task. And this is also going to allow us to make better hardware decisions and figure out how much you have to pay to get a better GPU, to shift the curve up into the left, get better accuracy and faster time.
5:20So I'm really excited about this and kind of saying. in terms of the quality of the individual task and start to incorporate more data points that are comparable. Because if you look at a video that I made 2 weeks ago, it'll have fewer tasks and easier tasks. And so, um, this is going to be the 1st time where I say, look, this is this is good enough to test these 30 B class models and we'll get a full data set to compare.
5:51So that's all I have for today. Guys, we do have a discord now. I put graphs in there as I'm making videos and you'll see discussion about various benchmarking tools and folks are chatting about local AI in there. It's very small, so please join us. And yeah, subscribe if you want to, and I will see you in the next one.