0:00  Say you want to turn your 16 gigabyte VRAM gaming PC
0:02  into an inference platform so you can code at home using
0:05  intelligence that's not that far off from what the frontier was
0:08  6 months ago, powered by the hardware that
0:11  you already have with no data leaving the house.
0:14  Here are the models and inference frameworks that I recommend.
0:18  It depends on how much system RAM you have available.
0:21  We're considering between 16 gigabytes and 128 gigs
0:24  of system RAM here, alongside your GPU.
0:26  If you have 32 gigabytes or less, I'm going to recommend Qwen 27B.
0:31  3.6 or 3.8.
0:33  The best performing variants I tested were the Swift 1.5
0:36  at the GSQ-RCO quant, expect over 70
0:39  tokens per second here, and this is a pretty smart coding agent.
0:43  And with the GSQ-RCO quant.
0:46  It performs really well at Q3.
0:49  So if you have an immense amount of RAM available, say
0:52  64 gigabytes dedicated to the task, or
0:55  even better, 128 gigs, you can get
0:59  better performance by downloading China's new hot model, Qwen
1:02  3.8 Flash Next.
1:03  This model and the framework that's largely popularizing it, is
1:07  taking everything by storm, is much larger, and it takes a lot
1:10  more resources to run.
1:12  So, With 128 gigs,
1:16  Grab the IQ3_XXS quant, it's going to take longer to run.
1:19  It'll use all the cores on your machine.
1:22  GPU will be at full utilization too.
1:24  You can even bump up to the IQ3_S quant.
1:27  Use Strata or a fork of Strata.
1:29  I have a fork that removes all the Python and bullshit to make it much like a
1:32  compiled C++ binary is possible.
1:36  This model is going to be significantly smarter, and it will take
1:40  about 30% longer than the 27B dense to complete tasks.
1:43  So expect around 90 tokens per second and a higher level
1:46  of peak intelligence.
1:49  Table of contents.
1:50  Are we going to go over in this video?
1:52  That was the intro and the configs.
1:55  We showed them on screen.
1:56  We're going to talk about developments in the space.
1:58  Why is this stuff moving so fast and what are the developments that are impacting us most?
2:02  In terms of our ability to run this stuff at home?
2:05  Second is a little bit more detail on the hardware and
2:08  configurations that I'm thinking about, making this recommendation.
2:12  There it is Strata.
2:14  This new framework is largely synonymous with running QFN,
2:18  and we need to talk about it in a little bit more detail, even though I
2:21  have a dedicated video on the subject on the channel.
2:23  How I tested these configurations, we'll talk about
2:26  Bob-Bench, how I measure this, and what my priorities
2:29  are, when I'm evaluating a model, and why I think it
2:32  might be generally relevant to look at C/C++ coding.
2:36  And then I've got a couple issues with lower bit quants that we'll look at.
2:40  And basically they have damage at Q2.
2:44  And then further work, additional knobs.
2:48  that we can turn an additional performance, tweaks
2:52  and measurements that we should be doing.
2:55  So why is this stuff changing so fast?
2:57  What the hell is going on?
2:58  2026, you wake up, every day there's a new
3:01  framework, a new thing.
3:03  I think the biggest change in what has been
3:06  the largest shift that I've seen in
3:09  terms of a step function up in what you can run at
3:12  home for local intelligence for coding is Qwen Flash Next.
3:16  This is a preview of the Qwen 4 architecture.
3:18  It has a large n-gram table that effectively increases
3:21  the amount of state that the model has to work with without
3:25  increasing the amount of VRAM that you need or the amount
3:28  of RAM that you need, which is rather expensive.
3:31  So this new MoE architecture, and others like it, are
3:34  making it much easier to run much larger
3:37  models at home.
3:40  Uh, with great performance.
3:43  So we're seeing specialization.
3:45  winning.
3:46  And that brings us to the second thing
3:50  that's developing really fast and it's the inference frameworks, okay?
3:53  So Strata, ninfer, freetoken.
3:55  Specialization seems to be winning. Dwarf Star, which
3:58  runs DS4.
4:00  Um, llama.cpp is not the best performance.
4:04  I think it was generally assumed that llama.cpp
4:07  CPP had excellent performance for most models.
4:11  This is a general purpose framework.
4:13  It's largely C, C++ CUDA.
4:16  It's definitely not true today.
4:18  And they are so far behind what
4:21  Strata is capable of doing in terms of running larger MoE
4:24  models at higher tokens per second on consumer hardware.
4:28  So, that's the second thing I'm seeing that's making it
4:31  so, it's difficult to track what's
4:34  best to run on your, on your, on whatever
4:37  GPU you happen to have.
4:38  So, and then third, and
4:41  these are all hitting so hard.
4:43  The third new quantization techniques.
4:45  Okay, so GSQ-RCO, Gumbel
4:48  softmax quantization, and Riemannian constrained
4:52  optimization, these are ways to quantize the weights,
4:56  and then optimally pick which weights to quantize how
4:59  hard, given a budget, in the end resulting model.
5:02  And these are making it so that Q3 is
5:05  an excellent high quality quant, which I don't think was true before.
5:09  So it's bringing up the size of the model that can be run, and
5:12  it's bringing down the amount of VRAM that you need to run it.
5:15  And so DASLab, the Austrian dudes who make this, doing an
5:19  absolutely fantastic job and releasing these GSQ-RCO
5:23  quants, which work with the existing GGUF format.
5:27  Um, absolutely banger quants and they work in llama.cpp
5:31  or whatever other framework uses a GGUF.
5:34  So, additionally, number
5:37  4th development that keeps happening, new fine-tunes.
5:41  And so Swift 1.5
5:42  is the first Qwen fine-tune that I
5:45  have tested that absolutely dominates the
5:49  base model, base Qwen.
5:51  It's excellent.
5:52  I have no idea what they fed this thing, but
5:55  so the fine-tune.
5:56  So we've got the architecture, the inference frameworks, new quantizations and new fine-tunes.
6:00  All 4 of these developments have been in the last couple
6:04  weeks and are influencing the recommendations.
6:07  And the benchmarking that I'm doing at a given hardware budget today.
6:10  So, and there's another thing that's
6:13  happening, which is new speculative decode methods,
6:17  and new quants of those are coming out every single day.
6:20  Also, there was a day last week where we had uh, 2 new
6:23  speculative decode.
6:26  Technologies drop on the same day.
6:27  I think it was Eagle and something else.
6:29  So, yeah.
6:31  That's going to offer more trade-offs of VRAM for performance,
6:35  or straight up just be improvements over the existing defaults.
6:39  So, let's take a look at the hardware.
6:42  And we're going to kick this off by looking
6:45  at this team hardware survey.
6:47  You got a gaming PC, DDR
6:50  4, DDR 5, probably Ryzen 7, or Intel i9,
6:53  or newer, anything from like the last 10 years will do.
6:55  Eight cores is nice.
6:56  My main inference box is Ryzen 7.
6:58  is definitely not new.
7:00  My other box is an i9.
7:03  And the i9 can be DDR 4 or DDR 5,
7:06  minus the older DDR 4 variant.
7:08  And this is a very, very common configuration and
7:11  popular configuration on steam.
7:14  And this episode should help you understand what you can expect.
7:18  So, 32 gigs of RAM,
7:21  16 gig GPU.
7:23  This is like the median gaming PC.
7:25  So for 5060 Ti owners.
7:28  This is for the folks for the 5080, 4080, 4070 Ti, 5070 Ti.
7:32  You get the picture.
7:33  There's a lot of variants like this.
7:34  I have a 5080 myself, and additionally,
7:38  uh, these 2 configuration should work with no issues at all on
7:41  AMD or Intel cards like the AMD 9070 XT and
7:46  the Intel B770.
7:48  Now, these are just absolutely non-existent among
7:51  the steam hardware survey and counterstrike players, but,
7:54  um, as far as I know, support for inference
7:57  frameworks is a little bit more experimental on Intel and AMD
8:00  graphics cards, but I've heard that it works absolutely fine.
8:05  So ideally, the box that we're talking about, you're running
8:08  Linux on it, headless, and it's a dedicated server for this task.
8:12  And Strata, if you decide to go that route and serve
8:15  Qwen Flash Next, we'll do its absolute best, use
8:18  every single resource on the device, CPU, GPU,
8:22  PCIe bandwidth, SSD.
8:24  Um, so I hope you're running Linux headless.
8:28  If you plan on going that route because say
8:31  you're running Windows on it and you want to use Chrome or
8:34  Edge or whatever you Windows Savages use to search
8:38  Bing for the latest clippy update, you're
8:42  going to want the 27B dense variant because your system is going to
8:45  be busy doing whatever else you're doing with the system memory.
8:49  So, Strata.
8:51  Again, this is the biggest step change I've seen in
8:55  the past year, 10 months looking
8:58  at local inference, in terms of boosting,
9:01  what can I run on my own hardware?
9:02  And there's 2 parts to Strata.
9:05  One is this incredible, optimized inference engine and C, C
9:08  ++ CUDA Python.
9:09  It borrows heavily from llama.
9:11  It's not a llama fork, keeps the GPU busy at all
9:14  times, keeps the CPU.
9:15  Busy, like we said.
9:17  It's amazing.
9:19  The second part of Strata is a bunch of bullshit, bundled website,
9:22  bundled harness.
9:24  It auto downloads binaries from GitHub during startup without asking.
9:27  Guys, auto update on startup without consent.
9:30  It's one compromised credential from a massive
9:33  synchronized supply chain attack similar to
9:36  what's happened to LiteLLM twice.
9:39  So it recommends that you install it through curling
9:42  a bunch of markdown in your agent of choice.
9:44  A bunch of this stuff is just non-starters for me.
9:47  I want a Swiss army knife.
9:48  I want the good parts and the performant parts in
9:52  a C++ binary that I can poke at,
9:55  attach a debugger to, have my agents run a bunch of different
9:58  tests on without all the annoyance of
10:01  the packaging stuff within, within Strata.
10:05  So, ain't nobody got time for that.
10:07  I have forked Strata.
10:08  I'm calling it UnbiasedStrata.
10:10  I know, I know, I'm a genius at naming things.
10:12  The goal of the fork is to be distributed as source,
10:15  and then you copy the source to your server, which is
10:18  completely offline, and then you run make.
10:20  So, you're going to need to get a few things from the internet,
10:23  which is going to be CUDA, your Nvidia
10:27  drivers, you're going to need the model, and then it
10:30  creates like a pack.
10:33  It creates a new version of the speculative head and
10:36  quantizes it.
10:38  Uh, and then once you have those binaries,
10:41  which is the model, the pack, the
10:45  spec head, and the, uh, you
10:48  know, the developer libraries on your machine, you
10:51  can just run make, and then it will never reach out to the internet again.
10:55  This is a much more palatable approach to an inference framework for me.
10:59  So, I, like I said, I want the performance benefits of
11:02  Strata without the packaging and distribution annoyances.
11:05  Now, I feel a little bit conflicted, recommending something that I have created,
11:09  and when we're going out there and we're trying to
11:12  find something to run for inference at home, I think we should be really careful
11:15  about running something that's being recommended by a content creator or a YouTuber.
11:19  In this case, I don't feel too bad about this particular instance
11:22  because it's MIT licensed.
11:24  It's not a binary.
11:26  It respects the upstream source.
11:28  And if you use it properly and the way it's intended, I
11:31  will never know because you will clone the source code.
11:34  And then you will download your model,
11:38  and then you'll copy the source code to a server that's completely cut
11:41  off from the broader internet, and you'll run inference there.
11:44  And there won't be any telemetry or auto updating or further
11:47  reaching out to GitHub or any other server.
11:50  So, and then that binary will sit there until the end of
11:54  time and just serve requests.
11:56  from, you know, we'll just serve requests from your machine.
12:02  So, as of today, I want to reiterate this.
12:06  I looked at the source code for Strata with an agent.
12:09  And as of today, every time you start Strata.
12:12  It looks for a new version of the binary and download it.
12:16  In addition, additionally with DeepSeek,
12:20  every time you run it with DeepSeek, it looks
12:23  for a new version of that engine and downloads it.
12:27  and runs that instead of whatever you had on disk.
12:29  So this is an absolute non-starter for me, and
12:32  it conflicts with the SAFETY.md notes
12:36  in that repo where it says that it asks you first, it does not.
12:39  So I recommend UnbiasedStrata report,
12:43  or if you're going to run Strata, do it in a docker container and
12:46  then wipe it when you're done using it for testing.
12:48  So, I recommend UnbiasedStrata.
12:51  Report bugs to me.
12:52  I try to fix them.
12:53  I don't intend to fix every single or merge every single one percent performance
12:56  gain, but I do plan to maintain this as a Swiss Army knife
12:59  offline first, no auto update bullshit way
13:02  to run, Qwen Flash Next.
13:05  Okay, so testing with Bob-Bench.
13:08  I am mostly focused on this channel on how to do coding
13:11  with LLMs, so we're not talking about chat bots or things like that.
13:14  General purpose coding agents.
13:16  And to me, the agents should be able to work in
13:19  code bases that are common for humans and agents today.
13:22  So for me, that means bash, C, C++, Python, Rust.
13:26  And if you think about an agent that's allowed to call tools, they need to be able to write
13:30  code to call the tools.
13:32  This is a basic requirement.
13:33  So my benchmark set reflects this.
13:36  Agents are typically dumped into a fairly large code
13:39  base, something like memcached has a lot of lines of
13:42  C, it's a public framework.
13:44  I do tests on memcached.
13:46  And uh, so yeah, the agents get bug reports,
13:50  they get performance issues, they get black box debugging,
13:53  they get feature requests, and a corpus of 86 tasks.
13:56  Everything is run on more or
13:59  less the same hardware and the same harness version and everything
14:03  that I can hold constant.
14:04  I hold constant to keep as
14:07  much of these arms comparable as possible.
14:10  And I like to organize it by VRAM budget, okay, 8 gigabytes,
14:14  16 gigabytes, and 24 gigabytes are the arbitrary cutoff points for
14:17  VRAM that I have chosen that I've been running the tests on.
14:21  So agents are rewarded for completing
14:24  tasks on time in my benchmark set also, 15 to 30 minute timeouts.
14:27  If I can run DeepSeek V4 Flash at 4 tokens per second.
14:31  It's going to get a 0 on my benchmark set.
14:33  Actually, I think it'll get a 10.
14:34  If you don't do anything, I think you get a 10%.
14:37  So it doesn't mean it's a good trade-off.
14:39  If it's possible to run the thing, but it's too slow.
14:43  Um, so the timeouts are critical for evaluating
14:47  models within that framework of actually using it.
14:50  in real life.
14:51  So, one thing I noticed is that the 27B
14:55  and the QFN variant that I'm recommending in
14:58  this video, both aced the same number of problems, and my task set
15:01  that was 71.
15:02  So they both got a 100% on 71 problems.
15:07  However, QFN lost
15:11  a bunch of points by timing out after it had solved the problem.
15:14  So the model goes off, it analyzes,
15:18  it looks at the source code, it writes a solution, it
15:22  comes to the correct conclusion and writes code that passes the test,
15:25  but then keeps cranking on and on and on, fuzzing, or
15:28  writing more unit tests or whatever, and lost points
15:32  by being too thorough.
15:34  And so this is a new behavior in Qwen
15:38  3.8 and Qwen 4
15:41  and I think we saw this in Glimmer, where they
15:44  do a lot of fuzzing on these more recent models.
15:47  And so runtime takes a lot longer.
15:49  And so honestly, we may need to reconsider
15:52  these sort of timeouts and the penalty for
15:55  timing out in this manner within Bob-Bench,
15:58  but as of today, the policy, that is standard for
16:01  every one of these runs is that if you get to the right answer, you get
16:04  a 10 out of 10, and then if you time
16:07  out because you're still working and you haven't yielded, at
16:11  the 30 minute cutoff, you get -5 points.
16:15  So Qwen got half points off on a significant
16:18  number of problems by acing them and then working for too long.
16:21  So if you watch this video, and you decide you're
16:24  going to try to deploy Strata, consider keeping an eye on it,
16:28  and adjusting your prompting to have it fuzz less, or
16:31  do less testing, or just check in with you when
16:34  it thinks that the work is complete before it goes off on a long
16:37  verification journey.
16:39  Okay, one problem with lower bit quants, and then we'll wrap up.
16:42  I've seen quant damage and a small number of runs on coder and Q2
16:45  and IQ2 runs.
16:47  And funnily enough, the Austrian Lingua Franca is German,
16:50  but almost everyone speaks English.
16:52  So how many countries is involved there?
16:55  So the funniest run on that I've seen is when the model
16:58  is doing a bunch of reasoning about numbers and gets stuck emitting nine.
17:02  Now, I think this is also what the engineers at DASLab
17:05  were yelling as they were examining the traces of
17:08  the quantize model fucking up.
17:10  999.
17:11  So I'm serious.
17:12  It has issues with run on nines.
17:15  In my testing, I recommend IQ3 or greater
17:19  to avoid this problem.
17:21  And the, uh, DASLab guys reached out and
17:24  commented on one of my live streams saying that they had updated the coder quant.
17:28  And the coder quant is very small, REAP variant.
17:32  And so apparently that's doing better.
17:35  And we need to we need to look into that.
17:37  So, I'm releasing this video without 100%
17:40  of the data that we need, and man, in 2026,
17:44  it is hard to evaluate everything fully.
17:46  We haven't talked about new developments and speculative decoding, really.
17:50  DASLab, you know, the coder quant
17:53  needs to be reevaluated.
17:55  We need to do further tuning of Strata.
17:57  I've mainly been using it with the default parameters and then varying
18:00  the different memory modes between arena and the other options.
18:04  So, there is a bunch of further work to do.
18:09  Even for me to understand how to run Strata
18:13  optimally on the hardware that I have, not
18:16  to mention a general purpose recommendation for everybody.
18:19  But if I were to draw a
18:22  line in the sand, and I think we should do this from time to time, I think we should hold ourselves to
18:25  a bar of saying, here's what we understand about the space, and
18:29  and say, here's a concrete recommendation.
18:32  And guys, sometimes it's wrong.
18:34  I got this completely wrong with the 12 gigabyte recommendation
18:37  video on the channel from July, August.
18:40  And so, but I think that that is
18:43  an important part of being an evaluator
18:47  in the public discourse is to put your
18:50  hypothesis and your journey out
18:53  there to be validated or disproved.
18:56  So we're doing that today, and I've
19:00  been delaying on it because I want to gather every single piece of information and
19:04  test every variant, and you just can't.
19:06  And so I said, all right, yeah, it's Tuesday,
19:09  Qwen 4 coming out on Thursday.
19:11  Let's let's get this thing out here and say what
19:15  we think the best is right now.
19:19  So, I'll wrap up with a little bit of update about the channel.
19:22  Qwen 4 coming out.
19:23  is rumored on Thursday, this is from an anonymous Twitter account.
19:27  I don't have any contacts at Qwen.
19:29  I have no information from the inside about it, but I'm
19:32  planning on streaming Thursday as the weights and inference frameworks come out.
19:36  Plan on being at the computer from 8 AM Pacific until whenever we
19:39  get the full benchmark set in and can put it on the Bob-Bench Wide plot.
19:43  So, join me on the live stream.
19:46  If you would like to support the channel, it's enough to watch the video.
19:49  I also post the full videos to my website also.
19:52  Do consider subscribing, sharing the video with a friend.
19:54  This actually helps the video and YouTube's
19:57  algorithm reach more people quite a bit if you share the video with others.
20:00  And a huge thank you goes out to our channel members,
20:04  Kruz, zircon, soupwizard, Mitchell, dinferno,
20:07  Eeee10, Dennis, psjoshooaj, Heroian,
20:10  and Josh G.
20:11  So, thank you for watching.
20:14  See you in the next one.