WEBVTT 00:00:00.000 --> 00:00:02.970 Say you want to turn your 16 gigabyte VRAM gaming PC 00:00:02.970 --> 00:00:05.939 into an inference platform so you can code at home using 00:00:05.973 --> 00:00:08.942 intelligence that's not that far off from what the frontier was 00:00:08.976 --> 00:00:11.945 6 months ago, powered by the hardware that 00:00:11.979 --> 00:00:14.948 you already have with no data leaving the house. 00:00:14.982 --> 00:00:17.951 Here are the models and inference frameworks that I recommend. 00:00:18.986 --> 00:00:21.955 It depends on how much system RAM you have available. 00:00:21.989 --> 00:00:24.958 We're considering between 16 gigabytes and 128 gigs 00:00:24.992 --> 00:00:26.960 of system RAM here, alongside your GPU. 00:00:26.994 --> 00:00:30.964 If you have 32 gigabytes or less, I'm going to recommend Qwen 27B. 00:00:31.999 --> 00:00:33.967 3.6 or 3.8. 00:00:33.967 --> 00:00:36.937 The best performing variants I tested were the Swift 1.5 00:00:36.970 --> 00:00:39.940 at the GSQ-RCO quant, expect over 70 00:00:39.973 --> 00:00:43.944 tokens per second here, and this is a pretty smart coding agent. 00:00:43.977 --> 00:00:45.946 And with the GSQ-RCO quant. 00:00:46.980 --> 00:00:48.949 It performs really well at Q3. 00:00:49.983 --> 00:00:52.953 So if you have an immense amount of RAM available, say 00:00:52.986 --> 00:00:55.989 64 gigabytes dedicated to the task, or 00:00:55.989 --> 00:00:58.959 even better, 128 gigs, you can get 00:00:59.993 --> 00:01:02.963 better performance by downloading China's new hot model, Qwen 00:01:02.996 --> 00:01:03.964 3.8 Flash Next. 00:01:03.997 --> 00:01:07.968 This model and the framework that's largely popularizing it, is 00:01:07.968 --> 00:01:10.938 taking everything by storm, is much larger, and it takes a lot 00:01:10.971 --> 00:01:11.939 more resources to run. 00:01:12.973 --> 00:01:16.944 So, With 128 gigs, 00:01:16.944 --> 00:01:19.947 Grab the IQ3_XXS quant, it's going to take longer to run. 00:01:19.980 --> 00:01:21.949 It'll use all the cores on your machine. 00:01:22.983 --> 00:01:24.952 GPU will be at full utilization too. 00:01:24.985 --> 00:01:26.954 You can even bump up to the IQ3_S quant. 00:01:27.988 --> 00:01:29.957 Use Strata or a fork of Strata. 00:01:29.990 --> 00:01:32.960 I have a fork that removes all the Python and bullshit to make it much like a 00:01:32.993 --> 00:01:35.963 compiled C++ binary is possible. 00:01:36.997 --> 00:01:40.968 This model is going to be significantly smarter, and it will take 00:01:40.968 --> 00:01:43.937 about 30% longer than the 27B dense to complete tasks. 00:01:43.971 --> 00:01:46.940 So expect around 90 tokens per second and a higher level 00:01:46.974 --> 00:01:47.941 of peak intelligence. 00:01:49.977 --> 00:01:50.978 Table of contents. 00:01:50.978 --> 00:01:51.945 Are we going to go over in this video? 00:01:52.980 --> 00:01:54.948 That was the intro and the configs. 00:01:55.983 --> 00:01:56.950 We showed them on screen. 00:01:56.984 --> 00:01:58.952 We're going to talk about developments in the space. 00:01:58.986 --> 00:02:02.956 Why is this stuff moving so fast and what are the developments that are impacting us most? 00:02:02.990 --> 00:02:04.958 In terms of our ability to run this stuff at home? 00:02:05.993 --> 00:02:08.962 Second is a little bit more detail on the hardware and 00:02:08.996 --> 00:02:12.966 configurations that I'm thinking about, making this recommendation. 00:02:13.000 --> 00:02:14.968 There it is Strata. 00:02:14.968 --> 00:02:17.938 This new framework is largely synonymous with running QFN, 00:02:18.972 --> 00:02:21.942 and we need to talk about it in a little bit more detail, even though I 00:02:21.975 --> 00:02:23.944 have a dedicated video on the subject on the channel. 00:02:23.977 --> 00:02:26.947 How I tested these configurations, we'll talk about 00:02:26.980 --> 00:02:29.950 Bob-Bench, how I measure this, and what my priorities 00:02:29.983 --> 00:02:32.953 are, when I'm evaluating a model, and why I think it 00:02:32.986 --> 00:02:36.957 might be generally relevant to look at C/C++ coding. 00:02:36.990 --> 00:02:39.960 And then I've got a couple issues with lower bit quants that we'll look at. 00:02:40.994 --> 00:02:44.965 And basically they have damage at Q2. 00:02:44.998 --> 00:02:47.968 And then further work, additional knobs. 00:02:48.969 --> 00:02:52.939 that we can turn an additional performance, tweaks 00:02:52.973 --> 00:02:53.940 and measurements that we should be doing. 00:02:55.976 --> 00:02:57.944 So why is this stuff changing so fast? 00:02:57.978 --> 00:02:58.945 What the hell is going on? 00:02:58.979 --> 00:03:01.948 2026, you wake up, every day there's a new 00:03:01.982 --> 00:03:02.949 framework, a new thing. 00:03:03.984 --> 00:03:06.953 I think the biggest change in what has been 00:03:06.987 --> 00:03:09.956 the largest shift that I've seen in 00:03:09.990 --> 00:03:12.959 terms of a step function up in what you can run at 00:03:12.993 --> 00:03:15.962 home for local intelligence for coding is Qwen Flash Next. 00:03:16.997 --> 00:03:18.965 This is a preview of the Qwen 4 architecture. 00:03:18.999 --> 00:03:21.968 It has a large n-gram table that effectively increases 00:03:21.968 --> 00:03:25.939 the amount of state that the model has to work with without 00:03:25.972 --> 00:03:28.942 increasing the amount of VRAM that you need or the amount 00:03:28.975 --> 00:03:30.944 of RAM that you need, which is rather expensive. 00:03:31.978 --> 00:03:34.981 So this new MoE architecture, and others like it, are 00:03:34.981 --> 00:03:37.951 making it much easier to run much larger 00:03:37.984 --> 00:03:39.953 models at home. 00:03:40.987 --> 00:03:42.956 Uh, with great performance. 00:03:43.990 --> 00:03:45.959 So we're seeing specialization. 00:03:45.992 --> 00:03:46.960 winning. 00:03:46.993 --> 00:03:50.964 And that brings us to the second thing 00:03:50.997 --> 00:03:53.967 that's developing really fast and it's the inference frameworks, okay? 00:03:53.967 --> 00:03:55.936 So Strata, ninfer, freetoken. 00:03:55.969 --> 00:03:58.939 Specialization seems to be winning. Dwarf Star, which 00:03:58.972 --> 00:03:59.940 runs DS4. 00:04:00.974 --> 00:04:04.945 Um, llama.cpp is not the best performance. 00:04:04.978 --> 00:04:07.948 I think it was generally assumed that llama.cpp 00:04:07.981 --> 00:04:11.952 CPP had excellent performance for most models. 00:04:11.985 --> 00:04:13.954 This is a general purpose framework. 00:04:13.987 --> 00:04:15.956 It's largely C, C++ CUDA. 00:04:16.990 --> 00:04:17.958 It's definitely not true today. 00:04:18.992 --> 00:04:21.962 And they are so far behind what 00:04:21.995 --> 00:04:24.965 Strata is capable of doing in terms of running larger MoE 00:04:24.998 --> 00:04:26.967 models at higher tokens per second on consumer hardware. 00:04:28.969 --> 00:04:31.938 So, that's the second thing I'm seeing that's making it 00:04:31.972 --> 00:04:34.941 so, it's difficult to track what's 00:04:34.975 --> 00:04:37.944 best to run on your, on your, on whatever 00:04:37.978 --> 00:04:38.945 GPU you happen to have. 00:04:38.979 --> 00:04:41.948 So, and then third, and 00:04:41.982 --> 00:04:42.949 these are all hitting so hard. 00:04:43.984 --> 00:04:45.952 The third new quantization techniques. 00:04:45.952 --> 00:04:47.954 Okay, so GSQ-RCO, Gumbel 00:04:48.989 --> 00:04:52.959 softmax quantization, and Riemannian constrained 00:04:52.993 --> 00:04:56.963 optimization, these are ways to quantize the weights, 00:04:56.997 --> 00:04:59.966 and then optimally pick which weights to quantize how 00:05:00.000 --> 00:05:02.969 hard, given a budget, in the end resulting model. 00:05:02.969 --> 00:05:05.939 And these are making it so that Q3 is 00:05:05.972 --> 00:05:08.942 an excellent high quality quant, which I don't think was true before. 00:05:09.976 --> 00:05:12.946 So it's bringing up the size of the model that can be run, and 00:05:12.979 --> 00:05:15.949 it's bringing down the amount of VRAM that you need to run it. 00:05:15.982 --> 00:05:18.952 And so DASLab, the Austrian dudes who make this, doing an 00:05:19.986 --> 00:05:23.957 absolutely fantastic job and releasing these GSQ-RCO 00:05:23.990 --> 00:05:27.961 quants, which work with the existing GGUF format. 00:05:27.994 --> 00:05:30.964 Um, absolutely banger quants and they work in llama.cpp 00:05:31.998 --> 00:05:33.967 or whatever other framework uses a GGUF. 00:05:34.968 --> 00:05:37.938 So, additionally, number 00:05:37.971 --> 00:05:41.942 4th development that keeps happening, new fine-tunes. 00:05:41.975 --> 00:05:42.943 And so Swift 1.5 00:05:42.976 --> 00:05:45.946 is the first Qwen fine-tune that I 00:05:45.979 --> 00:05:49.950 have tested that absolutely dominates the 00:05:49.983 --> 00:05:50.951 base model, base Qwen. 00:05:51.985 --> 00:05:52.953 It's excellent. 00:05:52.986 --> 00:05:55.956 I have no idea what they fed this thing, but 00:05:55.989 --> 00:05:56.957 so the fine-tune. 00:05:56.990 --> 00:06:00.961 So we've got the architecture, the inference frameworks, new quantizations and new fine-tunes. 00:06:00.994 --> 00:06:03.964 All 4 of these developments have been in the last couple 00:06:04.998 --> 00:06:07.968 weeks and are influencing the recommendations. 00:06:07.968 --> 00:06:09.970 And the benchmarking that I'm doing at a given hardware budget today. 00:06:10.971 --> 00:06:13.940 So, and there's another thing that's 00:06:13.974 --> 00:06:17.944 happening, which is new speculative decode methods, 00:06:17.944 --> 00:06:20.947 and new quants of those are coming out every single day. 00:06:20.981 --> 00:06:23.950 Also, there was a day last week where we had uh, 2 new 00:06:23.984 --> 00:06:25.952 speculative decode. 00:06:26.987 --> 00:06:27.988 Technologies drop on the same day. 00:06:27.988 --> 00:06:28.955 I think it was Eagle and something else. 00:06:29.990 --> 00:06:31.958 So, yeah. 00:06:31.992 --> 00:06:35.962 That's going to offer more trade-offs of VRAM for performance, 00:06:35.996 --> 00:06:39.966 or straight up just be improvements over the existing defaults. 00:06:40.000 --> 00:06:42.969 So, let's take a look at the hardware. 00:06:42.969 --> 00:06:45.939 And we're going to kick this off by looking 00:06:45.972 --> 00:06:47.941 at this team hardware survey. 00:06:47.974 --> 00:06:50.944 You got a gaming PC, DDR 00:06:50.977 --> 00:06:53.947 4, DDR 5, probably Ryzen 7, or Intel i9, 00:06:53.980 --> 00:06:55.949 or newer, anything from like the last 10 years will do. 00:06:55.982 --> 00:06:56.950 Eight cores is nice. 00:06:56.983 --> 00:06:58.952 My main inference box is Ryzen 7. 00:06:58.985 --> 00:07:00.954 is definitely not new. 00:07:00.987 --> 00:07:02.956 My other box is an i9. 00:07:03.990 --> 00:07:06.960 And the i9 can be DDR 4 or DDR 5, 00:07:06.993 --> 00:07:08.962 minus the older DDR 4 variant. 00:07:08.995 --> 00:07:11.965 And this is a very, very common configuration and 00:07:11.998 --> 00:07:13.967 popular configuration on steam. 00:07:14.968 --> 00:07:18.938 And this episode should help you understand what you can expect. 00:07:18.972 --> 00:07:21.942 So, 32 gigs of RAM, 00:07:21.975 --> 00:07:23.944 16 gig GPU. 00:07:23.977 --> 00:07:25.946 This is like the median gaming PC. 00:07:25.979 --> 00:07:28.948 So for 5060 Ti owners. 00:07:28.982 --> 00:07:32.952 This is for the folks for the 5080, 4080, 4070 Ti, 5070 Ti. 00:07:32.986 --> 00:07:33.954 You get the picture. 00:07:33.987 --> 00:07:34.954 There's a lot of variants like this. 00:07:34.988 --> 00:07:38.958 I have a 5080 myself, and additionally, 00:07:38.992 --> 00:07:41.962 uh, these 2 configuration should work with no issues at all on 00:07:41.995 --> 00:07:45.966 AMD or Intel cards like the AMD 9070 XT and 00:07:47.000 --> 00:07:48.968 the Intel B770. 00:07:48.968 --> 00:07:51.938 Now, these are just absolutely non-existent among 00:07:51.972 --> 00:07:54.941 the steam hardware survey and counterstrike players, but, 00:07:54.974 --> 00:07:57.978 um, as far as I know, support for inference 00:07:57.978 --> 00:08:00.947 frameworks is a little bit more experimental on Intel and AMD 00:08:00.980 --> 00:08:03.950 graphics cards, but I've heard that it works absolutely fine. 00:08:05.986 --> 00:08:08.955 So ideally, the box that we're talking about, you're running 00:08:08.988 --> 00:08:11.958 Linux on it, headless, and it's a dedicated server for this task. 00:08:12.992 --> 00:08:15.962 And Strata, if you decide to go that route and serve 00:08:15.996 --> 00:08:18.965 Qwen Flash Next, we'll do its absolute best, use 00:08:18.998 --> 00:08:21.968 every single resource on the device, CPU, GPU, 00:08:22.969 --> 00:08:24.938 PCIe bandwidth, SSD. 00:08:24.971 --> 00:08:27.941 Um, so I hope you're running Linux headless. 00:08:28.975 --> 00:08:31.945 If you plan on going that route because say 00:08:31.978 --> 00:08:34.948 you're running Windows on it and you want to use Chrome or 00:08:34.981 --> 00:08:37.951 Edge or whatever you Windows Savages use to search 00:08:38.985 --> 00:08:42.956 Bing for the latest clippy update, you're 00:08:42.989 --> 00:08:45.959 going to want the 27B dense variant because your system is going to 00:08:45.992 --> 00:08:49.963 be busy doing whatever else you're doing with the system memory. 00:08:49.996 --> 00:08:50.964 So, Strata. 00:08:51.998 --> 00:08:54.968 Again, this is the biggest step change I've seen in 00:08:55.969 --> 00:08:58.938 the past year, 10 months looking 00:08:58.972 --> 00:09:01.941 at local inference, in terms of boosting, 00:09:01.975 --> 00:09:02.942 what can I run on my own hardware? 00:09:02.976 --> 00:09:05.945 And there's 2 parts to Strata. 00:09:05.979 --> 00:09:08.948 One is this incredible, optimized inference engine and C, C 00:09:08.982 --> 00:09:09.949 ++ CUDA Python. 00:09:09.983 --> 00:09:10.950 It borrows heavily from llama. 00:09:11.985 --> 00:09:14.954 It's not a llama fork, keeps the GPU busy at all 00:09:14.988 --> 00:09:15.989 times, keeps the CPU. 00:09:15.989 --> 00:09:16.956 Busy, like we said. 00:09:17.991 --> 00:09:18.958 It's amazing. 00:09:19.993 --> 00:09:22.962 The second part of Strata is a bunch of bullshit, bundled website, 00:09:22.996 --> 00:09:23.963 bundled harness. 00:09:24.998 --> 00:09:27.967 It auto downloads binaries from GitHub during startup without asking. 00:09:27.967 --> 00:09:30.937 Guys, auto update on startup without consent. 00:09:30.970 --> 00:09:33.940 It's one compromised credential from a massive 00:09:33.973 --> 00:09:36.943 synchronized supply chain attack similar to 00:09:36.976 --> 00:09:38.945 what's happened to LiteLLM twice. 00:09:39.979 --> 00:09:42.949 So it recommends that you install it through curling 00:09:42.982 --> 00:09:44.951 a bunch of markdown in your agent of choice. 00:09:44.984 --> 00:09:47.954 A bunch of this stuff is just non-starters for me. 00:09:47.987 --> 00:09:48.955 I want a Swiss army knife. 00:09:48.988 --> 00:09:52.959 I want the good parts and the performant parts in 00:09:52.992 --> 00:09:55.962 a C++ binary that I can poke at, 00:09:55.995 --> 00:09:58.965 attach a debugger to, have my agents run a bunch of different 00:09:58.998 --> 00:10:01.968 tests on without all the annoyance of 00:10:01.968 --> 00:10:05.939 the packaging stuff within, within Strata. 00:10:05.972 --> 00:10:07.941 So, ain't nobody got time for that. 00:10:07.974 --> 00:10:08.942 I have forked Strata. 00:10:08.975 --> 00:10:09.943 I'm calling it UnbiasedStrata. 00:10:10.977 --> 00:10:12.946 I know, I know, I'm a genius at naming things. 00:10:12.946 --> 00:10:14.948 The goal of the fork is to be distributed as source, 00:10:15.982 --> 00:10:18.952 and then you copy the source to your server, which is 00:10:18.985 --> 00:10:20.954 completely offline, and then you run make. 00:10:20.987 --> 00:10:23.957 So, you're going to need to get a few things from the internet, 00:10:23.990 --> 00:10:26.960 which is going to be CUDA, your Nvidia 00:10:27.994 --> 00:10:30.964 drivers, you're going to need the model, and then it 00:10:30.997 --> 00:10:33.967 creates like a pack. 00:10:33.967 --> 00:10:36.936 It creates a new version of the speculative head and 00:10:36.970 --> 00:10:37.937 quantizes it. 00:10:38.972 --> 00:10:41.975 Uh, and then once you have those binaries, 00:10:41.975 --> 00:10:44.944 which is the model, the pack, the 00:10:45.979 --> 00:10:48.948 spec head, and the, uh, you 00:10:48.982 --> 00:10:51.951 know, the developer libraries on your machine, you 00:10:51.985 --> 00:10:54.954 can just run make, and then it will never reach out to the internet again. 00:10:55.989 --> 00:10:59.959 This is a much more palatable approach to an inference framework for me. 00:10:59.993 --> 00:11:02.962 So, I, like I said, I want the performance benefits of 00:11:02.996 --> 00:11:05.965 Strata without the packaging and distribution annoyances. 00:11:05.999 --> 00:11:08.968 Now, I feel a little bit conflicted, recommending something that I have created, 00:11:09.969 --> 00:11:12.939 and when we're going out there and we're trying to 00:11:12.972 --> 00:11:15.942 find something to run for inference at home, I think we should be really careful 00:11:15.975 --> 00:11:19.946 about running something that's being recommended by a content creator or a YouTuber. 00:11:19.979 --> 00:11:22.949 In this case, I don't feel too bad about this particular instance 00:11:22.982 --> 00:11:24.951 because it's MIT licensed. 00:11:24.984 --> 00:11:26.953 It's not a binary. 00:11:26.986 --> 00:11:28.955 It respects the upstream source. 00:11:28.988 --> 00:11:31.991 And if you use it properly and the way it's intended, I 00:11:31.991 --> 00:11:33.960 will never know because you will clone the source code. 00:11:34.994 --> 00:11:38.965 And then you will download your model, 00:11:38.998 --> 00:11:41.968 and then you'll copy the source code to a server that's completely cut 00:11:41.968 --> 00:11:44.938 off from the broader internet, and you'll run inference there. 00:11:44.971 --> 00:11:47.941 And there won't be any telemetry or auto updating or further 00:11:47.974 --> 00:11:49.943 reaching out to GitHub or any other server. 00:11:50.977 --> 00:11:53.947 So, and then that binary will sit there until the end of 00:11:54.981 --> 00:11:55.949 time and just serve requests. 00:11:56.983 --> 00:12:00.954 from, you know, we'll just serve requests from your machine. 00:12:02.989 --> 00:12:05.959 So, as of today, I want to reiterate this. 00:12:06.993 --> 00:12:09.963 I looked at the source code for Strata with an agent. 00:12:09.996 --> 00:12:11.965 And as of today, every time you start Strata. 00:12:12.999 --> 00:12:15.969 It looks for a new version of the binary and download it. 00:12:16.970 --> 00:12:20.940 In addition, additionally with DeepSeek, 00:12:20.974 --> 00:12:23.943 every time you run it with DeepSeek, it looks 00:12:23.977 --> 00:12:26.946 for a new version of that engine and downloads it. 00:12:27.981 --> 00:12:29.949 and runs that instead of whatever you had on disk. 00:12:29.983 --> 00:12:32.952 So this is an absolute non-starter for me, and 00:12:32.986 --> 00:12:35.955 it conflicts with the SAFETY.md notes 00:12:36.990 --> 00:12:39.959 in that repo where it says that it asks you first, it does not. 00:12:39.993 --> 00:12:42.962 So I recommend UnbiasedStrata report, 00:12:43.997 --> 00:12:46.966 or if you're going to run Strata, do it in a docker container and 00:12:47.000 --> 00:12:48.968 then wipe it when you're done using it for testing. 00:12:48.968 --> 00:12:51.971 So, I recommend UnbiasedStrata. 00:12:51.971 --> 00:12:52.939 Report bugs to me. 00:12:52.972 --> 00:12:53.940 I try to fix them. 00:12:53.940 --> 00:12:55.942 I don't intend to fix every single or merge every single one percent performance 00:12:56.976 --> 00:12:59.946 gain, but I do plan to maintain this as a Swiss Army knife 00:12:59.979 --> 00:13:02.949 offline first, no auto update bullshit way 00:13:02.982 --> 00:13:03.950 to run, Qwen Flash Next. 00:13:05.985 --> 00:13:07.954 Okay, so testing with Bob-Bench. 00:13:08.988 --> 00:13:11.958 I am mostly focused on this channel on how to do coding 00:13:11.991 --> 00:13:14.961 with LLMs, so we're not talking about chat bots or things like that. 00:13:14.994 --> 00:13:15.962 General purpose coding agents. 00:13:16.996 --> 00:13:19.966 And to me, the agents should be able to work in 00:13:19.999 --> 00:13:22.969 code bases that are common for humans and agents today. 00:13:22.969 --> 00:13:26.939 So for me, that means bash, C, C++, Python, Rust. 00:13:26.973 --> 00:13:29.942 And if you think about an agent that's allowed to call tools, they need to be able to write 00:13:30.977 --> 00:13:32.945 code to call the tools. 00:13:32.979 --> 00:13:33.946 This is a basic requirement. 00:13:33.980 --> 00:13:36.949 So my benchmark set reflects this. 00:13:36.983 --> 00:13:39.952 Agents are typically dumped into a fairly large code 00:13:39.986 --> 00:13:42.955 base, something like memcached has a lot of lines of 00:13:42.989 --> 00:13:44.957 C, it's a public framework. 00:13:44.991 --> 00:13:46.959 I do tests on memcached. 00:13:46.993 --> 00:13:49.962 And uh, so yeah, the agents get bug reports, 00:13:50.997 --> 00:13:53.966 they get performance issues, they get black box debugging, 00:13:54.000 --> 00:13:56.969 they get feature requests, and a corpus of 86 tasks. 00:13:56.969 --> 00:13:59.939 Everything is run on more or 00:13:59.972 --> 00:14:02.942 less the same hardware and the same harness version and everything 00:14:03.976 --> 00:14:04.944 that I can hold constant. 00:14:04.977 --> 00:14:07.947 I hold constant to keep as 00:14:07.980 --> 00:14:09.949 much of these arms comparable as possible. 00:14:10.983 --> 00:14:14.954 And I like to organize it by VRAM budget, okay, 8 gigabytes, 00:14:14.987 --> 00:14:17.957 16 gigabytes, and 24 gigabytes are the arbitrary cutoff points for 00:14:17.990 --> 00:14:20.960 VRAM that I have chosen that I've been running the tests on. 00:14:21.994 --> 00:14:24.964 So agents are rewarded for completing 00:14:24.997 --> 00:14:27.967 tasks on time in my benchmark set also, 15 to 30 minute timeouts. 00:14:27.967 --> 00:14:30.937 If I can run DeepSeek V4 Flash at 4 tokens per second. 00:14:31.971 --> 00:14:33.940 It's going to get a 0 on my benchmark set. 00:14:33.973 --> 00:14:34.941 Actually, I think it'll get a 10. 00:14:34.974 --> 00:14:36.943 If you don't do anything, I think you get a 10%. 00:14:37.977 --> 00:14:39.946 So it doesn't mean it's a good trade-off. 00:14:39.979 --> 00:14:42.949 If it's possible to run the thing, but it's too slow. 00:14:43.983 --> 00:14:47.954 Um, so the timeouts are critical for evaluating 00:14:47.987 --> 00:14:49.956 models within that framework of actually using it. 00:14:50.990 --> 00:14:51.958 in real life. 00:14:51.991 --> 00:14:54.961 So, one thing I noticed is that the 27B 00:14:55.995 --> 00:14:58.965 and the QFN variant that I'm recommending in 00:14:58.998 --> 00:15:01.968 this video, both aced the same number of problems, and my task set 00:15:01.968 --> 00:15:02.935 that was 71. 00:15:02.969 --> 00:15:06.939 So they both got a 100% on 71 problems. 00:15:07.974 --> 00:15:11.944 However, QFN lost 00:15:11.978 --> 00:15:14.947 a bunch of points by timing out after it had solved the problem. 00:15:14.981 --> 00:15:18.951 So the model goes off, it analyzes, 00:15:18.985 --> 00:15:22.955 it looks at the source code, it writes a solution, it 00:15:22.989 --> 00:15:25.958 comes to the correct conclusion and writes code that passes the test, 00:15:25.992 --> 00:15:28.995 but then keeps cranking on and on and on, fuzzing, or 00:15:28.995 --> 00:15:31.964 writing more unit tests or whatever, and lost points 00:15:32.999 --> 00:15:34.967 by being too thorough. 00:15:34.967 --> 00:15:37.937 And so this is a new behavior in Qwen 00:15:38.971 --> 00:15:41.941 3.8 and Qwen 4 00:15:41.974 --> 00:15:44.944 and I think we saw this in Glimmer, where they 00:15:44.977 --> 00:15:47.947 do a lot of fuzzing on these more recent models. 00:15:47.980 --> 00:15:49.949 And so runtime takes a lot longer. 00:15:49.982 --> 00:15:52.952 And so honestly, we may need to reconsider 00:15:52.985 --> 00:15:55.955 these sort of timeouts and the penalty for 00:15:55.988 --> 00:15:58.958 timing out in this manner within Bob-Bench, 00:15:58.991 --> 00:16:01.961 but as of today, the policy, that is standard for 00:16:01.994 --> 00:16:04.964 every one of these runs is that if you get to the right answer, you get 00:16:04.997 --> 00:16:07.967 a 10 out of 10, and then if you time 00:16:07.967 --> 00:16:10.937 out because you're still working and you haven't yielded, at 00:16:11.971 --> 00:16:14.941 the 30 minute cutoff, you get -5 points. 00:16:15.975 --> 00:16:18.945 So Qwen got half points off on a significant 00:16:18.978 --> 00:16:21.948 number of problems by acing them and then working for too long. 00:16:21.981 --> 00:16:24.951 So if you watch this video, and you decide you're 00:16:24.984 --> 00:16:27.954 going to try to deploy Strata, consider keeping an eye on it, 00:16:28.988 --> 00:16:31.958 and adjusting your prompting to have it fuzz less, or 00:16:31.991 --> 00:16:34.961 do less testing, or just check in with you when 00:16:34.994 --> 00:16:37.964 it thinks that the work is complete before it goes off on a long 00:16:37.997 --> 00:16:38.965 verification journey. 00:16:39.999 --> 00:16:42.969 Okay, one problem with lower bit quants, and then we'll wrap up. 00:16:42.969 --> 00:16:45.938 I've seen quant damage and a small number of runs on coder and Q2 00:16:45.972 --> 00:16:46.939 and IQ2 runs. 00:16:47.974 --> 00:16:50.943 And funnily enough, the Austrian Lingua Franca is German, 00:16:50.977 --> 00:16:51.944 but almost everyone speaks English. 00:16:52.979 --> 00:16:55.948 So how many countries is involved there? 00:16:55.982 --> 00:16:58.951 So the funniest run on that I've seen is when the model 00:16:58.985 --> 00:17:02.955 is doing a bunch of reasoning about numbers and gets stuck emitting nine. 00:17:02.989 --> 00:17:05.958 Now, I think this is also what the engineers at DASLab 00:17:05.992 --> 00:17:08.961 were yelling as they were examining the traces of 00:17:08.995 --> 00:17:09.962 the quantize model fucking up. 00:17:10.997 --> 00:17:11.964 999. 00:17:11.998 --> 00:17:12.965 So I'm serious. 00:17:12.999 --> 00:17:14.967 It has issues with run on nines. 00:17:15.968 --> 00:17:18.938 In my testing, I recommend IQ3 or greater 00:17:19.972 --> 00:17:20.940 to avoid this problem. 00:17:21.974 --> 00:17:24.944 And the, uh, DASLab guys reached out and 00:17:24.977 --> 00:17:28.948 commented on one of my live streams saying that they had updated the coder quant. 00:17:28.981 --> 00:17:32.952 And the coder quant is very small, REAP variant. 00:17:32.985 --> 00:17:34.954 And so apparently that's doing better. 00:17:35.988 --> 00:17:37.957 And we need to we need to look into that. 00:17:37.990 --> 00:17:40.960 So, I'm releasing this video without 100% 00:17:40.993 --> 00:17:43.963 of the data that we need, and man, in 2026, 00:17:44.997 --> 00:17:46.966 it is hard to evaluate everything fully. 00:17:46.999 --> 00:17:49.969 We haven't talked about new developments and speculative decoding, really. 00:17:50.970 --> 00:17:53.940 DASLab, you know, the coder quant 00:17:53.973 --> 00:17:54.941 needs to be reevaluated. 00:17:55.975 --> 00:17:57.944 We need to do further tuning of Strata. 00:17:57.977 --> 00:18:00.947 I've mainly been using it with the default parameters and then varying 00:18:00.980 --> 00:18:04.951 the different memory modes between arena and the other options. 00:18:04.984 --> 00:18:08.955 So, there is a bunch of further work to do. 00:18:09.989 --> 00:18:13.960 Even for me to understand how to run Strata 00:18:13.993 --> 00:18:16.963 optimally on the hardware that I have, not 00:18:16.996 --> 00:18:19.966 to mention a general purpose recommendation for everybody. 00:18:19.999 --> 00:18:22.969 But if I were to draw a 00:18:22.969 --> 00:18:25.938 line in the sand, and I think we should do this from time to time, I think we should hold ourselves to 00:18:25.972 --> 00:18:29.942 a bar of saying, here's what we understand about the space, and 00:18:29.976 --> 00:18:31.944 and say, here's a concrete recommendation. 00:18:32.979 --> 00:18:33.946 And guys, sometimes it's wrong. 00:18:34.981 --> 00:18:37.950 I got this completely wrong with the 12 gigabyte recommendation 00:18:37.984 --> 00:18:40.953 video on the channel from July, August. 00:18:40.987 --> 00:18:43.956 And so, but I think that that is 00:18:43.990 --> 00:18:46.959 an important part of being an evaluator 00:18:47.994 --> 00:18:50.963 in the public discourse is to put your 00:18:50.997 --> 00:18:53.966 hypothesis and your journey out 00:18:54.000 --> 00:18:55.968 there to be validated or disproved. 00:18:56.969 --> 00:19:00.940 So we're doing that today, and I've 00:19:00.973 --> 00:19:04.944 been delaying on it because I want to gather every single piece of information and 00:19:04.977 --> 00:19:06.946 test every variant, and you just can't. 00:19:06.979 --> 00:19:09.949 And so I said, all right, yeah, it's Tuesday, 00:19:09.982 --> 00:19:11.951 Qwen 4 coming out on Thursday. 00:19:11.984 --> 00:19:15.955 Let's let's get this thing out here and say what 00:19:15.988 --> 00:19:17.957 we think the best is right now. 00:19:19.992 --> 00:19:22.962 So, I'll wrap up with a little bit of update about the channel. 00:19:22.995 --> 00:19:23.963 Qwen 4 coming out. 00:19:23.996 --> 00:19:27.967 is rumored on Thursday, this is from an anonymous Twitter account. 00:19:27.967 --> 00:19:28.934 I don't have any contacts at Qwen. 00:19:29.969 --> 00:19:32.972 I have no information from the inside about it, but I'm 00:19:32.972 --> 00:19:35.941 planning on streaming Thursday as the weights and inference frameworks come out. 00:19:36.976 --> 00:19:39.945 Plan on being at the computer from 8 AM Pacific until whenever we 00:19:39.979 --> 00:19:43.949 get the full benchmark set in and can put it on the Bob-Bench Wide plot. 00:19:43.983 --> 00:19:45.951 So, join me on the live stream. 00:19:46.986 --> 00:19:49.955 If you would like to support the channel, it's enough to watch the video. 00:19:49.989 --> 00:19:52.958 I also post the full videos to my website also. 00:19:52.958 --> 00:19:54.960 Do consider subscribing, sharing the video with a friend. 00:19:54.994 --> 00:19:57.963 This actually helps the video and YouTube's 00:19:57.997 --> 00:20:00.966 algorithm reach more people quite a bit if you share the video with others. 00:20:01.000 --> 00:20:03.969 And a huge thank you goes out to our channel members, 00:20:04.970 --> 00:20:07.940 Kruz, zircon, soupwizard, Mitchell, dinferno, 00:20:07.973 --> 00:20:10.943 Eeee10, Dennis, psjoshooaj, Heroian, 00:20:10.976 --> 00:20:11.944 and Josh G. 00:20:11.977 --> 00:20:13.946 So, thank you for watching. 00:20:14.980 --> 00:20:15.948 See you in the next one.