# 16 GB VRAM: the run configurations

Two serving configurations from the episode, as run for the Bob-Bench Wide numbers. Both assume a
16 GB card (or a bigger card held to a 16 GiB frame), 128k context, q4_0 K/V cache, MTP speculative decode.

## A. 32 GB of system RAM or less: Qwen 3.8 27B, Swift 1.5 fine-tune

Everything on the card: one 12.1 GB GGUF, nothing in system RAM. Reasoning effort medium, not the xhigh default.

| | |
| --- | --- |
| Weights | `ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF`, `IQ3_S-mtp` |
| Engine | llama.cpp, MTP draft window 2 |
| Bob-Bench Wide | 0.873 · 71 of 86 at full marks · 5.5 h · 16,192 MiB peak |
| Decode | over 70 tok/s on a 4090 |

```sh
llama-server \
  --host 0.0.0.0 --port 8081 \
  -hf ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S-mtp \
  -c 131072 -ngl 99 -fa on --jinja \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  --chat-template-kwargs '{"reasoning_effort": "medium"}' \
  --reasoning on --reasoning-budget 8192 \
  --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0.0 \
  --presence-penalty 0.0 --repeat-penalty 1.0 \
  --n-predict 32768
```

## B. 64 GB of system RAM or more: Qwen 3.8 Flash Next on UnbiasedStrata

The 40 GB expert set and the 29 GB n-gram table live in page-locked system RAM; a many-core CPU serves the
expert-cache misses. Headless Linux, dedicated box: Strata uses every core, the GPU, PCIe and the SSD.

| | |
| --- | --- |
| Weights | `ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF`, `IQ3_XXS` (two shards) |
| Engine | UnbiasedStrata (`--engine strata`), MTP window 4, pack + expert profile built locally |
| Bob-Bench Wide, 128 GB + 24-core EPYC | 0.924 · 76 of 86 at full marks · 6.7 h · 16,243 MiB peak · 109 tok/s |
| Bob-Bench Wide, 64 GB + desktop i9, table on NVMe | 0.890 · 68 full · 7.7 h · 10 timeouts · 66 tok/s |
| Bob-Bench Wide, 32 GB, IQ2_XS instead | 0.894 · 71 full · 6.9 h |

```sh
M=IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS
unbiased-strata --engine strata -m $M-00001-of-00002.gguf \
  --host 0.0.0.0 --port 18090 --alias flash-next \
  -- --pack pack-iq3xxs --expert-profile pack-iq3xxs/expert-profile.bin \
     --native $M-00001-of-00002.gguf --ple-gguf $M-00002-of-00002.gguf \
     --expert-cache auto --prefill auto \
     --spec 4 --spec-min-p 0.5 --mtp mtp/ \
     --max-context 131072 --kv q4_0 --kv-resident 32768 \
     --ple-io ram --pool-workers 24 \
     --prompt-cache 6 --prompt-cache-root 0
```

- `--pool-workers`: your core count minus a couple (the pod had 24 of an EPYC).
- `--vram-reserve-mib 8850`: add it on a 24 GB card to hold the engine to the 16 GiB frame; on a 16 GB card
  leave the default.
- `--ple-io ram` needs the table in RAM (128 GB). With 64 GB the table stays on NVMe (`--ple-io direct`) and the
  run is CPU-bound: same expert hit rate as the pod, a third slower, timeouts on the long tasks.
- Reasoning effort is per request (`chat_template_kwargs`), medium for agentic work.

## Harness

Point the `pi` coding agent (https://pi.dev) at the local server, or whatever agent you are comfortable with.
Bob-Bench Wide: 86 tasks, 15 to 30 minute timeouts, 5-point deduction for timing out, one run per arm, same
harness for every configuration. Full board: https://unbiasedbob.com/bbw