Running a 125B model on one DGX Spark, from 33 to 52 tok/s
How I ran Qwen3.8-Flash-Next (125B total, 6B active) on one NVIDIA DGX Spark, and the changes that took decode from 33 to 52 tokens a second.
I ran Qwen3.8-Flash-Next on a single NVIDIA DGX Spark, a desk-sized machine with 128 GB of memory. The model has 125 billion parameters in total, and about 6 billion are active for each token. As published, it does not fit in that memory, and it runs anyway. With my published recipe it generates code at 52 tokens a second on average, across prompts from 325 to 259,000 tokens. The starting point was 33.
Why this matters
Most companies use large models through an API. That works, but the data leaves the building, and the bill grows with every token. A single box changes that.
- Private data. The weights, the prompts and the answers stay on one machine. The server runs offline and downloads nothing while it serves.
- Cost. The cost is one box, bought once, with no per-token bill.
- Latency. A long conversation stays usable. With prefix caching on, each new turn of a growing 110,000 to 241,000-token conversation took about 18 s. Without it, the same turn takes about 145 s.
- Owning the stack. Every file is pinned, so the published numbers apply to exactly what you build. The cost is that someone must maintain it. My build checks the fingerprint of each file it patches and stops if the upstream code has changed.
There are limits. Only 14 to 17 GB stay free after load, so nothing else large can run beside it. I have also not measured output quality.
The setup
- Hardware: one DGX Spark (GB10 Grace Blackwell), 128 GB of memory that the CPU and GPU share, and a fast NVMe disk.
- Model:
RadixArk/Qwen3.8-Flash-Next-NVFP4, a 4-bit version of the model. It is about 127 GB on disk, with a native context of 262,144 tokens. - Serving: vLLM with an OpenAI-compatible API. The model's own small draft head (MTP) guesses several tokens ahead, and the full model checks the guesses in one pass.
"Does not fit as published" means this: 127 GB of checkpoint against 128 GB of memory, which the operating system and the runtime must also share. The route that makes it fit is blazux's qwen3.8-Flash-DGX. It adds a patch to the official vLLM image that keeps a 44 GiB lookup table on the NVMe disk and reads it from there. After load, about 105 GB is in use.
Results
All numbers are decode speed on the same box. Thinking is off except in the one column that says so.
| Configuration | Code (tok/s) | Code, thinking on | Prose | Four real tasks |
|---|---|---|---|---|
| As published, 2 Sept | 33.5 to 34.3 | n/a | n/a | n/a |
| My launch settings, 9 Sept | 43.1 | 40.0 | 24.7 | 138.3 s |
| Launch settings plus four build overlays | 52.6 | 51.6 | 28.9 | 102.3 s |
The first row is an earlier sweep, two repeats per prompt size. The other rows share one day, one harness, four repeats and prefix caching off. The real tasks are two coding and two reasoning tasks, replayed from a coding agent.
Published recipe (grey), tuned build (blue) and a stronger build that is not yet public (green), measured on 9 September 2026.
My launch settings on the stock image, before the overlays. Decode stays between 37 and 44 tok/s across the whole context range.
What made the difference
Side layers in FP8 (+10 to 20%). The model reads some dense layers in full for every token. A one-time, 10-minute conversion to 8-bit cuts what it reads. The draft head accepts the same number of tokens.
Three draft tokens instead of two (+12 to 15% more). At two tokens the draft head was 93 to 97% saturated, so it could usefully guess further. Four tokens gave nothing more.
These two launch settings took decode from about 33 to between 42 and 46 tok/s.
Four build overlays (+22.0% on code, +28.9% with thinking on, +16.9% on prose). Each is a layer on the stock image:
- Move the table read out of the calculation. The model read its 44 GiB table in the middle of each step. That read stops the GPU from recording the step as a CUDA graph, a replay that removes most per-step overhead. I moved the read into a fixed buffer before the step, so the full decode step now records. This change makes the rest possible.
- A smaller draft vocabulary. The draft head read a 1.27 GB table to choose from 248,000 tokens. It now chooses from the 65,536 most frequent ones and reads a quarter of the table. The full model still checks every guess, so the output cannot change.
- A deterministic top-k kernel, so the same input gives the same output.
- A faster attention backend for this chip, backported from a merged vLLM change.
The draft head's hit rate did not change: 3.64 accepted tokens per pass before, 3.57 after. Each pass costs less. That is why thinking gains most, because a thinking trace is thousands of small decode steps.
Faster boot. Start-up took 904 s, with the disk at 0.11 GB/s of its 3.07 GB/s. The checkpoint holds 299,847 tensors, 98.4% of them small per-expert tensors, and the loader paid a fixed Python cost on each. A loader that moves whole files to the GPU cuts this to 283 s. It is still opt-in.
What did not work, and what is not proven
- The 57 tok/s figure I posted on X needs four more overlays that are not in the public repository. You cannot rebuild it from a clone yet, so the published recipe gives 52.
- Output quality is not measured. The overlays store the model's recurrent state in BF16, which changes the exact output. No task benchmark proves quality is the same.
- Prefix caching failed once. On 7 September it returned an answer about the wrong topic. Four open vLLM fixes for this class of bug, which Radar105 found first, did not change the result. The failure has not come back across 16 growing-conversation turns and 5 gate rounds, so caching is on by default again. I never found the cause, and the repository ships a test for your own workload.
- My own concurrency script disagrees with sparkDash by about four times, for reasons I have not found. The figure I quote, 218 tok/s across 8 streams on the stock recipe, is sparkDash's.
- The stock top-k kernel gives 17 to 24% faster prefill with non-deterministic output. I left it out.
- Most results come from one boot on one box.
Credit. blazux built the route that makes this model fit. RadixArk published the checkpoint. MiaAI-Lab made sparkDash and used the reduced draft vocabulary first, and the idea behind it comes from the FR-Spec paper. Jürgen Schmied wrote the deterministic top-k kernel, which I vendored. What I built: the tuned launch script, my implementation of the draft vocabulary, the table-staging overlay, the build, the loader fix, the sweep harness and the measurements.
Reproduce it
Everything is at github.com/pangoleen/qwen3.8-flash-next-dgx-spark. Build blazux's image, copy in my serve.sh, prepare the FP8 side layers, then start the server:
MODE=hybrid MTP=3 GPU_MEM=0.80 PORT=18300 scripts/serve.sh
./build/build.sh adds the four overlays in about 20 minutes. AGENT_SETUP.md lets a coding agent on the Spark do the whole setup.
If your team is weighing a private model against an API, I am happy to compare notes. Get in touch.