Qwen3.8-27B at 387 tokens a second on one DGX Spark
How I served Qwen3.8-27B on one DGX Spark with SGLang, NVFP4 and DFlash2: about 70 tok/s for one stream, 387 tok/s across 32 streams.
At the start of September I set up Qwen3.8-27B, an open-weights model, on one NVIDIA DGX Spark. The Spark is a desktop machine with 128 GB of unified memory. With SGLang, 4-bit NVFP4 weights and a DFlash2 speculative drafter, one request generates 64 to 78 tokens per second on a coding task, from a 320-token prompt up to 65k tokens of context. With 32 requests running at the same time, the machine produces 387 tokens per second in total. That figure is the sum across all 32 streams. Each stream gets about 15 tok/s once its first token arrives.
Why this matters
Most companies that run agents send every request to a hosted API. An agent sends its whole conversation back to the model on every turn, and that conversation often holds source code, contracts or customer records.
On one Spark, 16 agents can run at once at 360 tok/s in total, about 27 tok/s each, with a median wait of 0.70 s for the first token. The data stays on a machine you own. You pay for the machine and its power, with no charge per token.
For agents, I now check the prefix cache first. The server keeps the processed form of prompts it has seen, so it does not process a repeated prefix again. In the published sweep, a 257k-token prompt takes 277 s to its first token when cold, and 1.2 s when I send it a second time. In an earlier, unpublished test on the same box and model, a coding agent made real edits to a real repository. It sent 267 prompt tokens for every token it generated, and the server served 99.0% of those prompt tokens from the cache. At the measured prefill rate of 2,240 tok/s, the prefill for that task would take about 41 minutes without the cache. With the cache it took 24 seconds. If you plan hardware for agents, plan for cache capacity as well as decode speed.
The setup
The hardware is one DGX Spark: a GB10 Grace Blackwell chip with 128 GB of memory shared by the CPU and GPU, at about 231 GB/s measured bandwidth.
The model is RadixArk/Qwen3.8-27B-NVFP4, a 4-bit build of Qwen3.8-27B. Only 16 of its 64 layers use full attention, which helps it keep its speed at long context. The window is 262,144 tokens. The engine is SGLang, built from a pinned image plus two upstream pull requests, #35371 and #35496.
The drafter is maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal. Speculative decoding uses a small drafter to guess the next block of tokens. The large model then checks the whole block in one pass and keeps the tokens it agrees with, so one pass can produce many tokens. I set the draft budget to 16 tokens per pass. On code, the model accepts 6.8 to 8.3 of them. Thinking is off and temperature is 0 for every number here.
Results
| Workload | Streams | Total tok/s | Tok/s per stream | Accepted tokens per pass |
|---|---|---|---|---|
| 324-token prompt | 1 | 72.7 to 75.6 | same | 6.88 |
| 8,159-token prompt | 1 | 73.9 to 75.5 | same | 7.53 |
| 65,842-token prompt | 1 | 63.8 to 66.8 | same | 8.12 |
| 130,656-token prompt | 1 | 55.9 to 56.4 | same | 7.92 |
| 257,334-token prompt | 1 | 38.2 to 41.3 | same | 6.83 |
| 2k-token code prefix each | 4 | 199.4 | 57.5 | 7.4 |
| 2k-token code prefix each | 8 | 276.0 | 42.7 | 7.5 |
| 2k-token code prefix each | 16 | 359.5 | 26.8 | 7.3 |
| 2k-token code prefix each | 32 | 387.2 | 15.3 | 7.2 |
The single-stream rows are the median of 4 runs, and the range covers two boots. The concurrency rows are the mean of 2 runs, with 512 output tokens per stream. The 32-stream row uses a separate profile with 32 seats and a larger memory share.
Total throughput, per-stream speed and time to first token from 1 to 32 concurrent streams, for the 16-seat and 32-seat profiles.
The single-stream context sweep. Generation holds near 70 tok/s to 65k, then falls to 38 at 257k.
What made the difference
Speculative decoding. With no drafter, the same model decodes at 12.6 to 12.7 tok/s on short prompts. The drafter multiplies that by 5.5 to 5.9.
A draft budget of 16. Going from 8 to 16 draft tokens per pass gave +10.5% on fresh code and +69.3% on edit-heavy work, where the model re-emits code it has already seen.
A 4-bit drafter. The bf16 drafter read 3.85 GB per pass. The NVFP4 drafter reads 1.45 GB, and generation went up by 5 to 10%.
The FlashInfer tactic cache. At boot, FlashInfer times several kernels for each matrix shape and keeps the fastest. Timing noise makes each boot pick differently, so the same configuration measured differently every time. I called this the boot lottery. I had never mounted the cache directory into the container, so every boot tuned from scratch. With the cache mounted, decode on the older bf16-drafter recipe went from 60.85 to 70.6 tok/s. About one draw in three is good. The repository ships the draw I kept, which is worth 16% against a median boot.
SGLang PR #35496. This change folds the DFlash2 path selector into the draft CUDA graph when the output layer is 4-bit. Without it, the selector runs outside the graph and throughput drops. The recipe builds the engine image with it, and the launch script checks the boot log.
Memory for 32 seats. The CUDA graphs for each seat come out of the same memory share as the KV cache. At the default share of 0.65, 32 seats leave a pool of only 49,958 tokens and the total falls to 213 tok/s. At 0.80 the pool is 173,120 tokens and the total is 387.
Three recipes on the same box and prompts. I adopted the NVFP4 drafter and held back the fp8 KV cache until a quality check.
Caveats, replication and limits
On 2 September I ran a fresh copy of the repository on the same box, with its defaults and the shipped tactic draws. It booted in 210 s and gave 79.0 tok/s at 311 prompt tokens and 70.0 at 8,318, inside the two-boot range. @Luc_Gibson followed the README from a clean start and found a missing step for the offline weights cache. With that fixed, it boots in about 200 s. Another Spark owner reproduced my chat figure independently. Nobody else has reproduced the 387 yet, as far as I know.
These numbers are for code. On mixed chat prompts the same server does about 30 tok/s, because the drafter accepts about 3 tokens per pass on prose against 7 to 9 on code. With realistic 15,300-token agent prompts, 8 streams give 47 tok/s in total, and 16 streams do not fit in memory. More seats also cost context: 16 seats cut the KV pool from 413,460 tokens to 238,605. Past 65k, single-stream speed falls to 56 tok/s at 130k and 38 to 41 at 260k.
This is one box, and most configurations had one boot. Two boots of the same recipe differ by a median of 4.3% and at worst 8.1%. I have not measured output quality with a task benchmark. The weights carry Qwen's own licence, which includes a revenue and monthly-user clause.
Reproduce it
Everything is at github.com/pangoleen/qwen3.8-27b-dgx-spark-dflash2: the launch script, the engine image build, six benchmarks, the data behind every number and the kept tactic draws. After you download the two pinned checkpoints, the start is:
docker build -t qwen38-27b-sglang-dflash2-sm121:0.3.0 -f image/Dockerfile image/
./serve.sh
AGENT_SETUP.md is a prompt for a coding agent on the Spark, which then does the whole setup and checks it.
If your team is weighing a private model for agent work, I am happy to compare notes. Get in touch.