Recipe · Three or four DGX Sparks
DeepSeek V4.1 Flash on three or four DGX Sparks, with SGLang.
Serve DeepSeek V4.1 Flash 1017 from three DGX Sparks linked in a ConnectX-7 triangle, behind one OpenAI-compatible endpoint: native MXFP4 experts, FP8 dense weights, Engram tables on NVMe, DSpark speculative decoding, tool calling and vision. A fourth Spark adds an opt-in line with a 1M-token context and 16 requests at once.
What's new · 17 October 2026
The 1017 checkpoint, and ten pull requests.
The recipe now serves deepseek-ai/DeepSeek-V4.1-Flash-1017, pinned at 1bae0e1. It has the same config and shard count as the previous pin; its local folders end in -1017, so an older download is not reused and ./start.sh download fetches the new one. The full notes are in the CHANGELOG.
Loops end with a name
A runaway generation loop now ends as a named error (loop_abort: …) instead of a normal stop, which had handed agent clients cut-off tool calls that looked finished. By GanyX19; DSV41_LOOP_NAMED_EXIT=0 restores the old stop.
Broken tool calls kept
A truncated tool call left in a chat's history no longer fails the request with HTTP 400: it is passed on as unparseable arguments, so agent clients keep going. Also by GanyX19.
Anthropic and setup fixes
Anthropic's thinking: {"type":"enabled"} works without a budget, and start.sh finds SSH keys more reliably (100menotu001); an image placeholder in message text no longer loops on HTTP 500 (hushengkai); pack re-shares the checkpoint when a worker can't see it (BHCC2025).
And more
- Per-host SSH users (colonel-otto),
--env-filedefaults (Rusty's AI Lab),downloadin the quick start (njs-io), shared-expert padding that followsTP_SIZE(Rhys Jones). - The DSpark SPS table is ignored on this model, where it crashed CUDA-graph capture (reported by ecohash-co and hushengkai; warning by Zbigniew Majewski).
--fp8-gemm-backendis kept whenEXTRA_SGLANG_ARGSis set (reported by motzudao).
Performance
Measured, not promised.
Three Sparks · ./start.sh, TP=3
Greedy, thinking off. About 59 ms a speculative step at one request. Greedy decoding is deterministic run to run.
Decode, prose
All requests together, per request, and time to first token.
| Requests | Aggregate | Per request | First token |
|---|---|---|---|
| 1 | 51.0 tok/s | 51.0 tok/s | 223 ms |
| 2 | 75.1 tok/s | 39.0 tok/s | 290 ms |
| 4 | 85.4 tok/s | 22.8 tok/s | 356 ms |
Prefill and context
Prefill runs at about 2,000 tok/s. The window is set to 256k tokens (the model's maximum is 1M), with a 750k-token KV pool and up to 4 requests at once.
Verified: single prompts up to 208k tokens, and four 46k-token prompts at once. The head's memory is what limits long prompts, so the README suggests 32,768 for an unattended endpoint.
About 6 GB stay free on the head while serving (it was under 1 GB before the NCCL buffer changes).
Four Sparks · ./start-tp4.sh, the TP=4 production line
Measured from a fresh clone with sparkDash 1.8.8: greedy, 256 tokens a request, a switched fabric. Aggregate across the concurrent requests, per request in brackets.
Decode
Prose is the headline; code, structured and JSON prompts draft better.
| Prompt | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| Prose | 87.7 | 120.4 (61.9) | 163.6 (41.4) | 237.6 (30.8) | 342.7 (22.2) |
| Code | 124.8 | 175.0 (88.4) | 246.8 (63.2) | 309.8 (41.4) | 438.3 (29.3) |
| Structured | 152.4 | 177.6 (103.3) | 240.4 (70.4) | 295.5 (44.2) | 572.2 (44.4) |
| JSON | 118.9 | 174.2 (89.8) | 301.7 (76.2) | 471.5 (60.2) | 659.9 (42.7) |
tok/s. Prose aggregate is still climbing at 16 requests while each request falls from 87.7 to 22.2 tok/s.
Prefill
Cold prompts, synthetic filler.
| Prompt | Prefill |
|---|---|
| 4k | 4,059 tok/s |
| 16k | 5,855 tok/s |
| 32k | 5,900 tok/s |
| 64k | 5,925 tok/s |
| 128k | 5,797 tok/s |
| 262k | 5,394 tok/s |
The filler is one repeated token, so the 16k-128k rates sit on a hot Engram row. On real text (unique prompts, no cache hits) the same line runs 4.9k-5.0k tok/s at 16k-128k.
A million tokens, checked
A passphrase hidden in a 1,011,084-token prompt was found on the production line: 4,096-token chunks with the chunked indexer, an 8M-token KV pin and a 0.80 memory fraction.
The pool holds 8,000,000 tokens (13.4 GB of KV a rank): eight requests at about 1M tokens each, and up to 16 requests at once.
An independent switchless-ring run (four cables, no switch) measured 224.8 tok/s for eight requests of code; see its notes.
Speed preview
What 51.0 tok/s feels like.
Sample text played at the measured decode speed on three Sparks: one request at 51.0 tok/s, or four at once at 22.8 tok/s each (85.4 combined), at about four characters a token. Switch between them in the window.
Three or four Sparks
Why three, and what four adds.
It does not fit on two
The checkpoint is 476 GiB on disk. With its two Engram n-gram tables (189 GiB) moved to NVMe, about 305 GiB of MXFP4 experts and FP8 dense weights stay on the GPUs.
| Layout | Weights a GPU | Fits 121.7 GiB? |
|---|---|---|
| 2 Sparks | 145 GiB | No |
| 3 Sparks | ~101 GiB | Yes, with Engram on NVMe |
| 4 Sparks | ~77 GiB | Yes, with ~40 GB free a rank |
On a Spark host memory is GPU memory, and a head that runs out stalls every rank, so most of what the recipe adds is about memory. Never set OFFLOAD_MODE=ram: the Engram tables would push the model out.
Side by side
| 3 Sparks (TP3) | 4 Sparks (TP4) | |
|---|---|---|
| Free memory a rank | ~6 GB | ~40 GB |
| Attention shards | Heads padded 64→96; rank 2 is all padding | Exact: 16 heads, 2 groups a rank |
| KV pool / context | 750k tokens / 256k | 8M tokens / 1M |
| Requests at once | 4 | 16 |
| Prefill | ~2,000 tok/s | 4,059-5,925 tok/s cold |
| Fabric | A cable triangle between the three | A RoCE switch, or the switchless ring |
A Spark has two ConnectX-7 ports, so three form a full triangle but four can't. Four need a RoCE switch, or the opt-in switchless ring (four cables and a patched NCCL on every rank). The four-Spark line changes nothing for three.
The engine
Powered by SGLang.
SGLang serves the model, a dev build pinned by digest with FlashInfer kernels. The recipe adds an overlay of adapters that fit it to the Spark: the GB10's kernels, its unified memory and a model that does not divide by three.
Engram rows from NVMe
The two n-gram tables stay on each node's own NVMe, packed per rank (./start.sh pack, about 63 GiB a node). Rows are read with O_DIRECT by 96 threads, on a side stream, while the GPU works.
The right FP8 kernel
The model's FP8 blocks are 32×32, which SGLang would send to a slow Triton kernel. The recipe routes them to FlashInfer's b12x kernel: the dense projections went from about 52 ms a step to 17.
Three ranks, padded
64 heads, 8 groups and 128 draft experts don't divide by three, so they are padded to 96, 12 and 129, and the padded shards' block scales are repaired so no layer falls back to a slower path.
DSpark drafts, checked
Speculative decoding drafts five tokens a step, capped by the draft's confidence, with block verification that keeps sampled rows exact. Fused MoE finalize is off, so repeated greedy requests are byte-identical.
What you need
Three Sparks, or four.
Three or four DGX Sparks
GB10 systems with 121.7 GiB of unified memory each, linked over ConnectX-7 with RoCE: a cable triangle for three, a RoCE switch or the switchless ring for four. Keep other containers off them while serving.
Disk
The checkpoint is about 476 GiB on the head; the workers mount it over NFS instead of copying it. Each node also keeps its rank's Engram rows on local NVMe: about 63 GiB (about 48 GiB at four Sparks).
Docker and SSH
Docker on every node and SSH from the head to the workers. ./start.sh doctor checks SSH, Docker, the RoCE links, the checkpoint shards, the image and a busy GPU before anything starts.
Time
Packing the Engram shards takes about 10 minutes, once. A full boot takes 12-13 minutes: 8 to read 476 GiB, then the draft model, the KV pool and CUDA-graph capture.
Quick start
Seven commands, run once.
Set up three Sparks
Run these on the head (the first Spark), after cloning the recipe. Set the addresses and users in .env for your Sparks first: the example has the author's.
cp -n .env.example .env # IPs, users and knobs
./start.sh doctor # ssh, docker, RoCE, checkpoint shards, image, busy GPU
./start.sh download # the pinned 1017 checkpoint, once
./start.sh build # base image + the recipe's overlay, on all three nodes
./start.sh share # NFSv4 export + docker volumes on the workers
./start.sh pack # Engram shards onto each node's NVMe (once, ~10 min)
./start.sh serve # workers first, then the headTalk to it
The API listens on port 8888 with the model id deepseek-v4.1-flash. Always send max_tokens; a request without one is capped at 32,768.
curl http://<head-address>:8888/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"deepseek-v4.1-flash","messages":[{"role":"user","content":"What is 19 + 23?"}],
"max_tokens":32, "chat_template_kwargs":{"thinking":false}}'Or four Sparks
start-tp4.sh is the same engine with a profile of its own (.env.tp4, state-tp4/, logs-tp4/), so one checkout can drive both fleets. Its guide is docs/tp4.md.
cp -n .env.tp4.example .env.tp4 # IPs, ssh user, NFS addresses
./start-tp4.sh doctor && ./start-tp4.sh build && ./start-tp4.sh share
./start-tp4.sh pack # Engram shards, once per node
./start-tp4.sh serve # stop | status | logs worker3Day to day: ./start.sh status, logs [worker1|worker2] [N], smoke and stop. Set API_KEY in .env to require a bearer token.
Thinking and the API
Built for agent clients.
- Thinking is off with
"chat_template_kwargs": {"thinking": false};enable_thinkingworks as an alias. - The reasoning budget follows DeepSeek's own table (low / high / max = 50 / 75 / 100, default 75), set per request with
reasoning_effort. - Tool calling and image input, behind one OpenAI-compatible endpoint, with Anthropic's Messages API too.
- A generation that loops (a repeated n-gram, token or line) is ended with a named error instead of running to the 1M context.
Correctness
Same answer, every run.
- Greedy decoding is deterministic run to run: the fused MoE finalize, which sums expert outputs with atomic adds, is off (about 0.3 ms a step).
- Prefills of more than 64 tokens go to FlashInfer's sparse prefill path, and both kernel paths were checked deterministic.
- Never set
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True: with it, long prefills return NaN logits. - Batches of more than one request can still differ between runs, as in any SGLang server without deterministic inference.
Settings
The ones you might change.
Everything lives in .env (.env.tp4 for four Sparks), which bash sources, so quote values with spaces. The full list is in the README.
| Variable | 3 Sparks | 4 Sparks | Meaning |
|---|---|---|---|
CONTEXT_LENGTH | 262144 | 1048576 | The window a request; the model maximum is 1M |
MAX_TOTAL_TOKENS | 750000 | 8000000 | The KV pool, pinned |
MAX_RUNNING_REQUESTS | 4 | 16 | Requests at once (also the CUDA-graph batch sizes) |
MEM_FRACTION_STATIC | 0.95 | 0.80 | Lowering it on three Sparks doesn't free memory, it only starves the KV pool |
SPEC_ALGO / DSPARK_BLOCK_SIZE | DSPARK / 5 | DSPARK / 5 | off turns speculation off |
DSV41_MAX_NEW_TOKENS | 32768 | 32768 | The cap for a request without max_tokens |
OFFLOAD_MODE | nvme | nvme | Keep it: ram would push the model out |
Licenses
Free, with a few rules.
- The recipe is under the GNU Affero General Public License v3.0 or later. It builds on 0xSero's MIT-licensed recipe (its notice is kept) and includes one Apache-2.0 file from SGLang.
- The model weights are not part of the recipe: they download from Hugging Face, under the checkpoint's own license.
Credits
Built on the work of others.
- DeepSeek V4.1 Flash 1017 by DeepSeek
- The recipe's skeleton,
boot.py, the Engram row-store idea and the Docker layout from 0xSero's recipe for four RTX PRO 6000s - Decode adapters and the four-Spark production line by Zbigniew Majewski (knapcio), from the knapcio TP4 recipe
- On the four-Spark line: b12x / local-inference-lab (RoCEnante, fused MoE), rhys101, LuZ, sumsliu, FujitsuPolycom/sparkring, rsync, Saolence and kpham-sgl; full credits in docs/tp4.md
- SGLang and FlashInfer
- This update's pull requests: GanyX19, 100menotu001, hushengkai, BHCC2025, Rusty's AI Lab, njs-io, colonel-otto, Rhys Jones and Zbigniew Majewski
The TP3 padding, the b12x route, the NVMe Engram row store, loop abort, the NCCL and memory settings and the multi-node launcher are this recipe's own.
Run it on your Sparks.
Free and open, with every adapter and every number published.