Recipe · Two DGX Sparks
Qwen3.8 Flash Next on two DGX Sparks, with TensorFold.
Serve Qwen3.8 Flash Next from two DGX Sparks linked by their ConnectX-7 ports, through an OpenAI-compatible API, with a 1,048,576-token context, up to 16 requests at once and image and video input. It runs TensorFold's Zig engine on both Sparks, one rank on each, with no Python engine in the serving path, and two checkpoints to choose from: NVIDIA's NVFP4 or the faster INT4-AutoRound.
Performance
Measured, not promised.
Two DGX Sparks, the recipe's defaults (16 requests at once, FP8 KV cache, image input on, a 1,048,576-token window, MTP and copy drafts), measured with sparkDash through the OpenAI API on 2026-10-07.
Decode, prose
All requests together, per request, and the time to first token, for each checkpoint.
| Requests | NVFP4 | Per request | First token | INT4-AR | Per request | First token |
|---|---|---|---|---|---|---|
| 1 | 67.4 tok/s | 67.4 | 104 ms | 89.3 tok/s | 89.3 | 76 ms |
| 2 | 98.9 tok/s | 50.5 | 174 ms | 135.0 tok/s | 73.7 | 113 ms |
| 4 | 152.2 tok/s | 40.9 | 248 ms | 220.2 tok/s | 58.8 | 167 ms |
| 8 | 231.2 tok/s | 31.6 | 349 ms | 284.9 tok/s | 43.0 | 288 ms |
| 16 | 390.1 tok/s | 26.8 | 516 ms | 483.6 tok/s | 35.1 | 461 ms |
Prefill
How fast a prompt is read, and the time to first token. Measured earlier the same day (bf16 KV, 8 requests at once); this release's prompt path is faster still.
| Prompt | NVFP4 | First token | INT4-AR | First token |
|---|---|---|---|---|
| 4k | 2,648 | 1.56 s | 3,244 | 1.27 s |
| 8k | 2,597 | 3.17 s | 3,303 | 2.49 s |
| 16k | 2,757 | 5.96 s | 3,246 | 5.06 s |
| 32k | 2,575 | 12.74 s | 3,285 | 9.99 s |
| 64k | 2,511 | 26.12 s | 3,110 | 21.09 s |
| 128k | 2,601 | 50.40 s | 2,982 | 43.96 s |
| 256k | 2,458 | 106.69 s | 2,673 | 98.09 s |
tok/s. A ~1M-token prompt (998,986 tokens, a needle in a haystack, NVFP4) prefilled in 1,309 s with an earlier build, and the needle was found.
Checked, not assumed
Drafts only propose: every drafted token is checked against the model's own keyed sample, so drafted replies equal serial, one-token-at-a-time decoding (send "draft": false for that reference). The two ranks are built to give the same bits.
- Drafted, serial and concurrent replies identical (
tools/exact.py, also with ~100, 1k and 3k-token prompts) - Needles found at 200k and 1M tokens
- 26/26 exact at 128k and 256k (
tools/long_context.py) - Tool calls, array arguments whole and streamed (
tools/toolcheck.py)
The checks in tools/ run against your own server with Python's standard library only.
Speed preview
What 67.4 tok/s feels like.
Sample text played at the measured decode speed with the default NVFP4 checkpoint: one request at 67.4 tok/s, or four at once at 40.9 tok/s each (152.2 combined), at about four characters a token. Switch between them in the window.
Choose a checkpoint
The full model, or the faster cut.
QUANT=nvfp4 · default
NVIDIA's NVFP4
nvidia/Qwen3.8-Flash-Next-NVFP4, ~124 GiB: the full model, all 10 routed experts a token, 6.0B active parameters. Routed experts in NVFP4, the n-gram table and the MTP experts in FP8, bf16 elsewhere.
Prose 67.4 tok/s alone, 231.2 for 8 requests, 390.1 for 16; prefill ~2,500-2,750 tok/s.
QUANT=int4ar
INT4-AutoRound
azampatti's INT4-AutoRound, ~122 GiB: 5 routed experts a token instead of 10, with a healed shared expert, 4.8B active. Faster, and about 10% lower general capability on its authors' own harness (46.6-47.6 against the original's 51.8; tool use 91 against 91-92), per its model card.
Prose 89.3 tok/s alone, 284.9 for 8 requests, 483.6 for 16; prefill ~2,700-3,300 tok/s.
One image serves both, and prepare.sh downloads only the one you choose: set QUANT in scripts/local.sh or .env, or run ./start.sh --quant int4ar (./start.sh restart --quant ... switches a running server). Both serve the same API, window and drafts.
The engine
Powered by TensorFold.
TensorFold, by Ash Hart, is an open-source engine that serves language models on Apple Silicon and NVIDIA GPUs through an OpenAI-compatible API. This recipe runs its Zig engine, tensorfold-native, from the zig-flashnext branch.
Exact decoding
Drafts make it fast without changing a word: a draft is accepted only when it equals the token the same engine would produce one at a time. Send a request with "draft": false to compare.
Two ranks, one model
Each Spark holds half of the attention and DeltaNet heads, so half of every token's cache. The ranks talk over the ConnectX-7 link with NCCL on its RoCE devices, plus a one-shot RoCE all-gather.
No Python in the path
The Zig engine serves the API itself, in NVIDIA's PyTorch container. Python only decodes images and video.
In this recipe
Nine patches on TensorFold's Zig engine: the Flash Next port (ported from TensorFold's Python Flash Next CUDA engine), two ranks, serving, the compiled kernel set, and the vision helper. Built on Jürgen Schmied's Zig CUDA serving path (TensorFold PR #443).
What you need
Two Sparks, one cable.
Two DGX Sparks
Or two GB10 systems with 128 GB unified memory, with nothing else large on their GPUs. Each needs about 102 GiB free at start for the default 1M window (~72 GiB at 262,144); start.sh stops before launching below it.
A ConnectX-7 link
A QSFP cable between the CX7 ports and an IPv4 address on each end in one private subnet (ping must work), with a RoCE v2 GID. start.sh checks it.
ssh and Docker
Key-based ssh from the first Spark (the head) to the second (the worker). Docker with the NVIDIA container runtime, your user in the docker group, and rsync, on both.
Disk
About 150 GB on each Spark: the checkpoint (~133 GB) under ~/.cache/huggingface and the image. With WORKER_WEIGHTS=nfs the worker needs only the image.
Quick start
One command, on the head.
Clone, name the worker, start
git clone https://github.com/MiaAI-Lab/Qwen3.8-Flash-Dual-DGX-Sparks-TensorFold.git
cd Qwen3.8-Flash-Dual-DGX-Sparks-TensorFold
cp scripts/local.sh.example scripts/local.sh # set [email protected] in it
./start.shWORKER is the worker's ssh target; if it is on another network than the link, set FABRIC_PEER to its CX7 address. The worker needs no copy of the repository. The first run sets up both Sparks: the image on each (pulled from GitHub Container Registry) and the checkpoint downloaded on the head. Later starts load half of the weights on each Spark in about two minutes. It runs a smoke test through both ranks and prints Qwen3.8-Flash-Next is now LIVE! on port 8888.
Talk to it
Any OpenAI client works with base_url = "http://<head-address>:8888/v1" and the model Qwen3.8-Flash-Next (Qwen3.8-Flash-Next-INT4-AR with QUANT=int4ar). The model thinks before it answers (reasoning_content), so give replies enough max_tokens.
curl -s http://<head-address>:8888/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "Qwen3.8-Flash-Next",
"messages": [{"role": "user", "content": "Write a Python fibonacci function."}],
"max_tokens": 2000
}'Run it day to day
./start.sh restart # restart both ranks, e.g. after changing a setting
./stop.sh # stop both ranks and free their GPU memory
docker logs -f qwen38-fn-tf # rank 0's log (here)
ssh <worker> docker logs -f qwen38-fn-tf # rank 1's log
curl -s http://<head-address>:8888/health # status, memory, live streamsImages and video
Up to 50 images a request.
The checkpoint's own 27-layer vision tower (0.84 GiB) turns pictures and video frames into tokens, and the engine takes them as it takes text. They work with the FP8 KV cache, drafts, and both checkpoints. Send an image_url or video_url part in a user message.
| Images | Videos | |
|---|---|---|
| Formats | JPEG, PNG, WebP | MP4, WebM, MOV, MKV (anything FFmpeg decodes) |
| Per request | Up to 50 (all of a chat's turns count) | Up to 4 |
| Tokens | Up to 16,384 for all images, at most 4,096 an image | 2 frames a second (at most 256), up to 16,384 tokens a request |
Only data URLs by default; VISION_URLS=1 also fetches public https:// URLs. VISION=0 serves text only and gives its 2 GiB back to the KV pool.
Worker weights over NFS
One copy of the weights.
The setup this recipe is built around: with WORKER_WEIGHTS=nfs the worker keeps no copy of the checkpoint and reads the head's Hugging Face cache over NFS, read-only. Export it from the head once (this needs root; the worker needs nothing installed):
sudo apt install nfs-kernel-server
echo "$HOME/.cache/huggingface <worker CX7 address>(ro,no_subtree_check)" | sudo tee -a /etc/exports
sudo exportfs -raThen WORKER_WEIGHTS=nfs in scripts/local.sh and ./start.sh restart. WORKER_WEIGHTS=copy (the default without scripts/local.sh) copies the ~124 GiB to the worker over the link instead.
Memory and settings
A 3.9M-token pool across two Sparks.
The KV pool
The engine sizes its pool when it loads, from the memory left after the weights, less a 10 GiB reserve. Caches grow inside it as requests need them, and a request is admitted when its prompt plus its reply budget fits beside the others.
| At the defaults, per rank | NVFP4 | INT4-AR |
|---|---|---|
| Requests at once | 16 | 16 |
| Window per request | 1,048,576 | 1,048,576 |
| Engine on its GPU after loading | ~69 GiB | ~65 GiB |
| KV pool | 26.5 GiB | 31.0 GiB |
| KV pool in tokens (FP8) | ~3.9M | ~4.6M |
| Full 1M-token conversations at once | 3 | 4 |
The FP8 cache is lossy: its top-1 token agreed with a bf16 cache's 98.8% of the time, and it makes the pool 1.8-1.9x larger at the same speed. KV_DTYPE=bf16 gives the exact cache.
The ones you might change
Every setting lives in scripts/config.sh; override one from the environment (PARALLEL=2 ./start.sh restart), scripts/local.sh or a .env file. The full list is in the README.
| Variable | Default | Meaning |
|---|---|---|
WORKER | empty | The worker's ssh target (FABRIC_PEER: its CX7 address) |
QUANT | nvfp4 | nvfp4 or int4ar |
PARALLEL | 16 | Requests decoded together, 1 to 16 |
CONTEXT | 1048576 | Window per request; past 262,144 it uses YaRN |
KV_DTYPE | fp8 | fp8 (lossy, larger pool) or bf16 (exact) |
WORKER_WEIGHTS | copy | copy or nfs (the example file sets nfs) |
VISION | 1 | Image and video input; 0 serves text only |
THINKING | 1 | Think before answering; 0 answers directly |
MAX_TOKENS | 32768 | Reply budget for a request without max_tokens |
PORT | 8888 | Where the API listens |
Licenses
Free, with a few rules.
- The recipe is Apache 2.0; its NOTICE carries TensorFold's notices, and TensorFold's code stays under TensorFold's licenses.
- The NVFP4 checkpoint is under the NVIDIA Open Model License, with the Qwen Community License 1.0 as additional information; the INT4-AR checkpoint is under the license on its model card.
- The image is based on NVIDIA's PyTorch container; by pulling or running it you accept NVIDIA's terms, which the container prints at every start.
Credits
Built on the work of others.
- TensorFold and its Zig engine by Ash Hart and the TensorFold contributors
- The Zig CUDA serving path and CUDA family registry by Jürgen Schmied (TensorFold #443)
- Qwen3.8 Flash Next by the Qwen team (Alibaba)
- The NVFP4 checkpoint by NVIDIA, made with NVIDIA Model Optimizer
- The INT4-AR checkpoint, its expert cut and shared-expert healing by azampatti, on Intel's AutoRound quantization and Saren-Arterius's hybrid checkpoint and FP8 n-gram table
- The RoCEnante protocol of b12x by local-inference-lab, which the one-shot RoCE all-gather implements in new code
- Zig by the Zig Software Foundation and contributors
The full list, including the runtime stack, is in the recipe's CREDITS.md.
Run it on your Sparks.
Free and open, with every patch and every number published.