Recipe · One DGX Spark
Qwen3.8 Flash Next on one DGX Spark, with TensorFold.
Serve Qwen3.8 Flash Next from a single DGX Spark through an OpenAI-compatible API, with up to 5 requests at once at a 262,144-token context and image and video input. It runs TensorFold 1.1.4's Zig engine with two checkpoints to choose from: NVIDIA's FP4 QAD export, close to bf16, by default, or the faster INT4-AutoRound.
What's new · 18 October 2026
NVIDIA's FP4, now the default.
./start.sh now serves NVIDIA's quantization-aware-distilled FP4 checkpoint. azampatti's INT4-AutoRound, the checkpoint of recipe 1.0, stays one setting away (CHECKPOINT=int4). The full notes are in the recipe's CHANGELOG.
Close to bf16
NVIDIA trained the 4-bit weights against the bf16 model as teacher. On NVIDIA's own numbers it scores 91.5% on GPQA Diamond and 88.7% on ToolEvalBench, against 92.2% and 89.4% for bf16 (model card, not measured here). It keeps all 512 routed experts, 10 a token.
Served by the Zig engine
Two new patches teach the Zig engine NVIDIA's formats: CSF-compressed NVFP4 expert scales, MXFP8 linears, the quantized vision tower and the 26.8 GiB NVFP4 n-gram table, read from disk with read-ahead. A third tunes the drafts and keeps a repeated prompt's state.
Prompts read 13 to 15% faster
New on 18 October (patch 0014): a prompt's routed experts run through a shared-memory tile kernel, on by default, with bit-identical replies. Prefill went from 2,325 to 2,665 tok/s at 4k and from 1,947 to 2,197 at 128k, measured on that build. Decode is unchanged.
Fair prefill, opt-in
Patch 0015: with TF_FLASHNEXT_FILL_FAIR=1, prompts are read in slices, shortest first. A short request behind five 100k-token prompts waited 1.8 s at the median instead of 187 s, and a running reply kept flowing. The long prompts finish later, and a mixed load takes about 20% longer in total.
INT4 is one setting away
CHECKPOINT=int4 ./start.sh restart switches to INT4-AutoRound, which is faster on this hardware and about 10% lower in general capability on its authors' harness. Both can sit in the cache side by side, so switching is just a restart.
Fixes and changes
- Five requests at once by default (was 8): five full 262,144-token windows fit the pool the FP4 checkpoint leaves.
start.shreads free memory on non-English systems, reported by shuaixn and Siphi1239.tools/bench.pyreads the 1.0.0 stream stats, reported by naeemakhtar110. The ready summary names the KV cache in use, by mcclanahanaman.
Performance
Measured, not promised.
NVIDIA FP4 QAD · the default
One DGX Spark on TensorFold 1.1.4, the recipe's defaults (a 262,144-token window, 5 requests at once, FP8 KV cache, vision on, MTP drafts), measured with sparkDash on the Spark itself. Aggregate is the sum of the per-request decode speeds. Prose was measured at temperature 0.7 with thinking on, code at temperature 0 with thinking off.
Decode, prose
All requests together, per request, and time to first token. Temperature 0.7, thinking on.
| Requests | Aggregate | Per request | First token |
|---|---|---|---|
| 1 | 55.6 tok/s | 55.6 tok/s | 136 ms |
| 2 | 95.4 tok/s | 49.4 tok/s | 234 ms |
| 4 | 145.1 tok/s | 38.1 tok/s | 300 ms |
| 5 | 156.1 tok/s | 35.0 tok/s | 370 ms |
Decode, code
The same benchmark with a code prompt, at temperature 0 with thinking off.
| Requests | Aggregate | Per request | First token |
|---|---|---|---|
| 1 | 110.5 tok/s | 110.5 tok/s | 164 ms |
| 2 | 147.2 tok/s | 78.7 tok/s | 212 ms |
| 4 | 192.9 tok/s | 52.7 tok/s | 374 ms |
| 5 | 217.6 tok/s | 48.0 tok/s | 403 ms |
Prefill
Cold prompts, one request: how fast a prompt is read before the reply starts.
| Prompt | Prefill | First token |
|---|---|---|
| 4,134 tokens | 3,050 tok/s | 1.52 s |
| 8,231 tokens | 2,956 tok/s | 2.95 s |
| 16,421 tokens | 2,839 tok/s | 5.95 s |
| 32,807 tokens | 2,735 tok/s | 12.16 s |
| 65,574 tokens | 2,230 tok/s | 29.57 s |
| 131,110 tokens | 2,269 tok/s | 57.94 s |
The 18 October build read 2,665 tok/s at 4k, 2,501 at 16k and 2,197 at 128k.
Checked, not assumed
- Greedy replies identical drafted, serial and repeated, on seven prompts.
- A passphrase found in a 194,723-token prompt (prefill 99.4 s).
- Five 100k-token prompts and six short ones at once: all answered right, no crash or hang, with memory in use peaking at 98.6 GiB. Prefill is serial by default, so short requests waited for the long ones;
TF_FLASHNEXT_FILL_FAIR=1cuts that wait to about 2 s. - Tool calls and image input pass (
tools/toolcheck.py,tools/visioncheck.py).
INT4-AutoRound · CHECKPOINT=int4
The numbers recipe 1.0 published, measured with sparkDash on 8 October 2026 before the FP4 support was added: FP8 KV, vision on, PARALLEL 8, a 262,144-token window, greedy, thinking off. Here the aggregate is every request's tokens over the time from the first token of the earliest to the last token of the latest, so it is not measured the same way as the FP4 tables.
Decode, prose
All requests together, per request, and time to first token.
| Requests | Aggregate | Per request | First token |
|---|---|---|---|
| 1 | 64.4 tok/s | 64.4 tok/s | 98 ms |
| 2 | 89.0 tok/s | 46.4 tok/s | 180 ms |
| 4 | 140.7 tok/s | 37.2 tok/s | 216 ms |
| 8 | 200.9 tok/s | 26.7 tok/s | 405 ms |
Decode, code
The lab bench on the same boot: a short Python function, 256 tokens.
| Requests | Aggregate | Per request | First token |
|---|---|---|---|
| 1 | 57.5 tok/s | 57.5 tok/s | 113 ms |
| 2 | 92.7 tok/s | 73.9 tok/s | 166 ms |
| 4 | 130.8 tok/s | 49.6 tok/s | 194 ms |
| 8 | 189.3 tok/s | 40.1 tok/s | 379 ms |
Prefill
One request: how fast a prompt is read before the reply starts.
| Prompt | Prefill | First token |
|---|---|---|
| 4,134 tokens | 2,526 tok/s | 1.64 s |
| 8,232 tokens | 2,606 tok/s | 3.16 s |
| 16,422 tokens | 2,643 tok/s | 6.21 s |
| 32,804 tokens | 2,630 tok/s | 12.47 s |
| 65,578 tokens | 2,564 tok/s | 25.57 s |
| 131,111 tokens | 2,415 tok/s | 54.30 s |
Long prompts, images and video
A passphrase hidden in a 199,730-token prompt was found, after a 100.2 s prefill.
One 320×240 image reached the first token in 0.22 s, ten of them in 0.65 s, and a 30-second 320×240 video in 2.02 s.
Against the earlier Python recipe (MLX 4-bit, int8 KV, n-gram tables on SSD): prose decode was 62.4 / 90.5 / 106.7 / 119.3 tok/s at 1 / 2 / 4 / 5 requests, and prefill 2,503 / 2,520 / 2,499 / 2,414 / 2,200 tok/s from 8k to 128k.
Speed preview
What 55.6 tok/s feels like.
Sample text played at the measured decode speed with the default FP4 checkpoint: one request at 55.6 tok/s, or four at once at 38.1 tok/s each (145.1 combined), at about four characters a token. Switch between them in the window.
Choose a checkpoint
Close to bf16, or the faster cut.
CHECKPOINT=nvfp4 · default
NVIDIA FP4 QAD
nvidia/Qwen3.8-Flash-Next-FP4-Local, ~96 GiB: all 512 routed experts, 10 a token. Routed experts in NVFP4, attention and shared-expert linears in MXFP8, the MTP experts and the vision tower quantized too, and an n-gram table read from disk. Quantization-aware distilled, so close to bf16.
Prose 55.6 tok/s alone, 156.1 for 5 requests; prefill ~2,230-3,050 tok/s.
CHECKPOINT=int4
INT4-AutoRound
azampatti's INT4-AutoRound, ~122 GiB: 5 routed experts a token instead of 10 (4.8B active parameters), GPTQ int4 experts and an FP8 n-gram table on the GPU. Faster on this hardware, and about 10% lower general capability on its authors' own harness (46.6-47.6 against 51.8); tool use is unchanged.
Prose 64.4 tok/s alone, 140.7 for 4 requests, 200.9 for 8; prefill ~2,400-2,650 tok/s.
FP4 (nvfp4, default) | INT4 (int4) | |
|---|---|---|
| Experts a token | 10 of 512 (all kept) | 5 (top-5 cut, 4.8B active) |
| Quality | Quantization-aware distilled, close to bf16 | About 10% lower general capability on its authors' harness |
| Download / weights in memory | ~96 GiB / ~81 GiB | ~122 GiB / ~66 GiB |
| Free memory needed at start | ~108 GiB | ~93 GiB |
| KV pool on the measured Spark | 24.36 GiB: five full windows | 30.22 GiB: six full windows |
Requests at once (PARALLEL) | 5 | 5 (the INT4 numbers were measured with 8) |
CHECKPOINT=int4 ./start.sh restart # switch to INT4 (downloads it the first time)
echo 'CHECKPOINT=int4' >> .env # keep INT4 as your default
CHECKPOINT=nvfp4 ./start.sh restart # back to FP4
The image and the patches are the same for both: the engine reads the format from the checkpoint. Both can sit in ~/.cache/huggingface side by side (~96 + ~122 GiB), so switching only restarts the server. docker inspect -f '{{.Config.Cmd}}' qwen38-flash-next-tf shows which one is running.
The engine
Powered by TensorFold.
TensorFold, by Ash Hart, is an open-source engine that serves language models on Apple Silicon and NVIDIA GPUs through an OpenAI-compatible API. Each model family brings its own kernels and its own draft verification, tuned for that model instead of one generic path for all.
Exact decoding
Drafts make it fast without changing a word: a draft is accepted only when it equals the token the same engine would produce one at a time. Send a request with "draft": false to compare.
Apple Silicon and NVIDIA
MLX on Macs, and CUDA on NVIDIA GPUs from compute capability 8.9: Ada (RTX 40), Hopper and Blackwell, including the DGX Spark's GB10 and the RTX 50 series.
Many model families
Qwen3.8 Flash Next, Qwen3.8-27B, GLM-5.3-Flash, Nemotron 3.5 Lightning, Gemma 4 and DeepSeek-V4-Flash, among others, each with its own kernels.
In this recipe
TensorFold's Zig engine, tensorfold-native, on one GPU: the Flash Next CUDA engine ported from TensorFold's Python engine, on the Zig serving path by Jürgen Schmied (TensorFold PR #443). The recipe's patches are the engine itself, applied to TensorFold 1.1.4 at a pinned commit; five of them (0011 to 0015), by Mia's AI Lab, add NVIDIA's FP4 checkpoint, tune the drafts and speed up the prompt pass. Drafts are still checked: a drafted reply matches "draft": false.
What you need
One Spark, nothing else.
A DGX Spark
Or another GB10 system with 128 GB unified memory, with nothing else large on the GPU: the default window needs about 108 GiB free when the server starts with the FP4 checkpoint (loading takes ~81 GiB), or 93 GiB with INT4 (weights ~66 GiB). start.sh stops before launching below that.
Docker
Docker with the NVIDIA container runtime, and your user in the docker group.
Disk
About 160 GB free on a fresh machine: the checkpoint under ~/.cache/huggingface (~96 GiB for FP4, ~122 GiB for INT4), plus the image; scripts/prepare.sh checks both. Keep at least 100 GB free.
Optional
The hf CLI on the host and a Hugging Face token in ~/.cache/huggingface/token or HF_TOKEN. The checkpoint is public, so neither is required.
Quick start
Three lines, that's all.
Clone and start
git clone https://github.com/MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold.git
cd Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold
./start.sh
The first run sets everything up: it pulls the matching prebuilt engine image (or builds it: TensorFold 1.1.4 at the pinned commit plus the recipe's patches, then the GPU kernels) and downloads the checkpoint. Later starts just load the weights. It runs a smoke test and prints Qwen3.8-Flash-Next is now LIVE! on port 8888.
Talk to it
Any OpenAI client works with base_url = "http://<spark-address>:8888/v1" and the model Qwen3.8-Flash-Next. Streaming, tool calls, reasoning content, images and videos are supported, and so is Anthropic's /v1/messages. The model thinks before it answers, so give replies enough max_tokens; a request without one gets 32,768.
curl -s http://<spark-address>:8888/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "Qwen3.8-Flash-Next",
"messages": [{"role": "user", "content": "Write a Python fibonacci function."}],
"max_tokens": 1000
}'
Run it day to day
./start.sh restart # restart it, e.g. after changing a setting
./stop.sh # stop the server and free the GPU memory
docker logs -f qwen38-flash-next-tf # server log
curl -s http://<spark-address>:8888/health # server health
Images and video
Up to 50 images a request.
The model's own vision tower (27 layers, 0.84 GiB, from the same checkpoint) turns images and video frames into tokens. Send them as OpenAI-style content parts in a user message: an image_url part, or a video_url part for a clip.
| Images | Videos | |
|---|---|---|
| Formats | JPEG, PNG, WebP | MP4, WebM, MOV, MKV (anything FFmpeg decodes) |
| Per request | Up to 50 (all of a chat's turns count), 10 MB each, 64 MB in all | Up to 4, 64 MB each, 96 MB in all, up to an hour of footage |
| Tokens | Up to 16,384 for all images, at most 4,096 an image (256 with "detail": "low") | 2 frames a second (at most 256 frames), up to 16,384 tokens a request |
Image and video prompts are not kept for reuse, so each turn of a chat with images reads them again. VISION=0 ./start.sh restart serves text only and gives the helper's 2 GiB back to the cache.
Thinking and the API
OpenAI and Anthropic, one server.
By default the model thinks first; the reasoning comes back in reasoning_content and the answer in content. "chat_template_kwargs": {"enable_thinking": false} answers directly, and "reasoning_effort": "low" (or "medium", "xhigh") in the same place sets the effort.
/v1/chat/completions,/v1/completions,/v1/responses, Anthropic's/v1/messages,/v1/models,/tokenize,/healthand Prometheus/metrics.- Tool calls, including
tool_choice: "required"and a named function. - A conversation's next turn resumes from the kept state of its earlier turns.
- Refused with HTTP 400: structured outputs (
response_format,guided_*),logprobs, andnother than 1.
Memory and settings
Five full windows on one Spark.
The KV pool
The FP4 checkpoint takes about 81 GiB to load (its 26.8 GiB n-gram table is read from disk, not held on the GPU). What is left, minus a 10 GiB floor and 2 GiB for the vision helper, is one shared pool, and each request's cache grows as it goes. On the measured Spark that was 24.36 GiB, about 1.46 million FP8 tokens: room for five requests at the full 262,144 tokens. With INT4 (65.7 GiB of weights) it was 30.22 GiB, six full windows. A request is admitted when its prompt plus max_tokens fits beside the others; one that doesn't fit is refused, not queued.
| KV cache | A token | A 262,144-token request |
|---|---|---|
fp8 (default) | 17,888 bytes | 4.37 GiB |
bf16 (exact) | 30,784 bytes | 7.52 GiB |
A 1,048,576-token window (CONTEXT=1048576) is one 17.46 GiB FP8 cache; it is not the default. Keep at least 10 GiB free under load: on this machine, running out of unified memory freezes it.
The ones you might change
Every setting lives in scripts/config.sh; override one from the environment (PARALLEL=2 ./start.sh) or a .env file. The full list is in the README.
| Variable | Default | Meaning |
|---|---|---|
CHECKPOINT | nvfp4 | NVIDIA's FP4 QAD; int4 serves INT4-AutoRound |
PARALLEL | 5 | Requests decoded together |
CONTEXT | 262144 | Prompt + reply window; 1048576 turns on YaRN factor 4 |
KV_DTYPE | fp8 | bf16 is the exact cache, with a smaller pool |
VISION | 1 | Image and video input; 0 serves text only |
DRAFTS | 1 | MTP drafts; 0 turns them off |
TF_FLASHNEXT_FILL_FAIR | 0 | 1 reads prompts in slices, shortest first, so short requests don't wait behind long ones (a mixed load takes ~20% longer) |
TF_FLASHNEXT_PROMPT_EXPERT_TILE | 2 | The faster prompt kernel (+13 to 15% prefill, same output); 0 uses the older kernels |
THINKING | 1 | Think before answering; 0 answers directly |
MAX_TOKENS | 32768 | Reply length for a request without max_tokens |
TENSORFOLD_MEMORY_RESERVE_GIB | 10 | Kept free when the pool is sized; the scripts refuse a Spark with less |
PORT | 8888 | Where the API listens |
Licenses
Free, with a few rules.
- The recipe is MIT, and its LICENSE also carries TensorFold's MIT notice for the patches.
- The model weights, downloaded from Hugging Face, are under their checkpoint's own license: the default FP4 checkpoint under the NVIDIA Open Model Agreement (its base model is under the Qwen Community License 1.0), and the INT4-AR checkpoint under the Qwen Community License 1.0, per its model card.
- The image is based on NVIDIA's PyTorch container; by pulling or running it you accept NVIDIA's terms, which the container prints at every start. Its vision helper uses Hugging Face transformers, PyAV with FFmpeg, and Pillow.
Credits
Built on the work of others.
- TensorFold and its Zig engine by Ash Hart and the TensorFold contributors
- The Zig CUDA serving path by Jürgen Schmied (TensorFold PR #443)
- NVIDIA FP4 support in the Zig engine (patches 0011 to 0015) by Mia's AI Lab, built on TensorFold's Python implementation of the same formats
- Qwen3.8 Flash Next by Qwen
- The default FP4 QAD checkpoint by NVIDIA, made with NVIDIA Model Optimizer
- The optional INT4-AR checkpoint by azampatti (its top-5 expert cut and shared-expert healing), on Intel's AutoRound quantization, with the hybrid checkpoint and FP8 n-gram table by Saren-Arterius
- The full-window admission fix by 321sssrt-bit; script fixes reported by shuaixn, Siphi1239 and naeemakhtar110, and one by mcclanahanaman
- From the earlier Python recipe: Vontra's checkpoint, a prompt-chunk change by MovieMaker93 and the draft-language lists by Javier (jvr0x)
The full list, including the runtime stack, is in the recipe's CREDITS.md.
Run it on your Spark.
Free and open, with every patch and every number published.