Recipe · One DGX Spark

Qwen3.8 Flash Next on one DGX Spark, with TensorFold.

Serve Qwen3.8 Flash Next from a single DGX Spark through an OpenAI-compatible API, with up to 5 requests at once at a 262,144-token context and image and video input. It runs TensorFold 1.1.4's Zig engine with two checkpoints to choose from: NVIDIA's FP4 QAD export, close to bf16, by default, or the faster INT4-AutoRound.

Get the free recipe

Install with an AI agent

Install this model

Paste this into an AI coding agent that can run commands on the machine that will serve the model (the head node, for a Spark cluster), such as Claude Code or Codex. It reads the recipe, adapts it to your setup, starts the server and checks that it works.


        
Qwen3.8 Flash Next with TensorFold's Zig engine on one DGX Spark: NVIDIA's FP4 QAD checkpoint at 55.6 tok/s for one request and 156.1 for five at once, or INT4-AutoRound at 64.4 tok/s

What's new · 18 October 2026

NVIDIA's FP4, now the default.

./start.sh now serves NVIDIA's quantization-aware-distilled FP4 checkpoint. azampatti's INT4-AutoRound, the checkpoint of recipe 1.0, stays one setting away (CHECKPOINT=int4). The full notes are in the recipe's CHANGELOG.

Close to bf16

NVIDIA trained the 4-bit weights against the bf16 model as teacher. On NVIDIA's own numbers it scores 91.5% on GPQA Diamond and 88.7% on ToolEvalBench, against 92.2% and 89.4% for bf16 (model card, not measured here). It keeps all 512 routed experts, 10 a token.

Served by the Zig engine

Two new patches teach the Zig engine NVIDIA's formats: CSF-compressed NVFP4 expert scales, MXFP8 linears, the quantized vision tower and the 26.8 GiB NVFP4 n-gram table, read from disk with read-ahead. A third tunes the drafts and keeps a repeated prompt's state.

Prompts read 13 to 15% faster

New on 18 October (patch 0014): a prompt's routed experts run through a shared-memory tile kernel, on by default, with bit-identical replies. Prefill went from 2,325 to 2,665 tok/s at 4k and from 1,947 to 2,197 at 128k, measured on that build. Decode is unchanged.

Fair prefill, opt-in

Patch 0015: with TF_FLASHNEXT_FILL_FAIR=1, prompts are read in slices, shortest first. A short request behind five 100k-token prompts waited 1.8 s at the median instead of 187 s, and a running reply kept flowing. The long prompts finish later, and a mixed load takes about 20% longer in total.

INT4 is one setting away

CHECKPOINT=int4 ./start.sh restart switches to INT4-AutoRound, which is faster on this hardware and about 10% lower in general capability on its authors' harness. Both can sit in the cache side by side, so switching is just a restart.

Fixes and changes

  • Five requests at once by default (was 8): five full 262,144-token windows fit the pool the FP4 checkpoint leaves.
  • start.sh reads free memory on non-English systems, reported by shuaixn and Siphi1239.
  • tools/bench.py reads the 1.0.0 stream stats, reported by naeemakhtar110. The ready summary names the KV cache in use, by mcclanahanaman.

Performance

Measured, not promised.

One request55.6tok/s, prose (FP4)
Five at once156.1tok/s combined (FP4)
With INT4-AR64.4tok/s, one request
Context262ktokens a request, up to 5 at once

NVIDIA FP4 QAD · the default

One DGX Spark on TensorFold 1.1.4, the recipe's defaults (a 262,144-token window, 5 requests at once, FP8 KV cache, vision on, MTP drafts), measured with sparkDash on the Spark itself. Aggregate is the sum of the per-request decode speeds. Prose was measured at temperature 0.7 with thinking on, code at temperature 0 with thinking off.

Decode, prose

All requests together, per request, and time to first token. Temperature 0.7, thinking on.

RequestsAggregatePer requestFirst token
155.6 tok/s55.6 tok/s136 ms
295.4 tok/s49.4 tok/s234 ms
4145.1 tok/s38.1 tok/s300 ms
5156.1 tok/s35.0 tok/s370 ms

Decode, code

The same benchmark with a code prompt, at temperature 0 with thinking off.

RequestsAggregatePer requestFirst token
1110.5 tok/s110.5 tok/s164 ms
2147.2 tok/s78.7 tok/s212 ms
4192.9 tok/s52.7 tok/s374 ms
5217.6 tok/s48.0 tok/s403 ms

Prefill

Cold prompts, one request: how fast a prompt is read before the reply starts.

PromptPrefillFirst token
4,134 tokens3,050 tok/s1.52 s
8,231 tokens2,956 tok/s2.95 s
16,421 tokens2,839 tok/s5.95 s
32,807 tokens2,735 tok/s12.16 s
65,574 tokens2,230 tok/s29.57 s
131,110 tokens2,269 tok/s57.94 s

The 18 October build read 2,665 tok/s at 4k, 2,501 at 16k and 2,197 at 128k.

Checked, not assumed

  • Greedy replies identical drafted, serial and repeated, on seven prompts.
  • A passphrase found in a 194,723-token prompt (prefill 99.4 s).
  • Five 100k-token prompts and six short ones at once: all answered right, no crash or hang, with memory in use peaking at 98.6 GiB. Prefill is serial by default, so short requests waited for the long ones; TF_FLASHNEXT_FILL_FAIR=1 cuts that wait to about 2 s.
  • Tool calls and image input pass (tools/toolcheck.py, tools/visioncheck.py).

INT4-AutoRound · CHECKPOINT=int4

The numbers recipe 1.0 published, measured with sparkDash on 8 October 2026 before the FP4 support was added: FP8 KV, vision on, PARALLEL 8, a 262,144-token window, greedy, thinking off. Here the aggregate is every request's tokens over the time from the first token of the earliest to the last token of the latest, so it is not measured the same way as the FP4 tables.

Decode, prose

All requests together, per request, and time to first token.

RequestsAggregatePer requestFirst token
164.4 tok/s64.4 tok/s98 ms
289.0 tok/s46.4 tok/s180 ms
4140.7 tok/s37.2 tok/s216 ms
8200.9 tok/s26.7 tok/s405 ms

Decode, code

The lab bench on the same boot: a short Python function, 256 tokens.

RequestsAggregatePer requestFirst token
157.5 tok/s57.5 tok/s113 ms
292.7 tok/s73.9 tok/s166 ms
4130.8 tok/s49.6 tok/s194 ms
8189.3 tok/s40.1 tok/s379 ms

Prefill

One request: how fast a prompt is read before the reply starts.

PromptPrefillFirst token
4,134 tokens2,526 tok/s1.64 s
8,232 tokens2,606 tok/s3.16 s
16,422 tokens2,643 tok/s6.21 s
32,804 tokens2,630 tok/s12.47 s
65,578 tokens2,564 tok/s25.57 s
131,111 tokens2,415 tok/s54.30 s

Long prompts, images and video

A passphrase hidden in a 199,730-token prompt was found, after a 100.2 s prefill.

One 320×240 image reached the first token in 0.22 s, ten of them in 0.65 s, and a 30-second 320×240 video in 2.02 s.

Against the earlier Python recipe (MLX 4-bit, int8 KV, n-gram tables on SSD): prose decode was 62.4 / 90.5 / 106.7 / 119.3 tok/s at 1 / 2 / 4 / 5 requests, and prefill 2,503 / 2,520 / 2,499 / 2,414 / 2,200 tok/s from 8k to 128k.

Speed preview

What 55.6 tok/s feels like.

Sample text played at the measured decode speed with the default FP4 checkpoint: one request at 55.6 tok/s, or four at once at 38.1 tok/s each (145.1 combined), at about four characters a token. Switch between them in the window.

TensorFold · Qwen3.8-Flash-Next · 1× DGX Spark
55.6 tok/sSample text, played at the measured speed

Choose a checkpoint

Close to bf16, or the faster cut.

CHECKPOINT=nvfp4 · default

NVIDIA FP4 QAD

nvidia/Qwen3.8-Flash-Next-FP4-Local, ~96 GiB: all 512 routed experts, 10 a token. Routed experts in NVFP4, attention and shared-expert linears in MXFP8, the MTP experts and the vision tower quantized too, and an n-gram table read from disk. Quantization-aware distilled, so close to bf16.

Prose 55.6 tok/s alone, 156.1 for 5 requests; prefill ~2,230-3,050 tok/s.

CHECKPOINT=int4

INT4-AutoRound

azampatti's INT4-AutoRound, ~122 GiB: 5 routed experts a token instead of 10 (4.8B active parameters), GPTQ int4 experts and an FP8 n-gram table on the GPU. Faster on this hardware, and about 10% lower general capability on its authors' own harness (46.6-47.6 against 51.8); tool use is unchanged.

Prose 64.4 tok/s alone, 140.7 for 4 requests, 200.9 for 8; prefill ~2,400-2,650 tok/s.

FP4 (nvfp4, default)INT4 (int4)
Experts a token10 of 512 (all kept)5 (top-5 cut, 4.8B active)
QualityQuantization-aware distilled, close to bf16About 10% lower general capability on its authors' harness
Download / weights in memory~96 GiB / ~81 GiB~122 GiB / ~66 GiB
Free memory needed at start~108 GiB~93 GiB
KV pool on the measured Spark24.36 GiB: five full windows30.22 GiB: six full windows
Requests at once (PARALLEL)55 (the INT4 numbers were measured with 8)
CHECKPOINT=int4 ./start.sh restart     # switch to INT4 (downloads it the first time)
echo 'CHECKPOINT=int4' >> .env         # keep INT4 as your default
CHECKPOINT=nvfp4 ./start.sh restart    # back to FP4

The image and the patches are the same for both: the engine reads the format from the checkpoint. Both can sit in ~/.cache/huggingface side by side (~96 + ~122 GiB), so switching only restarts the server. docker inspect -f '{{.Config.Cmd}}' qwen38-flash-next-tf shows which one is running.

The engine

Powered by TensorFold.

TensorFold, by Ash Hart, is an open-source engine that serves language models on Apple Silicon and NVIDIA GPUs through an OpenAI-compatible API. Each model family brings its own kernels and its own draft verification, tuned for that model instead of one generic path for all.

Exact decoding

Drafts make it fast without changing a word: a draft is accepted only when it equals the token the same engine would produce one at a time. Send a request with "draft": false to compare.

Apple Silicon and NVIDIA

MLX on Macs, and CUDA on NVIDIA GPUs from compute capability 8.9: Ada (RTX 40), Hopper and Blackwell, including the DGX Spark's GB10 and the RTX 50 series.

Many model families

Qwen3.8 Flash Next, Qwen3.8-27B, GLM-5.3-Flash, Nemotron 3.5 Lightning, Gemma 4 and DeepSeek-V4-Flash, among others, each with its own kernels.

In this recipe

TensorFold's Zig engine, tensorfold-native, on one GPU: the Flash Next CUDA engine ported from TensorFold's Python engine, on the Zig serving path by Jürgen Schmied (TensorFold PR #443). The recipe's patches are the engine itself, applied to TensorFold 1.1.4 at a pinned commit; five of them (0011 to 0015), by Mia's AI Lab, add NVIDIA's FP4 checkpoint, tune the drafts and speed up the prompt pass. Drafts are still checked: a drafted reply matches "draft": false.

What you need

One Spark, nothing else.

A DGX Spark

Or another GB10 system with 128 GB unified memory, with nothing else large on the GPU: the default window needs about 108 GiB free when the server starts with the FP4 checkpoint (loading takes ~81 GiB), or 93 GiB with INT4 (weights ~66 GiB). start.sh stops before launching below that.

Docker

Docker with the NVIDIA container runtime, and your user in the docker group.

Disk

About 160 GB free on a fresh machine: the checkpoint under ~/.cache/huggingface (~96 GiB for FP4, ~122 GiB for INT4), plus the image; scripts/prepare.sh checks both. Keep at least 100 GB free.

Optional

The hf CLI on the host and a Hugging Face token in ~/.cache/huggingface/token or HF_TOKEN. The checkpoint is public, so neither is required.

Quick start

Three lines, that's all.

1

Clone and start

git clone https://github.com/MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold.git
cd Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold
./start.sh

The first run sets everything up: it pulls the matching prebuilt engine image (or builds it: TensorFold 1.1.4 at the pinned commit plus the recipe's patches, then the GPU kernels) and downloads the checkpoint. Later starts just load the weights. It runs a smoke test and prints Qwen3.8-Flash-Next is now LIVE! on port 8888.

2

Talk to it

Any OpenAI client works with base_url = "http://<spark-address>:8888/v1" and the model Qwen3.8-Flash-Next. Streaming, tool calls, reasoning content, images and videos are supported, and so is Anthropic's /v1/messages. The model thinks before it answers, so give replies enough max_tokens; a request without one gets 32,768.

curl -s http://<spark-address>:8888/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "Qwen3.8-Flash-Next",
  "messages": [{"role": "user", "content": "Write a Python fibonacci function."}],
  "max_tokens": 1000
}'
3

Run it day to day

./start.sh restart                            # restart it, e.g. after changing a setting
./stop.sh                                     # stop the server and free the GPU memory
docker logs -f qwen38-flash-next-tf           # server log
curl -s http://<spark-address>:8888/health    # server health

Images and video

Up to 50 images a request.

The model's own vision tower (27 layers, 0.84 GiB, from the same checkpoint) turns images and video frames into tokens. Send them as OpenAI-style content parts in a user message: an image_url part, or a video_url part for a clip.

ImagesVideos
FormatsJPEG, PNG, WebPMP4, WebM, MOV, MKV (anything FFmpeg decodes)
Per requestUp to 50 (all of a chat's turns count), 10 MB each, 64 MB in allUp to 4, 64 MB each, 96 MB in all, up to an hour of footage
TokensUp to 16,384 for all images, at most 4,096 an image (256 with "detail": "low")2 frames a second (at most 256 frames), up to 16,384 tokens a request

Image and video prompts are not kept for reuse, so each turn of a chat with images reads them again. VISION=0 ./start.sh restart serves text only and gives the helper's 2 GiB back to the cache.

Thinking and the API

OpenAI and Anthropic, one server.

By default the model thinks first; the reasoning comes back in reasoning_content and the answer in content. "chat_template_kwargs": {"enable_thinking": false} answers directly, and "reasoning_effort": "low" (or "medium", "xhigh") in the same place sets the effort.

  • /v1/chat/completions, /v1/completions, /v1/responses, Anthropic's /v1/messages, /v1/models, /tokenize, /health and Prometheus /metrics.
  • Tool calls, including tool_choice: "required" and a named function.
  • A conversation's next turn resumes from the kept state of its earlier turns.
  • Refused with HTTP 400: structured outputs (response_format, guided_*), logprobs, and n other than 1.

Memory and settings

Five full windows on one Spark.

The KV pool

The FP4 checkpoint takes about 81 GiB to load (its 26.8 GiB n-gram table is read from disk, not held on the GPU). What is left, minus a 10 GiB floor and 2 GiB for the vision helper, is one shared pool, and each request's cache grows as it goes. On the measured Spark that was 24.36 GiB, about 1.46 million FP8 tokens: room for five requests at the full 262,144 tokens. With INT4 (65.7 GiB of weights) it was 30.22 GiB, six full windows. A request is admitted when its prompt plus max_tokens fits beside the others; one that doesn't fit is refused, not queued.

KV cacheA tokenA 262,144-token request
fp8 (default)17,888 bytes4.37 GiB
bf16 (exact)30,784 bytes7.52 GiB

A 1,048,576-token window (CONTEXT=1048576) is one 17.46 GiB FP8 cache; it is not the default. Keep at least 10 GiB free under load: on this machine, running out of unified memory freezes it.

The ones you might change

Every setting lives in scripts/config.sh; override one from the environment (PARALLEL=2 ./start.sh) or a .env file. The full list is in the README.

VariableDefaultMeaning
CHECKPOINTnvfp4NVIDIA's FP4 QAD; int4 serves INT4-AutoRound
PARALLEL5Requests decoded together
CONTEXT262144Prompt + reply window; 1048576 turns on YaRN factor 4
KV_DTYPEfp8bf16 is the exact cache, with a smaller pool
VISION1Image and video input; 0 serves text only
DRAFTS1MTP drafts; 0 turns them off
TF_FLASHNEXT_FILL_FAIR01 reads prompts in slices, shortest first, so short requests don't wait behind long ones (a mixed load takes ~20% longer)
TF_FLASHNEXT_PROMPT_EXPERT_TILE2The faster prompt kernel (+13 to 15% prefill, same output); 0 uses the older kernels
THINKING1Think before answering; 0 answers directly
MAX_TOKENS32768Reply length for a request without max_tokens
TENSORFOLD_MEMORY_RESERVE_GIB10Kept free when the pool is sized; the scripts refuse a Spark with less
PORT8888Where the API listens

Licenses

Free, with a few rules.

  • The recipe is MIT, and its LICENSE also carries TensorFold's MIT notice for the patches.
  • The model weights, downloaded from Hugging Face, are under their checkpoint's own license: the default FP4 checkpoint under the NVIDIA Open Model Agreement (its base model is under the Qwen Community License 1.0), and the INT4-AR checkpoint under the Qwen Community License 1.0, per its model card.
  • The image is based on NVIDIA's PyTorch container; by pulling or running it you accept NVIDIA's terms, which the container prints at every start. Its vision helper uses Hugging Face transformers, PyAV with FFmpeg, and Pillow.

Credits

Built on the work of others.

  • TensorFold and its Zig engine by Ash Hart and the TensorFold contributors
  • The Zig CUDA serving path by Jürgen Schmied (TensorFold PR #443)
  • NVIDIA FP4 support in the Zig engine (patches 0011 to 0015) by Mia's AI Lab, built on TensorFold's Python implementation of the same formats
  • Qwen3.8 Flash Next by Qwen
  • The default FP4 QAD checkpoint by NVIDIA, made with NVIDIA Model Optimizer
  • The optional INT4-AR checkpoint by azampatti (its top-5 expert cut and shared-expert healing), on Intel's AutoRound quantization, with the hybrid checkpoint and FP8 n-gram table by Saren-Arterius
  • The full-window admission fix by 321sssrt-bit; script fixes reported by shuaixn, Siphi1239 and naeemakhtar110, and one by mcclanahanaman
  • From the earlier Python recipe: Vontra's checkpoint, a prompt-chunk change by MovieMaker93 and the draft-language lists by Javier (jvr0x)

The full list, including the runtime stack, is in the recipe's CREDITS.md.

Run it on your Spark.

Free and open, with every patch and every number published.