Recipe · 1× RTX PRO 6000 · Zig engine
Qwen3.8 Flash Next on one RTX PRO 6000, with TensorFold's Zig engine.
Serve Qwen3.8 Flash Next from one 96 GB RTX PRO 6000 Blackwell, through an OpenAI-compatible API, with image and video input and 3 requests at once at a full 262,144-token window each. It runs TensorFold's Zig engine, v1.1.4 with 25 patches on top, and NVIDIA's FP4 checkpoint in its original form. 8,519.9 tok/s prefill on a 32K prompt at the card's default 600 W, and 271.4 tok/s for one request at 400 W. One command builds the engine, downloads the weights, starts the server and runs a smoke test.
Performance
Measured, not promised.
Speed depends on the card's power limit, and its default is 600 W. The official prefill numbers are at 600 W. Decode has not been measured at 600 W yet, so its numbers are at 400 W. NVIDIA's FP4 with all 25 patches, 3 requests at once at 262,144 tokens each, FP8 KV, vision and MTP drafts on, one run at each limit with sparkDash.
Prefill, at 600 W and 400 W
How fast a prompt is read, by prompt size, and the time to first token.
| Prompt | 600 W tok/s | First token | 400 W tok/s | First token |
|---|---|---|---|---|
| 4K | 9,218.8 | 518 ms | 8,321.9 | 567 ms |
| 8K | 8,811.8 | 1.00 s | 7,792.2 | 1.13 s |
| 16K | 8,878.6 | 1.92 s | 7,580.0 | 2.24 s |
| 32K | 8,519.9 | 3.92 s | 7,245.7 | 4.60 s |
| 64K | 8,357.7 | 7.92 s | 7,024.4 | 9.41 s |
| 128K | 7,886.9 | 16.69 s | 6,593.9 | 19.95 s |
Both of these runs overlapped a 5 GB checkpoint download on the same host, at a host load of about 1 on 24 threads.
Decode, prose, at 400 W
Temperature 0.7, thinking on: all requests together, per request, and the time to first token.
| Requests | Total tok/s | Per request | First token |
|---|---|---|---|
| 1 | 271.4 | 271.4 | 36 ms |
| 2 | 395.7 | 202.2 | 111 ms |
| 3 | 503.7 | 173.1 | 109 ms |
The recipe's README also lists both tables at 300 W.
Speed preview
What 271.4 tok/s feels like.
Sample text played at the speed measured at 400 W, at about four characters a token: one request at 271.4 tok/s, or three at once at 173.1 tok/s each (503.7 combined). Switch between them in the window.
Three ways to start it
NVIDIA's checkpoint, as it was published.
| Start with | What it serves | Status on this card |
|---|---|---|
./start.sh | NVIDIA's FP4 checkpoint in its original form, at a pinned revision: 3 requests at once, 262,144 tokens each | Checked |
ABLIT=1 ./start.sh | The same checkpoint with drowzeys's overlay, which removes safety refusals. Gated: accept its terms, then set HF_TOKEN | Never started |
YARN=1 ./start.sh | YaRN x4: one request at a time, up to 1,000,000 tokens. Off by default, because it changes replies on short prompts too | Not run on NVIDIA's FP4 yet |
The first ABLIT=1 start downloads the overlay's sidecar of about 190 MB. The checkpoint itself is neither downloaded again nor changed, and ABLIT=0 goes back to it. The results so far for the long window are in the recipe's details.
The engine
Powered by TensorFold.
TensorFold, by Ash Hart, is an open-source engine that serves language models on Apple Silicon and NVIDIA GPUs through an OpenAI-compatible API. This recipe runs its Zig engine, tensorfold-native, built in Docker from the v1.1.4 release with Zig 0.17.0 for the RTX PRO 6000's sm_120.
One card, TP=1
Every weight the kernels read is on the card. The checkpoint's n-gram table stays on disk, 26.8 GiB of it, and the host reads its rows while serving, which is why the checkpoint belongs on an NVMe drive.
Three full windows
FP8 KV takes 17,888 bytes a token against bf16's 30,784, so three windows of 262,144 tokens fit beside the weights, where bf16 fits one. It is lossy: about 98.8% top-1 agreement with a bf16 cache.
Prompts kept in host memory
A chat's next turn resumes from a state kept in pinned host memory, up to 16 GiB. On a 60,000-token prompt, follow-up turns resumed at worst 99.6% of it and took 0.5 to 0.6 s, against 11.9 s cold.
MTP drafts
The model's MTP head drafts tokens ahead, which speeds up decoding and never changes a reply. tools/exact.py checks that drafted replies equal the "draft": false ones.
On top of TensorFold
What the 25 patches do.
| Patches | What they do |
|---|---|
| 0001-0002 | The vision tower loads from NVIDIA's quantized weights, with an image budget of 16,384 tokens |
| 0003 | --kv-cache fp8 for Flash Next, so three full windows fit on the card where bf16 fits one |
| 0004, 0006 | Prefill: no drain between prompt chunks, vectorised hyper-connection kernels, a 128x128 MXFP8 prompt tile, 3,072-row chunks |
| 0005 | The FP8 attention reader keeps the next tile in registers; the MTP head's cache stays bf16 |
| 0007-0011 | Sampled requests join the batched draft pass: GPU top-k candidates, a round-cost model (off), n-gram rows of all lanes gathered at once, host draws on worker threads |
| 0012-0013 | A partly rejected draft window no longer replays the DeltaNet state; the n-gram table is read with pread and read ahead instead of through an mmap |
| 0014-0015 | Fused hyper-connection kernels, expert row tiles and MTP head pieces for prompt passes; a two-stage MX128 prompt tile |
| 0016 | The shared expert runs on a second stream in decode rounds; a shared round takes every lane's DeltaNet rows in three launches a layer |
| 0017 | The draft chain stops on the running product of its draft probabilities |
| 0018, 0020, 0022 | Video input on the Flash Next CUDA path: libtfvideo links on x86_64, each frame pair is one group through the tower, and the prompt is laid out as the checkpoint's processor does it |
| 0019 | Kept prompt states go to pinned host memory instead of what the streams leave of the card |
| 0021 | The engine's host tests set the FP8 patch's KV format, so its whole suite passes |
| 0023 | The fused read-out's down tile is chosen from the card's SM count, read from the device |
| 0024 | A ModelOpt checkpoint's routed experts a token and shared-expert width come from config.json; NVIDIA's keeps 10 and 640 |
| 0025 | --context past 262,144, up to 1,048,576, on the CUDA path: static YaRN x4 (YARN=1) |
Checked on this card
With all 25 patches, on NVIDIA's checkpoint: six greedy replies identical, token for token, to those on patches 0001-0020; 16 context-boundary checks; tool calls, images and video; follow-up turns at worst 99.6% resumed on a 60,000-token prompt; and a passphrase found in a 199,730-token prompt, with a prefill of 42.56 s.
Not checked yet
On NVIDIA's checkpoint with these patches: forced and named tool calls, a JSON schema and JSON mode, the byte-for-byte resume check and three full windows at once. They ran only on a checkpoint this recipe no longer serves. The speed settings have not been swept on this card either; they were carried over from a different GPU, and they change speed only, never a reply.
Quality
96.8% on sparkDash's bench.
sparkDash's quality bench on NVIDIA's FP4 with all 25 patches: 749 of 797 fixed questions right at temperature 0, 2 requests at a time, one run. The overall figure averages the 8 categories equally.
| Category | Questions | Score |
|---|---|---|
| QA | 150 | 91.3% |
| Reasoning | 40 | 100% |
| Arithmetic chain | 40 | 100% |
| State tracking | 40 | 100% |
| GSM8K | 200 | 97.5% |
| MMLU | 285 | 90.2% |
| Instruction following | 40 | 95.0% |
| Long-context recall | 2 | 100% |
| Overall | 797 | 96.8% |
What you need
One GPU, nothing else on it.
One RTX PRO 6000
One RTX PRO 6000 Blackwell (96 GB) in a Linux x86_64 host, with a driver for CUDA 13.
Free GPU memory
A start needs 96,845 MiB free on the card, which an empty card has and a card driving a desktop may not. On a machine with two cards, take the one without the monitors; on the other one use PARALLEL=2.
Docker
Docker with the NVIDIA container runtime, and your user in the docker group.
Disk and memory
About 160 GB of free disk, with the checkpoint on an NVMe drive. Developed on 89 GB of RAM; the prompt cache pins up to 16 GiB of it.
Quick start
One command, start to finish.
Clone and start
git clone https://github.com/MiaAI-Lab/Qwen3.8-Flash-Next-RTX-PRO-6000-TernsorFold.git
cd Qwen3.8-Flash-Next-RTX-PRO-6000-TernsorFold
DRY_RUN=1 ./start.sh # show what a start would do, and change nothing
./start.shThe first run builds the engine (about 5 minutes) and downloads the checkpoint (about 96 GiB). Later starts take about 2.5 minutes. It runs a smoke test and then prints Qwen3.8-Flash-Next is now LIVE! on port 8888.
Talk to it
Any OpenAI client works with base_url = "http://<host>:8888/v1". Streaming, tool calls, structured output and reasoning content are supported. The model thinks before it answers, so give replies enough max_tokens.
curl -s http://<host>:8888/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "Qwen3.8-Flash-Next",
"messages": [{"role": "user", "content": "Write a Python fibonacci function."}],
"max_tokens": 1000
}'./start.sh restart # stop, then start again
./stop.sh # stop
docker logs -f qwen38-flash-next-tf # logsImages and video
It can see, and watch.
Send them as OpenAI-style image_url and video_url content parts, as data URLs (VISION_URLS=1 also allows public https:// URLs). VISION=0 serves text only.
| Formats | Per request | |
|---|---|---|
| Images | JPEG, PNG, WebP | Up to 50, 10 MB each, 16,384 tokens in all |
| Video | MP4, WebM, MOV, MKV | Up to 4, 16 MB each, 16,384 tokens in all |
Memory
How the 96 GB is spent.
| On the card, measured (vision on, 3 requests) | GiB |
|---|---|
| Free on an empty card (of 95.6) | 95.04 |
| The model: weights, tower, scratch | 75.92 |
| Other engine allocations the budget counts | ~1.1 |
| Reserve | 3.0 |
| Left for streams | 15.0 |
| 3 streams at 262,144 tokens, 4.86 each | 14.58 |
| Unused | 0.42 |
A fourth full window does not fit. A request whose prompt plus max_tokens does not fit its window is refused, and one past PARALLEL waits for a stream.
Settings
The ones you might change.
Set these in the environment (PARALLEL=2 ./start.sh) or in .env next to start.sh; the environment wins. Every setting, patch, check and measurement is in the recipe's details.
| Setting | Default | Meaning |
|---|---|---|
GPU | 0 | The card, by its nvidia-smi index |
PORT | 8888 | Where the API listens |
PARALLEL | 3 | Requests decoded together. Four full windows do not fit the card |
CONTEXT | 262144 | Prompt + reply window. Above this, YaRN turns on |
MAX_TOKENS | 32768 | The reply budget when a request sets none |
THINKING | 1 | 0 answers without thinking |
DRAFTS | 1 | MTP drafts, which speed up decoding and never change a reply |
KV_DTYPE | fp8 | bf16 is the exact cache, but only one stream fits |
PROMPT_CACHE_GIB | 16 | Host memory for kept prompts, so a chat's next turn resumes instead of reading again |
The API
/v1/chat/completions, /v1/completions, /v1/responses, /v1/messages (Anthropic), /v1/models, /tokenize, /detokenize, /health and Prometheus /metrics, with the model Qwen3.8-Flash-Next.
Thinking
By default the model thinks first, and its reasoning comes back in reasoning_content. "chat_template_kwargs": {"enable_thinking": false} answers without thinking. A small max_tokens can return empty content, because the model was still thinking.
Licenses
Free, with a few rules.
- The recipe is MIT. That covers the repository's scripts only.
- The weights are downloaded from Hugging Face under their own terms: the NVIDIA Open Model Agreement and the Qwen Community License 1.0.
- The
ABLIT=1overlay has its own Responsible Use Agreement. - The image
scripts/prepare.shbuilds is based on NVIDIA's PyTorch container, under the NVIDIA Software License Agreement and the Product-Specific Terms for NVIDIA AI Products. By pulling or running the image you accept them.
Credits
Built on the work of others.
- TensorFold and its Zig engine by Ash Hart and the TensorFold contributors; the Flash Next CUDA engine this recipe builds on is TensorFold's own
- Jürgen Schmied (@jschmied): the Zig CUDA serving path and CUDA family registry (#443)
- @tempeste and @BadAd84: patches in the earlier series, in this repository's history
- @321sssrt-bit: the full-window admission fix and
tools/context_boundary.py - @MovieMaker93 (#40): the prompt-chunk change on the earlier Python engine; Javier (@jvr0x): the language draft vocabularies of the earlier Python recipe
- b12x by local-inference-lab: the RoCEnante protocol behind the engine's one-shot all-gather, which this recipe leaves off
- Qwen3.8 Flash Next by the Qwen team (Alibaba), and NVIDIA's FP4 QAD export (Model Optimizer)
- drowzeys: the
ABLIT=1overlay - xgrammar by the MLC-AI team, Zig by the Zig Software Foundation and its contributors, and FFmpeg for video
Patches 0001-0025 are mine, with where each idea comes from listed patch by patch in the recipe's CREDITS.md, along with the full list.
Run it on your RTX PRO 6000.
Free and open, with every patch published.