Recipe · 2× RTX PRO 6000 · Zig engine
GLM 5.3 Flash on two RTX PRO 6000s, with TensorFold's Zig engine.
Serve GLM-5.3-Flash from one Linux host with two 96 GB RTX PRO 6000 Blackwell GPUs, through an OpenAI-compatible API, with 4 streams at once and image input. It runs TensorFold's native Zig/CUDA server, one rank on each GPU in one container, with 93 engine patches on top and my own EXL3 quant. 245.5 tok/s for one request, with both cards capped at 400 W. One command checks the host, builds the image, downloads the weights, starts both ranks and runs a smoke test.
Performance
Measured, not promised.
The recipe's official numbers: two RTX PRO 6000 GPUs in one host, with both cards capped at 400 W.
Decode, prose
Temperature 0.7, thinking on: all streams together, per stream, and the time to first token.
| Streams | Total tok/s | Per stream | First token |
|---|---|---|---|
| 1 | 245.5 | 245.5 | 54 ms |
| 2 | 336.6 | 169.1 | 74 ms |
| 3 | 385.5 | 132.0 | 99 ms |
| 4 | 436.6 | 110.3 | 122 ms |
Prefill
How fast a prompt is read, by prompt size, and the time to first token.
| Prompt | Tokens counted | tok/s | First token |
|---|---|---|---|
| 4K | 4,129 | 5,943.1 | 753 ms |
| 8K | 8,220 | 5,753.0 | 1.49 s |
| 16K | 16,414 | 6,006.2 | 2.79 s |
| 32K | 32,802 | 6,033.1 | 5.49 s |
| 64K | 65,564 | 5,961.5 | 11.06 s |
| 128K | 131,103 | 5,787.7 | 22.71 s |
| 256K | 262,179 | 5,503.1 | 47.70 s |
The other recipe for this hardware, made with Marc Seal of Aevonix Research, runs TensorFold v0.6.5 with its own patch series. It publishes its numbers at 300 W, at temperature 0 with thinking off, so the two sets are not a like-for-like comparison.
Speed preview
What 245.5 tok/s feels like.
Sample text played at the measured decode speed, at about four characters a token: one request at 245.5 tok/s, or four at once at 110.3 tok/s each (436.6 combined). Switch between them in the window.
The weights
Downloaded for you, at a pinned revision.
The default
My 4 bpw EXL3 quant
Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold, about 164 GiB. The first start downloads it into DATA_DIR/model and mounts it read-only, and an interrupted download resumes on the next start. There is nothing to set.
The default drafter, DFlash2, is downloaded the same way when it is missing. It is non-commercial: for commercial use, set DRAFTER=mtp or DRAFTER=none.
Already have a checkpoint?
Serve a directory you already have as it is, with no download:
MODEL_SOURCE=local
MODEL_DIR=/path/to/GLM-5.3-Flash-EXL3-checkpointStock TensorFold reads only 4-bit experts; the recipe's engine patches and config overlay also serve mixed-width quants this way, and the checkpoint itself is never changed.
ABLIT=1
The Ablit weights
Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold-Ablit: the same checkpoint, abliterated, at a pinned revision. The repository is gated, so open its page on Hugging Face, log in and agree to its terms, create a read token with the same account, and set both in scripts/local.sh:
ABLIT=1
HF_TOKEN=hf_...Then ./start.sh restart. It downloads about 164 GiB into DATA_DIR/model-ablit, beside the published checkpoint, and fits its own window. Once downloaded, the weights need no token, and ABLIT=0 goes back to the published checkpoint, which stays in the cache.
Thinking is off by default in this recipe, which is also how the Ablit weights give their best results. They keep the stock DFlash2 drafter, which accepts a little less than on the published checkpoint.
The engine
Powered by TensorFold.
TensorFold, by Ash Hart, is an open-source engine that serves language models on Apple Silicon and NVIDIA GPUs through an OpenAI-compatible API. This recipe runs its Zig engine, tensorfold-native, built in Docker from commit f350cd3e with Zig 0.17.0 for the RTX PRO 6000's sm_120.
Two ranks, one container
One container runs both. Rank 0 serves the API and holds the vision tower; rank 1 follows its steps. If either fails, the other stops. Each rank holds half of the experts, attention and KDA heads, and vocabulary.
One host, over PCIe
NCCL runs over loopback, with peer-to-peer between the two cards and a copy-engine prompt exchange. Peer access is checked but not required: without it, NCCL falls back to slower paths.
One shared KV pool
With the FP8 cache on shared pages, every stream draws from one pool of 512-row pages. CONTEXT is then one request's cap, and a request that doesn't fit waits for pages.
In this recipe
93 patches applied in order at image build, most of them mine, a few carried from other authors with their names on them. The image tag is a hash of the Dockerfile, the patches and the pins, so editing scripts or docs never forces a rebuild.
On top of TensorFold
What the 93 patches do.
| Patches | What they do |
|---|---|
| 0001-0004 | EXL3 experts at mixed whole bit widths (for mixed-width quants), and 3-bit prompt kernels |
| 0005-0012 | Decode: FP8 attention loads, the shared expert on a side stream. Prefill: one expert call a pass, no MTP pass beside DFlash2, 2-stage int8 prompt GEMMs |
| 0013 | Upstream #1623: DFlash2 reads complete prompt taps at TP>1 |
| 0014-0016 | No target MTP layer beside DFlash2 (about 2 GiB a GPU back), and a pinned host-RAM prompt-state tier (8 GiB a rank by default) |
| 0017-0025 | FP8 KV on shared pages: paged E4M3 attention kernels, page planning, snapshots, admission |
| 0026-0033 | Decode: EXL3 load-ahead (upstream #1505 and #1534), fused MoE glue with new EXL3 decode tiles, KDA in one launch, one broadcast a follower step |
| 0034-0035 | A one-shot peer-to-peer all-gather for decode exchanges, and the copy-engine prompt exchange |
| 0036-0040 | DFlash2 copy rounds without the drafter pass, 16-row copy windows |
| 0042-0048 | Review fixes, "fits:" refusals that name what would fit, the device prompt-cache share into the pool |
| 0049-0063 | Paired prompt chunks (upstream #1423) extended to DFlash2, with a smaller twin buffer |
| 0067-0068 | Decode DSA select on many CTAs (long context); prompt sparse attention merged in the kernel (frees 353 MiB a GPU) |
| 0069-0071, 0077, 0083-0084 | Audited programmatic dependent launches along the decode chain; KDA step warps by row count |
| 0072-0075 | Decode glue: greedy picks in one launch, two-problem projection launches, fused router select-and-group, a shared window's KDA chains and latent attention in one launch each |
| 0078-0082 | Faster EXL3 prompt experts and packed int8 prompt GEMMs (prefill); grouped EXL3 decode reads each trellis word once, as a dependent-launch chain |
| 0086, 0088-0091 | Split hc_finish, batched shared-window cache writes, hc_partial8 split by outputs, decode DSA index scores for 4-16-row windows |
| 0092-0093 | A JSON answer stays JSON beside offered tools: response_format, guided_json and structured_outputs replies are never parsed as tool calls, on chat, Responses and Messages. A reply grammar is freed when its stream cannot start |
The rest are the opt-in settings below, plus test and contract-record changes. The full table is in the README.
Tested before they stayed
Every default-on patch was A/B tested against the image before it: greedy (T=0) replies identical, plus a measured gain. Patches 0067-0091 each keep an exact off switch for diagnosis, and before loading weights the two ranks compare a fingerprint of their settings, the checkpoint and the binary.
Off unless you ask
Two settings make prefill faster by changing rounding, so they stay off. ABSORB_MMA=1 only changes the fp32 summation order (+6.6% prefill), but replies are no longer byte-identical. W8A8=1 is a real rounding of activations (+10%, or +19% with both): validate it on your own eval before turning it on.
What you need
Two GPUs, one host.
Two RTX PRO 6000s
Two 96 GB RTX PRO 6000 Blackwell GPUs in one Linux x86_64 host, with a driver for CUDA 13.
Free GPU memory
At least 88,000 MiB free on each GPU, and under 4,096 MiB in use, which leaves room for a desktop on one of them. Rank 0 is the heavier one: keep it on the GPU without the desktop (GPUS=1,0 with the monitors on GPU 0).
Docker
Docker with the NVIDIA Container Toolkit and BuildKit, and your login in the docker group. Plus curl, python3, flock, tar, gzip, sha256sum and nvidia-smi, and hf (huggingface_hub[cli]) for the downloads.
Disk and memory
180 GiB free for the checkpoint (about 164 GiB), 10 GiB for DFlash2 and 40 GiB for the image build; the start checks for them. The ranks stage each layer through host memory while they load, and it warns below 16 GiB available.
Quick start
One command, start to finish.
Clone and start
git clone https://github.com/MiaAI-Lab/GLM-5.3-Flash-2x-RTX-PRO-6000-TensorFold.git
cd GLM-5.3-Flash-2x-RTX-PRO-6000-TensorFold
DRY_RUN=1 ./start.sh # print the whole plan: docker build, both rank command lines, docker run, smoke test
./start.sh # check, build the image once, download the weights once, start both ranks, wait for the API, smoke-testThe downloads need the hf command: python3 -m pip install -U "huggingface_hub[cli]". The first start builds the image, a few minutes with BuildKit's caches, and downloads the weights. Each rank also hashes every checkpoint shard once; that hash is cached, so later starts only read and load the weights. It checks that /v1/models lists the model, sends a smoke request and prints GLM-5.3-Flash-EXL3 is now LIVE! on port 8020.
To change a setting, copy scripts/local.sh.example to scripts/local.sh and edit it.
Talk to it
Any OpenAI client works with base_url = "http://<host>:8020/v1" and the model GLM-5.3-Flash. Thinking is off by default; a request turns it on with "chat_template_kwargs": {"enable_thinking": true}.
curl -s http://127.0.0.1:8020/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "GLM-5.3-Flash",
"messages": [{"role": "user", "content": "Hello"}]
}'./start.sh restart # apply changed settings
./stop.sh # stop, and archive the logs under logs/
docker logs -f glm53-zig # rank 0's log (rank 1's: logs/current/rank1.log)
curl -s http://127.0.0.1:8020/healthIf something stops it
The checks print a code. The ones you are most likely to see:
| Symptom | What to do |
|---|---|
E_DOWNLOAD | The download failed: check free space and your connection, and run ./start.sh again to resume. |
E_HF_TOKEN | ABLIT=1 and the gated weights can't be fetched. A 401 means Hugging Face doesn't accept the token, a 403 that its account hasn't agreed to the terms yet. Or set ABLIT=0. |
E_GPU_MEMORY / E_GPU_BUSY | Another workload holds a GPU: stop it. Raise MAX_GPU_USED_MIB only for a display GPU. |
E_PORT | Port 8020 or 29583-29584 is in use. Stop what holds it, or set PORT / NCCL_PORT. |
--parallel N needs ... GiB | The window or stream count doesn't fit. CONTEXT=auto retries by itself; otherwise lower CONTEXT or PARALLEL. |
| Out of memory mid-request | The desktop or prompt scratch grew into the reserve. Raise MEMORY_RESERVE_GIB, or lower PARALLEL / CONTEXT. |
W_P2P | NVML reports no peer-to-peer between the cards (IOMMU/ACS, BIOS). NCCL still works over host memory, but more slowly. |
| A slow first start | Each rank hashes the whole checkpoint once. WAIT_TIMEOUT (2400 s) covers it; raise it if needed. |
The full list is in the README.
Images
It can see, too.
Image input is on by default. Send an image_url part in a user message. The vision tower runs on rank 0 and takes about 1.8 GiB there; VISION=0 serves text only and gives that back.
To free GPU memory
- Lower
PARALLEL - Pack more:
DENSE=q6orq4, orHEAD=q8 VISION=0(rank 0 only)DRAFTER=mtp
Memory
The largest window that fits.
VRAM and system RAM are separate pools here: every weight must fit on the GPUs, and nothing is offloaded to host RAM. The engine's context is fixed at start and it refuses a window that doesn't fit, so CONTEXT=auto starts by asking for 1,048,576 tokens. If that is refused, start.sh works out the window the engine's own figures allow, starts once more, and keeps it in .state/context-fit.tsv for later starts.
So the window you get depends on your cards and your settings, and I don't quote one number for it. The startup line in rank0.log shows what was measured on yours, and start.sh warns if fewer than four streams came up. The default reserve is 4 GiB a GPU; fractions are allowed, and with the 4 bpw checkpoint 3.5 fits four streams on pages.
Settings
The ones you might change.
Settings are layered: the defaults in scripts/config.sh, then .env, then scripts/local.sh, then exported environment variables, which win over everything else. ./start.sh restart applies a change. The full list is in the README.
| Variable | Default | Meaning |
|---|---|---|
MODEL_SOURCE | download | download fetches the pinned 4 bpw checkpoint when it is missing; local serves MODEL_DIR as it is |
ABLIT / HF_TOKEN | 0 / unset | 1 serves the gated Ablit weights; the read token for that download is never passed to the ranks or printed |
GPUS | 0,1 | GPU indices in rank order: the first runs rank 0 |
PARALLEL | 4 | Concurrent streams. A number is admitted only if it fits; auto serves up to 8, as many as fit |
CONTEXT | auto | Up to 1,048,576 tokens of prompt plus reply; auto is the largest that fits |
KV / KV_PAGES | fp8 / on | The KV cache (bf16 is exact), and every stream drawing from one shared pool of pages |
DENSE / HEAD | q8 / bf16 | Dense linears packed at load, and the LM head: bf16, q8, q6 or q4 |
MEMORY_RESERVE_GIB | 4 | Memory per GPU left out of the engine's budget, which absorbs a desktop's growth |
DRAFTER | dflash2 | Inco AI's DFlash2 (non-commercial), mtp (the checkpoint's MTP head) or none |
THINKING / REASONING_EFFORT | 0 / high | Thinking off by default, and its effort when a request turns it on |
MAX_TOKENS | 32768 | The default reply limit |
VISION | 1 | Image input (rank 0's tower) |
PORT / HOST | 8020 / 0.0.0.0 | The API's port and address |
The API
/v1/chat/completions with the model GLM-5.3-Flash, /v1/models and /health. It listens on every address by default. Export TENSORFOLD_API_KEY and every route except /health needs Authorization: Bearer KEY.
Logs
logs/current points to the running session: rank0.log, rank1.log, runner.sh, models.json and smoke.json. ./stop.sh warns about any requests still open, gives rank 0 20 s to finish them, archives the logs as logs/glm53-zig-<time>.tar.gz and removes the container. It only ever stops this recipe's own.
Licenses
Free, with a few rules.
- The recipe is Apache 2.0. It holds scripts, a Dockerfile, a chat template and patches: no TensorFold, no weights, no binaries.
- The checkpoint's quantization is Apache 2.0, and the base model, GLM-5.3-Flash, stays under Z.ai's MIT. The Ablit weights are MIT and gated on Hugging Face, where their terms apply.
- The DFlash2 drafter is CC BY-NC-ND 4.0: research and evaluation only, commercial use needs a licence from Inco AI, and the recipe doesn't redistribute it.
DRAFTER=mtporDRAFTER=noneserves without it. - TensorFold is Apache 2.0 from 0.6.0; its earlier code keeps its MIT notice.
- NVIDIA's CUDA images stay under NVIDIA's own terms.
Credits
Built on the work of others.
- TensorFold and its native engine by Ash Hart and its contributors; the native EXL3 experts build on @justincasey's #837
- Aevonix Research: this recipe's
start.sh,stop.shand setup scripts are adapted from their four-GPU recipe (by Kurcide), and patch 0078's main loop follows the K12 loop in its patch series - Adam Hillson (@BadAd84): patches 0013 and 0028-0029 (#1623, #1534)
- @ajensenwaud: patches 0026-0027 (#1505)
- @Bizuayeu: patches 0049-0056 (#1423), which my patches 0057-0063 extend to DFlash2
- @SvangenStudios: patch 0092 adapts their #31 and patch 0093 part of their #28 on my DeepSeek recipe, and the stricter checks in
start.shandstop.shfollow their #24 - @tfolkman:
start.sh's log follower closing the lock's descriptor follows their #22 - GLM-5.3-Flash by Z.ai, and the DFlash2 drafter by Inco AI
- The EXL3 format (ExLlamaV3) by turboderp
The full list is in the recipe's CREDITS.md, and the required notices in its NOTICE.md.
Run it on your two GPUs.
Free and open, with every patch published.