Recipe 1104 · two or four DGX Sparks · Mia's favourite

GLM 5.3 Flash on DGX Sparks, with TensorFold.

Serve GLM-5.3-Flash from NVIDIA DGX Sparks through an OpenAI-compatible API: TensorFold's native Zig engine, one rank a Spark, over the ConnectX-7 link. On a pair of them: 47.8 tok/s for one request and 87.3 tok/s for four at once, the full 1,048,576-token window beside a prompt cache, and one command that checks both Sparks, builds the engine and starts it. Four Sparks run the same recipe at TP=4: 77.2 tok/s for one request, 172.7 for eight. A new engine line: 121 patches, every default measured.

Get the free recipe Mia on X

Install with an AI agent

Install this model

Paste this into an AI coding agent that can run commands on the machine that will serve the model (the head node, for a Spark cluster), such as Claude Code or Codex. It reads the recipe, adapts it to your setup, starts the server and checks that it works.


        
GLM-5.3-Flash on DGX Sparks with TensorFold, recipe 1104 — Now with TP4 support: a gold and blue wave on black

Recipe 1104 · 4 November 2026

A new engine line, not an update.

This recipe is brand new. It is not version 1.11 of my GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold — that Python-engine recipe and its Zig-engine branch stay as they are, and still serve three and four Sparks. 1104 descends from my two-RTX-PRO-6000 recipe instead: the same engine pin and 93 of its patches unchanged, plus a two-Spark launch over the ConnectX-7 link, unified-memory budgeting and its own patches — twelve at launch, and since 11 November a four-Spark mode, 121 in all. Different engine lineage, different patch stack, different numbers.

From the RTX recipe

The engine pin (f350cd3e) and patches 0001-0093 are the two-RTX-PRO-6000 recipe's, unchanged. They already carried the work of upstream patch authors, named in the credits below.

New for the pair

A launch that puts one rank on each Spark over the ConnectX-7 link, budgets memory the way a unified-memory box counts it, and twelve own patches (0094-0105): a larger prompt chunk with GB10's measured cost tables, and env-gated knobs the decode campaign measured.

Every default measured

The served configuration is the pair's: PARALLEL=8, the full 1,048,576-token window, FP8 KV on shared pages, q8 dense and head, ABSORB_MMA=1, DFlash2 at 4 bits, a 9.5 GiB reserve. The complete record, traps and retrials included, is the repo's docs/speed/RESULTS.md.

Performance

Measured, not promised.

One request47.8tok/s, prose, thinking on
Four at once87.3tok/s combined
Context1Mtokens a request
Prefill1,801tok/s, 32k prompt

The pair at the served configuration, measured with sparkDash on 4 November 2026. Decode is prose at temperature 0.7 with thinking on; prefill is a cold prompt of the given size.

Decode, official numbers

Prose at temperature 0.7, thinking on. Total across the concurrent requests, per request, and time to first token.

StreamsTotal tok/sPer streamTTFT
147.847.8200 ms
266.533.6324 ms
377.027.3437 ms
487.322.6557 ms

The server admits up to 8 streams (PARALLEL=8): free at 1-4, and the lab's harness below reads 94-95 tok/s at 6 and 105-106 at 8.

Prefill, official numbers

How fast a cold prompt is read before the reply starts.

Prompt tokenstok/sFirst token
4K1,6012.8 s
8K1,6885.1 s
16K1,7679.5 s
32K1,80118.4 s
64K1,81236.4 s
128K1,77874.0 s
256K1,708153.7 s

The lab's harness

Greedy, thinking off, fixed prompts, the client on a third Spark — the shape every engine change is gated on (docs/speed/harness/).

StreamsProse, tok/sSampled, T=0.7
138.4 · code 56.639.1-39.3
479-8175-78
694-95
8105-106

Same checkpoint, prompts and day, prose at T=0.7 (1 / 4 streams): this build 39.1 / 75-78, TensorFold's Python engine v0.6.0 29.6 / 68-71, the Zig v1.0.7 branch 21.3 / 30-31. Under greedy decoding that branch leads at 4 streams: 89-95 against 79-81.

Decode beside a long prefill

A long prompt is read in atomic 4,096-row chunks, about 2.5 s each; other streams decode between chunks.

PREFILL_SHARE=p makes each chunk owe the decoders part of its own time in decode rounds first. At the default 0.68, a chat beside a 50k-token prefill keeps 12.9 tok/s with a worst pause of 2.7 s (one chunk), and the prompt's first token comes about 47% later than at 1. Replies are byte-identical at any value.

Measure cold prefill with fresh prompt content: the prompt cache resumes a shared prefix and reports a rate that is not prefill (cached_tokens is 0 when nothing was resumed).

What the prefill tuning found

Tuned on 2 November 2026; the record is in the repo's RESULTS.md.

ChangeResult
ABSORB_MMA (on by default)+8.0% prefill with identical answers
The 4,096-row prompt chunk (on by default)+7% prefill, byte-identical replies
W8A8 (off)+10.5%, but rounds activations: one of ten answers wrong
q4 denseRejected on quality
The ConnectX-7 linkAbout 3% of a long prompt's time

At equal dense width the engine's prefill is at parity with the v1.0.7 build's.

Speed preview

What 47.8 tok/s feels like.

Sample text played at the measured decode speed, at about four characters a token: one request at 47.8 tok/s, or four at once at 22.6 tok/s each, 87.3 combined.

TensorFold Zig · GLM-5.3-Flash · 2× DGX Spark
47.8 tok/sSample text, played at the measured speed

The engine on a Spark

Built for this pair.

The engine is TensorFold 1.1.9 — its native Zig/CUDA server (tensorfold-native --version) at commit f350cd3e, compiled by the recipe's Dockerfile with Zig 0.17.0 for sm_121 — the engine's own default target for the GB10. The patches live in the repo's patches/engine/ and are applied at image build, in filename order.

RoCE across the cable

The engine's all-gather carries the decode and verify exchanges as one-shot RDMA writes over the Sparks' ConnectX-7 link: same bytes, same bits as NCCL, less time. Link discovery finds both PCIe twins of the cabled port and the RoCE v2 GID holding each side's address; a second rail is probed with a full-size ping before it is used.

Unified-memory budgeting

On a GB10, VRAM and system RAM are one pool. The engine budgets from /proc/meminfo plus the pages parked on the kernel's per-CPU lists — not from a card's free bytes — and the recipe reserves a floor of free memory on top (memory below).

Single-host tricks step aside

The two single-host optimisations in the patch stack, the peer-store all-gather and the copy-engine prompt exchange, step aside across hosts instead of failing. CE_EXCHANGE needs CUDA IPC between the ranks, which two Sparks do not have.

What is checked

  • All 121 patches apply to f350cd3e with no rejects; the image builds on aarch64 for sm_121 in about 6 minutes and reaches the worker by content.
  • NCCL and the engine's RDMA all-gather run over the cable.
  • The full 1,048,576-token window serves 8 streams beside a 3.6 GiB prompt cache.
  • Image input was checked by a user (issue 8: a PNG, then four frames in order); video input needs the WITH_VIDEO=1 image and was checked on a three-colour clip.
  • Greedy replies are byte-stable across every change.

Not checked (yet)

  • W8A8 is stashed on quality: it is +10.5% prefill but rounds activations.

Memory on a 128 GB unified Spark

One pool, spent in order.

The budget is MemAvailable plus parked pages, minus MEMORY_RESERVE_GIB. Per Spark on the served pair: model 83.3 GiB, KV pool 12.5 GiB (4,098 pages, about 2.1M tokens), prompt cache 3.6 GiB, and 9.5 GiB kept free. Any one request grows to the full 1,048,576-token window; requests draw from the one shared pool.

The page pool takes whole 1M windows first and the prompt cache gets the rest, so PROMPT_CACHE_GIB only caps it, and each stream past the first costs about 0.42 GiB. Cache hits land on a 4,096-token grid (up to 4,095 trailing tokens re-prefilled, about 2.5 s). The 9.5 GiB reserve holds the pool at two windows; after a start, rank 0's startup line must show 4098 pages, a prompt cache: N GiB, and no cannot keep one state warning.

HOST_CACHE_GIB=0 here: on GB10 a pinned host tier takes the pool's own memory. The memory guard (MEMGUARD) stops the pair in order when a Spark stays under 2.5 GiB available (logs/current/memguard.log). To free memory, lower PARALLEL, the DENSE/HEAD bits or VISION; on out-of-memory errors, raise MEMORY_RESERVE_GIB.

Long agent sessions

Three knobs for the big jobs.

CACHE_FIRST=1 PROMPT_CACHE_GIB=12 sets the cache aside before the streams' windows: the eight streams share one 1M window and rank 0 funds a 9.3-9.7 GiB cache. A 380k-token conversation resumed 99% cached (3.0 s instead of 234 s) beside eight ~31k ones. The trade: a second request past ~1M tokens of live context waits.

LEARN_GIB=30 keeps prompt states on disk too, so a conversation resumes after an eviction or a restart: 102.5k tokens came back in 1.8 s instead of 58.4 s cold, for about 1% decode cost while writing and nothing written under LEARN_MIN_FREE_GIB (15) free.

KV_SHARD=1 (TP=2 only) keeps each stream's latent rows on one Spark and doubles the pool to 8,196 pages, about 4.2M tokens, for 1-4% decode with identical greedy replies; pair it with CACHE_FIRST=1 PROMPT_CACHE_GIB=6. The four-Spark mode refuses it.

CONTEXT=auto

A window that fits, guaranteed.

The engine refuses a window that does not fit after loading, and names the bytes. CONTEXT=auto (the pair's setting) asks for CONTEXT_MAX — 1,048,576 — and, on a refusal with figures, restarts both ranks at the window those figures allow, times a safety margin, the tighter rank winning. It retries up to 3 times, and a window that served is kept in .state/context-fit.tsv for the same settings. CONTEXT_RETRY=0 turns the dance off.

A fixed CONTEXT works too: any token count up to 1,048,576, with the same shared pool behind it.

What you need

Two Sparks, one cable.

Four instead of two? The same requirements, three more Sparks and a ring or a switch: see Four Sparks.

Two DGX Sparks, cabled

GB10, 128 GB, CUDA 13 driver, cabled to each other on their ConnectX-7 ports with an IPv4 address on each end of one subnet. One cable is two netdevs and two RDMA devices — the port's two PCIe links — and both are used.

Docker, ssh, RDMA

Key-based ssh from this Spark to the worker; your login in docker on both; Docker with the NVIDIA Container Toolkit and BuildKit; rdma-core — without it every exchange stays on NCCL, more slowly.

The weights

The checkpoint Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold at the pinned revision, about 164 GiB, downloaded on the first start. Both ranks read every shard, so the worker gets it over the link (WORKER_WEIGHTS=rsync) or mounts yours by NFS.

Disk and small tools

About 40 GiB for the image on each Spark and 180 GiB for the checkpoint on each Spark that holds it. curl, python3, flock, tar, gzip, sha256sum, nvidia-smi, ssh, rsync, ip, awk, and hf for downloads.

The default drafter, incoai's DFlash2 at 4 bits, is CC BY-NC-ND 4.0 — non-commercial use only. For commercial use set DRAFTER=mtp (the checkpoint's own head) or none.

Quick start

One command, on the head.

1

Clone and point it at the worker

git clone https://github.com/MiaAI-Lab/GLM-5.3-Flash-DGX-Sparks-1104.git
cd GLM-5.3-Flash-DGX-Sparks-1104
cp scripts/local.sh.example scripts/local.sh   # set [email protected] in it

WORKER is the worker's ssh target; its address on the direct link is ideal — one hop, no LAN. The link, its RDMA devices and the RoCE v2 GID are found from the route to WORKER. The worker needs no copy of the repository. Settings layer: scripts/config.sh defaults, then .env, then scripts/local.sh, then the environment.

2

Look, then start

DRY_RUN=1 ./start.sh   # the whole plan: the link, the docker build, both ranks' command lines
./start.sh

The first run checks both Sparks, builds the image once (about 6 minutes) and sends it to the worker by content, places the weights, starts rank 1 on the worker and rank 0 here, each PID 1 in its own container, then waits for /health, /v1/models and a greedy smoke test. A rank that dies stops the other. Later starts are quicker; the very first also hashes all 164 GiB once per rank, cached.

3

Talk to it

Any OpenAI client works with base_url = "http://<head-address>:8020/v1" — the port is PORT in scripts/local.sh — and the model id GLM-5.3-Flash (SERVED_NAME). Thinking is off by default; a request can ask for it with reasoning_effort.

curl -s http://<head-address>:8020/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "GLM-5.3-Flash",
  "messages": [{"role": "user", "content": "Write a Python fibonacci function."}],
  "max_tokens": 2000
}'
./stop.sh   # rank 0 first (it drains for 20 s), then rank 1; logs archived to logs/glm53-spark-*.tar.gz

logs/current holds rank0.log and rank1.log, collected from each Spark's Docker output. stop.sh only ever stops containers labelled tf.recipe=glm53-spark. Every rank carries the recipe's TF_*/TENSORFOLD_*/NCCL_* settings; any other such variable you export reaches both ranks by name, TENSORFOLD_API_KEY included (never printed).

If something stops

SymptomWhat it means
[E_WORKER]ssh, Docker or the NVIDIA runtime is not ready on the worker; the message names which.
[E_FABRIC]No directly reached address, or no RoCE v2 GID holding an IPv4 on the link's port: cable the Sparks on one subnet, or set FABRIC_PEER / NCCL_IB_GID_INDEX.
[W_FABRIC] a rail left outThe second rail does not carry a full-size ping: check the cable, the twin's address and the MTU.
[E_MEMORY], [E_PORT]Less than MIN_AVAIL_GIB available (another rank, usually), or a port in use.
RankMismatch, RankNeverCame, NCCL timeoutsThe ranks were started differently, or rank 1 did not reach rank 0 within 300 s: check rank1.log and the ports; NCCL_DEBUG=INFO for detail.
vendor_err 0x81A rail without a path to its partner was set by hand through TF_ROCE_HCA.
A long prompt dies in a RoCE gatherROCE=off puts every exchange on NCCL.
--parallel N needs … GiB, [memguard]The window or stream count does not fit, or a Spark went under its floor: lower CONTEXT/PARALLEL or raise MEMORY_RESERVE_GIB.
No prompt cache in the startup lineThe page pool took a third window: raise MEMORY_RESERVE_GIB.
A reply ends with <|assistant|>, then repeats itselfThe checkpoint's EOS list leaves out <|assistant|>: keep EXTRA_EOS at its default.
Thinking-off content opens with a draft ("We need …") and holds </think>The model reasoned anyway: set THINK_OFF_SPLIT=1024 so the draft becomes reasoning_content.
HTTP 500 on a video_url partThe default image builds no video runtime: WITH_VIDEO=1 and restart.
[E_CACHE] a rank funds less cache than PROMPT_CACHE_MIN_GIBThat start's room varied: lower PARALLEL/CONTEXT or the minimum.
Raw <tool_call> text in an agent's replyThe turn was sent without the tool the model called: the tool-call guard, or fix the client.
git pull will not fast-forward an old cloneThe history was rewritten on 2026-11-04: take a fresh clone and carry scripts/local.sh and .env over.

The full table, and everything else, is in the recipe's README.

TP=4 · experimental

Four Sparks, one recipe.

Since 11 November the same image and settings also run as tensor parallel over four Sparks, measured on this kit. Two launchers, each checking the cabling and stopping — naming the script that fits — when it finds the other's. Rank 0 is the Spark that runs the script (the API); ./stop-tp4.sh stops the four ranks of either, and DRY_RUN=1 prints the links it finds and every rank's command, changing nothing.

A ring, no switch

./start-tp4-switchless.sh — the one run here. Spark A to B, B to C, C to D, D to A: each Spark's two ConnectX-7 ports to its two neighbours, one subnet per cable. The two Sparks across the ring reach each other through a neighbour or the LAN. WORKER, WORKER2 and WORKER3 are the other three in the ring's order from rank 0 — WORKER and WORKER3 cabled to it, WORKER2 across.

A switch

./start-tp4.sh — every Spark's ports share the switch's subnet, so the workers may be in any order; NCCL picks its own algorithms and devices and carries every exchange, and the ring's ROCE_RING relay is refused there. Written from the earlier Python-engine recipe's switch launcher, not yet run on a switch with this engine.

The weights: EXL3 or FP8

EXL3 4-bit (the pair's checkpoint, the default) is the faster and the one to start with, at MEMORY_RESERVE_GIB=12. FP8=1 serves Z.AI's own block-FP8 release instead (patch 0107): a little closer to the original (4-passage NLL 0.7425 against EXL3's 0.7487), 31-52% slower in decode, ~306 GiB more on rank 0, and it fits only at four Sparks — about 81 GiB of weights a rank, reserve 9.5. Switching is a restart with or without FP8=1; the image is the same.

Decode at four Sparks, official numbers

Prose at temperature 0.7, thinking on, top_k 20 — served-traffic numbers, not the pair's greedy ones. Measured 11 November 2026 on the ring, 121 patches, 8 streams at the full 1,048,576-token window, FP8 KV on pages, q8 dense and head, DFlash2.

StreamsEXL3, totala streamTTFTFP8, totala streamTTFT
177.277.2128 ms52.252.2204 ms
298.049.6246 ms69.535.7367 ms
3116.340.1309 ms88.630.6466 ms
4134.335.5382 ms88.623.6563 ms
8172.723.0686 ms119.015.81.08 s

EXL3 decodes 31-52% faster than FP8 at every load — it reads half the expert bytes a round.

Prefill at four Sparks

One cold prompt, tok/s: EXL3 4-bit, then Block FP8.

PromptEXL3FP8
4K1,582.01,534.9
8K1,630.01,806.6
16K1,793.71,833.1
32K1,879.91,843.4
64K1,918.71,849.3
128K1,908.21,839.2
256K1,902.61,802.0

Prefill is close: FP8 is about 10% faster at 8K, and from 16K up the two are within 2-6% (EXL3 ahead from 32K), both holding 1.8-1.9K tok/s through 256K.

From the pair to four Sparks and back

  1. Cable the four Sparks — a ring (A-B, B-C, C-D, D-A, each cable its own subnet) for ./start-tp4-switchless.sh, or every Spark's ports to one switch on one subnet for ./start-tp4.sh.
  2. Pick rank 0 and name the workers: on a ring, WORKER and WORKER3 are the two cabled to it and WORKER2 the one across; on a switch, any order. From rank 0, ssh -o BatchMode=yes <worker> true must succeed for all three.
  3. Stop everything else on the four Sparks — each serves one rank and needs its GPU and memory free. Stop any pair first (./stop.sh on its rank 0); the start checks every Spark's memory, disk and ports and stops with a reason when one falls short.
  4. The weights: rank 0 holds the checkpoint and DFlash2 at their pinned revisions; the workers mount them over NFS (WORKER_WEIGHTS=nfs, the default here — rank 0 exports the folders read-only; NFS_MODEL_PATH/NFS_DFLASH2_PATH when the export is another view of the path) or hold their own copy (check).
  5. Write rank 0's scripts/local.sh: the three workers plus the weights, everything else the pair's settings — WORKER=user@spark2, WORKER2=user@spark3, WORKER3=user@spark4, WORKER_WEIGHTS=nfs, PORT=8888, PARALLEL=8, CONTEXT=auto, MEMORY_RESERVE_GIB=12 (9.5 with FP8). A two-Spark local.sh starts nothing here: the start stops and names what is missing.
  6. Look, then start: DRY_RUN=1 ./start-tp4-switchless.sh prints the links and every rank's command, and stops with E_FABRIC when the cabling is the other launcher's or the workers are out of ring order, with E_CONFIG for a setting this mode does not take. Without DRY_RUN it builds the image when the patches changed (about 6 minutes), sends it to the three workers, starts ranks 3, 2 and 1, then rank 0, and waits for a greedy smoke reply. Logs: logs/current/rank0..3.log.
  7. Back to the pair: ./stop-tp4.sh stops all four ranks (rank 0 first) and archives their logs; restore the pair's local.sh and ./start.sh on its rank 0.

What differs at TP=4: NCCL runs as a ring over the cables (the launcher sets NCCL_ALGO=Ring, the graph files and subnet-aware routing itself; do not export NCCL_SOCKET_IFNAME yourself — SOCKET_IFNAME is the launcher's setting for the bootstrap netdev), ROCE is off with the opt-in ROCE_RING=1 relay instead (patch 0108), and each rank holds a quarter of the routed experts while the KV pool is replicated — eight 1M windows and a prompt cache usually fit at the default reserve. KV_SHARD is refused (it shards over exactly two ranks) and PROMPT_CACHE_MIN_GIB is not checked here. The full tables, and everything else, are in the recipe's README.

Settings

The ones you might change.

Every setting lives in scripts/config.sh, with its default. Set one for a single run from the environment, or keep it in scripts/local.sh or a .env file. The served pair's own local.sh, for the record: PORT=8888, PARALLEL=8, CONTEXT=auto, KV=fp8, DENSE=q8, HEAD=q8, ABSORB_MMA=1, HOST_CACHE_GIB=0, MEMORY_RESERVE_GIB=9.5, MODEL_SOURCE=local, WORKER_WEIGHTS=check.

VariableDefaultMeaning
WORKERrequiredThe second Spark's ssh target on the direct link, e.g. [email protected]
FABRIC_PEER, FABRIC_SELF, TF_ROCE_HCA, NCCL_IB_GID_INDEXautoFound from the route to WORKER, the port's RDMA devices and its RoCE v2 GID table
NCCL_CHANNELS4NCCL_MIN_NCHANNELS and NCCL_MAX_NCHANNELS
TP2Two Sparks with ./start.sh; four use the 4 Sparks launchers
PORT / HOST / NCCL_PORT8020 / 0.0.0.0 / 29583The API address, and rank 0's bootstrap port on the link
ROCE / ROCE_MAX_KBon / 4096The engine's one-shot RDMA all-gathers, and the largest shard they carry; off puts everything on NCCL
WORKER_WEIGHTSrsyncrsync the checkpoint over the link, nfs (the worker mounts it; NFS_MODEL_PATH/NFS_DFLASH2_PATH when the export is another view of the path), or check (refuse unless present)
WORKER_MODEL_DIR, WORKER_DFLASH2_DIR, WORKER_CACHE_DIRthe same pathsWhere the worker keeps them; the home-based defaults re-root on the worker's own home when the two logins differ
MODEL_SOURCE / MODEL_DIRdownloadFetch the pinned checkpoint when missing, or local to serve MODEL_DIR as it is
SSH_OPTS-o BatchMode=yes -o ConnectTimeout=10For every call to the worker
PARALLEL4Requests decoded together; auto serves up to 8. The pair runs 8: free at 1-4 streams, 94-95 tok/s at 6, 105-106 at 8
CONTEXT / CONTEXT_MAXauto / 1048576The window dance, or a fixed token count
KV / KV_PAGESfp8 / onOne shared pool of 512-row pages, CONTEXT as one request's cap
DENSE / HEADq8 / bf16bf16, q8, q6 or q4 for the dense linears and the LM head; the pair runs both at q8
MEMORY_RESERVE_GIB / HOST_CACHE_GIB / PROMPT_CACHE_GIB12 / 0 / emptyThe free-memory floor, the pinned host tier (keep it at 0 on a Spark), and the prompt cache's cap (above); the pair runs the reserve at 9.5
LEARN_GIB / LEARN_MIN_FREE_GIB0 / 15Keep prompt states on disk too, up to that many GiB (above)
KV_SHARD01 keeps each stream's latent rows on one Spark: about twice the tokens for 1-4% decode; TP=2 only (above)
CACHE_FIRST / PROMPT_CACHE_MIN_GIB0 / empty1 sets the prompt cache aside before the streams' windows; the minimum is checked before LIVE (E_CACHE), the four-Spark start never checks it
L2PF01 prefetches each decode window's next weights into L2 on a side stream (patch 0117, from upstream): +2% at one stream here, flat at four and eight, so it stays opt-in
WITH_VIDEO01 builds libtfvideo.so (FFmpeg, LGPL) into its own image tag, so VISION=1 serves video_url parts; a few minutes more at build
PREFILL_SHARE0.68How much of each prompt chunk's time is owed to the decoders first; above
DRAFTER / DRAFTER_BITSdflash2 / 4dflash2 (non-commercial license), mtp (the checkpoint's own head) or none
ABLIT01: serve the gated Ablit weights; needs HF_TOKEN
THINKING / REASONING_EFFORT / MAX_TOKENS0 / xhigh / 32768Request defaults (xhigh renders the checkpoint's trained "Reasoning Effort: Max" line); earlier turns' reasoning is dropped from the prompt
EFFORT_LINEthinkingThe template's "Reasoning Effort: Max" line with thinking on only; always serves the checkpoint's template as it is — below
EXTRA_EOS<|assistant|>Tokens that end a reply besides the checkpoint's EOS ids (patch 0120); none serves the checkpoint's list only
THINK_OFF_SPLIT0With thinking off, a reply that writes </think> within its first this-many tokens is split into reasoning_content and content (patch 0121)
ABSORB_MMA / W8A81 / 0Prompt numerics: ABSORB_MMA is free (+8.0%, identical answers); W8A8 rounds activations and stays off
VISION / VISION_MAX_IMAGES / CHAT_TEMPLATE1 / 50 / autoImage input on rank 0's tower, checked by a user (issue 8); video_url parts too with WITH_VIDEO=1; 0 serves text only
MEMGUARD / MEMGUARD_FLOOR_GIB1 / 2.5The memory guard, and the available-memory floor it keeps
SM / ZIG_CPU121 / nativeGB10's target; baseline builds an image that runs elsewhere too
SERVED_NAME / SERVE_EXTRA_ARGS / TENSORFOLD_API_KEYGLM-5.3-Flash / empty / unsetThe model id clients send; more serve flags; a bearer key for every route but /health. Requests that send no sampling fields get the checkpoint's generation_config.json (temperature 1.0, top_p 0.95); the pair sets SERVE_EXTRA_ARGS='--top-k 20'
DRY_RUN01 prints the whole plan, and changes nothing

The effort line

The checkpoint's chat template opens every prompt with a "Reasoning Effort: Max" system line — thinking off included, so a thinking-off reply then reasons in its answer. EFFORT_LINE=thinking (the default) has start.sh serve a template that writes the line only when thinking is on; EFFORT_LINE=always serves the checkpoint's template as it is, and a CHAT_TEMPLATE of your own is always served as it is. Measured with sparkDash's quality suite (greedy, 755 items): with the line in every prompt, QA 79.3% against TensorFold v0.6.0's 84.0% and MMLU 91.2%; with EFFORT_LINE=thinking, QA 86.0% (140 of 150 replies byte-identical to v0.6.0's) and MMLU 88.4%, thinking-on categories unchanged.

Hermes and other agents

When a tool call arrives as text.

The server turns the <tool_call> markup GLM writes into OpenAI tool_calls only for tools the request offers, as vLLM does. An agent sometimes sends a turn without the tool the model then calls — no tools, tool_choice: "none", or a list that lacks it; typically the final-summary turn after Hermes Agent reaches its iteration limit — and that call reaches the agent as raw <tool_call>…</tool_call> text. Fix the agent side first: raise its iteration limit, and keep the toolset that holds the tool enabled for the agent you run.

The guard (patch 0111, off by default)

TF_TOOL_CALL_GUARD=1 ./start.sh restart

To keep it on for every start, export TF_TOOL_CALL_GUARD=1 in scripts/local.sh — it must be exported to reach the ranks. The first start after pulling this version builds the new image (the patch set changed); check it took effect with docker exec glm53-spark env | grep TF_TOOL_CALL_GUARD.

On: a request that offers no tools but whose history called them gets a closing system note that tools are unavailable, and any <tool_call> the model writes anyway is removed — the reply is then prose, or empty if it only wanted to call. A request that offers tools has well-formed calls to tools it did not offer removed; calls to offered tools come back as tool_calls as before. Malformed calls stay as text, and a reply is never emptied by the guard (patch 0116). Streamed or whole.

What it logs, and when to leave it off

Every call the server does not turn into tool_calls gets one log line — the request id, the reason (unknown_tool_name, malformed or unclosed), the tool name, the request's thinking mode, and whether it was kept or hidden: docker logs glm53-spark 2>&1 | grep "tool call rejected". Each hide also logs a tool-call guard: line. TF_TOOL_CALL_DEBUG=1 in scripts/local.sh adds the envelope's text — it is the model's output, so leave it off where logs are shared.

Off, every reply is byte-identical to a build without the patch; on, replies to ordinary requests are unchanged too — same greedy hashes, same speed. Leave it off for a client that reads tool calls out of the reply text itself (tools described in its own system prompt, no tools field): the guard would remove the calls that client is waiting for.

Ablit weights · ABLIT=1

The same model, abliterated.

ABLIT=1 serves Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold-Ablit, the same checkpoint abliterated, at the revision pinned in scripts/config.sh — everything else (drafts, the window, the memory budget) works the same. Thinking off, the recipe's default, is also how these weights give their best results.

Turn it on

The weights are gated on Hugging Face, so the server needs your token.

  1. Open the repo's page on Hugging Face, log in and agree to its terms.
  2. With the same account, create a token with read access at huggingface.co/settings/tokens.
  3. Set both in scripts/local.sh (or .env), then restart.
ABLIT=1
HF_TOKEN=hf_...

Good to know

  • Checked before anything happens: start.sh verifies that HF_TOKEN reaches the repo before building or downloading. A 401 means Hugging Face doesn't accept the token; a 403, that its account hasn't agreed to the terms yet.
  • The first Ablit start downloads it, beside the published checkpoint, and the worker gets it like any other set.
  • ABLIT=0 goes back to the published checkpoint, which stays in the cache.
  • Exact ids and hashes: asked to return one in schema-constrained JSON, the Ablit weights can drop a digit (issue 9) — the published weights copy it exactly, and the Ablit weights do too with thinking on. For replies that must copy ids exactly, serve ABLIT=0 or turn thinking on for those requests.
  • Drafting: DFlash2 accepts fewer drafts on these weights — measure them at your own memory settings before choosing a drafter.
  • A partial overlay: scripts/graft.py builds a tree with self_attn.o_proj from only the Ablit layers you name (--layers 15-29 --verify), hardlinking the rest, rebuilding the 164 GiB tree in about 30 s.

Access needs you to agree to a Responsible Use Agreement on Hugging Face: you must be 18 or older, and you take responsibility for adding your own safety filtering, review and access controls. The weights carry the upstream GLM-5.3-Flash license.

The patch stack

121 patches, each earned.

patches/engine/ holds 121 patches on TensorFold f350cd3e, applied in filename order at image build. 0001-0093 are my two-RTX-PRO-6000 recipe's, unchanged; 0094-0106 and 0111-0121 are this pair's own or ported from that recipe or from upstream; 0107-0110 are the four-Spark patches. None is GB10-specific: the arch-gated kernels test for compute capability 1200 and up, which sm_121 satisfies. Every default-on patch was A/B tested against the image before it: greedy replies identical, plus a measured gain.

PatchesWhat they do
0001-0012EXL3 experts at mixed bit widths, 3-bit prompt kernels; FP8 attention loads, the shared expert on a side stream; one expert call a prompt pass, no MTP pass beside DFlash2
0013-0016DFlash2 reads complete prompt taps at TP>1 (upstream #1623); no target MTP layer beside DFlash2; a pinned host prompt-state tier (off here)
0017-0033FP8 KV on shared pages; EXL3 load-ahead, fused MoE glue, KDA in one launch, one broadcast a follower step
0034-0035Peer-store all-gather and copy-engine prompt exchange, both stepping aside across hosts
0036-0048DFlash2 copy rounds, ABSORB_MMA (opt-in), "fits:" refusals, the device prompt-cache share
0049-0066Paired prompt chunks (upstream #1423) extended to DFlash2; W8A8 (opt-in)
0067-0091DSA select on many CTAs, audited programmatic dependent launches along the decode chain, glue fusions, EXL3 prompt and decode kernels; each keeps an off switch. Never ship 0086 or 0089 without 0090
0092-0093A JSON answer stays JSON beside offered tools; a reply grammar is freed when its stream cannot start
0094-0097This pair's. A 4,096-row prompt chunk with DFlash2's MAX_CTX to match (+7% prefill, byte-identical replies); the GB10's measured cost tables and draft-rule seed
0098-0102This pair's, env-gated, defaults behaviour-identical: the decode campaign's exact knobs, each measured neutral
0103-0104From the RTX recipe: a tool call after prose that quotes a tool-call opener is the call; warm turns, so a user message after an agent loop resumes past the loop
0105An untyped tool parameter takes the JSON object or array the model wrote; text stays text
0106Four Sparks: the start's peer warm-up follows the ring's all-gather mode instead of posting to every rank
0107-0110Four Sparks. Block FP8 checkpoints (Z.AI's own release, FP8=1); the four-rank RoCE ring all-gather (opt-in); the DFlash2 drafter MLP split across every rank; TF_GLM_CHUNK_SOLO (opt-in)
0111-0112The tool-call guard (TF_TOOL_CALL_GUARD=1, opt-in): tool-call markup never reaches the client as text, streamed or whole, each hide logged; CACHE_FIRST=1 sets the prompt cache aside before the streams' windows
0113-0114LEARN_GIB (--learn-conversations, a backport of upstream #1411): kept prompt states on disk, so a conversation resumes after an eviction or a restart; KV sharding (KV_SHARD=1): each stream's latent rows on one Spark, twice the pool, byte-identical replies
0115-0116This pair's. A clean image build prints no compiler warning (all six files' fatbins byte-identical); each tool envelope the parser rejects is logged on its own line, TF_TOOL_CALL_DEBUG=1 adding the text
0117-0119From upstream: L2PF=1 prefetches a decode window's next weights into L2 (opt-in; +2% at one stream on the pair, flat at four and eight); a reply cut by max_tokens inside a JSON tool call returns the call as far as written; decode windows of 32 rows, so a 1,195-token rewrite decoded at 201-205 tok/s where 151 before
0120-0121EXTRA_EOS (on by default with <|assistant|>): tokens that end a reply besides the checkpoint's EOS ids; THINK_OFF_SPLIT (opt-in) splits a thinking-off reply that writes </think> into reasoning_content and content

The engine

Powered by TensorFold.

TensorFold, by Ash Hart, is an open-source engine that serves language models on Apple Silicon and NVIDIA GPUs through an OpenAI-compatible API. Each model family brings its own kernels and its own draft verification, tuned for that model instead of one generic path for all.

Exact decoding

Drafts make it fast without changing a word: a draft is accepted only when it equals the token the same engine would produce one at a time.

Apple Silicon and NVIDIA

MLX on Macs, and CUDA on NVIDIA GPUs from compute capability 8.9: Ada (RTX 40), Hopper and Blackwell, including the DGX Spark's GB10 and the RTX 50 series.

Many model families

GLM-5.3-Flash, Qwen3.8-27B, Qwen3.8 Flash Next, Nemotron 3.5, Gemma 4 and DeepSeek-V4-Flash, among others, each with its own kernels.

In this recipe

The native Zig server at pin f350cd3e, one rank per Spark over RoCE, with 121 patches: FP8 KV on shared pages, DFlash2 drafts, the 4,096-row prompt chunk, and the pair's own memory accounting.

Licenses

Free, with a few rules.

  • The recipe is Apache License 2.0. Its patches modify TensorFold at f350cd3e, whose code keeps TensorFold's licenses.
  • The checkpoint, Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold, is by Mia's AI Lab, from zai-org's GLM-5.3-Flash (MIT).
  • The DFlash2 drafter is CC BY-NC-ND 4.0, non-commercial use only. DRAFTER=mtp or none serves without it.
  • The Ablit weights are gated behind a Responsible Use Agreement and carry the upstream GLM-5.3-Flash license.
  • Downloaded software and weights keep their own terms; required notices are in the repo's NOTICE.md.

Credits

Built on the work of others.

The full list is in the recipe's CREDITS.md.

Run it on your Sparks.

Free and open, with every patch and every number published.