MiaAI LAB Contact

Recipe 1104 · two DGX Sparks · Mia's favourite

GLM 5.3 Flash on two DGX Sparks, with TensorFold.

Serve GLM-5.3-Flash from two NVIDIA DGX Sparks through an OpenAI-compatible API: TensorFold's native Zig engine, one rank a Spark, over the ConnectX-7 link. 47.8 tok/s for one request and 87.3 tok/s for four at once, the full 1,048,576-token window beside a prompt cache, and one command that checks both Sparks, builds the engine and starts it. A new engine line: 105 patches, every default measured on the pair.

Get the free recipe Mia on X

Install with an AI agent

Install this model

Paste this into an AI coding agent that can run commands on the machine that will serve the model (the head node, for a Spark cluster), such as Claude Code or Codex. It reads the recipe, adapts it to your setup, starts the server and checks that it works.


        

          
          
        

      

    

    

    

      GLM-5.3-Flash on two DGX Sparks, recipe 1104: a gold and blue wave on black
    

  







  

    
Recipe 1104 · 4 November 2026

    
A new engine line, not an update.

    
This recipe is brand new. It is not version 1.11 of my GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold — that Python-engine recipe and its Zig-engine branch stay as they are, and still serve three and four Sparks. 1104 descends from my two-RTX-PRO-6000 recipe instead: the same engine pin and 93 of its patches unchanged, plus a two-Spark launch over the ConnectX-7 link, unified-memory budgeting and twelve patches of its own for this pair. Different engine lineage, different patch stack, different numbers.

    

      
From the RTX recipe
The engine pin (f350cd3e) and patches 0001-0093 are the two-RTX-PRO-6000 recipe's, unchanged. They already carried the work of upstream patch authors, named in the credits below.

      
New for the pair
A launch that puts one rank on each Spark over the ConnectX-7 link, budgets memory the way a unified-memory box counts it, and twelve own patches (0094-0105): a larger prompt chunk with GB10's measured cost tables, and env-gated knobs the decode campaign measured.

      
Every default measured
The served configuration is the pair's: PARALLEL=8, the full 1,048,576-token window, FP8 KV on shared pages, q8 dense and head, ABSORB_MMA=1, DFlash2 at 4 bits, a 9.5 GiB reserve. The complete record, traps and retrials included, is the repo's docs/speed/RESULTS.md.

    

  







  

    
Performance

    
Measured, not promised.

    

      
One request47.8tok/s, prose, thinking on

      
Four at once87.3tok/s combined

      
Context1Mtokens a request

      
Prefill1,801tok/s, 32k prompt

    

    
The pair at the served configuration, measured with sparkDash on 4 November 2026. Decode is prose at temperature 0.7 with thinking on; prefill is a cold prompt of the given size.


    

      

        
Decode, official numbers

        
Prose at temperature 0.7, thinking on. Total across the concurrent requests, per request, and time to first token.

        

          
StreamsTotal tok/sPer streamTTFT

          
            
147.847.8200 ms

            
266.533.6324 ms

            
377.027.3437 ms

            
487.322.6557 ms

          
        

        
The server admits up to 8 streams (PARALLEL=8): free at 1-4, and the lab's harness below reads 94-95 tok/s at 6 and 105-106 at 8.

      

      

        
Prefill, official numbers

        
How fast a cold prompt is read before the reply starts.

        

          
Prompt tokenstok/sFirst token

          
            
4K1,6012.8 s

            
8K1,6885.1 s

            
16K1,7679.5 s

            
32K1,80118.4 s

            
64K1,81236.4 s

            
128K1,77874.0 s

            
256K1,708153.7 s

          
        

      

      

        
The lab's harness

        
Greedy, thinking off, fixed prompts, the client on a third Spark — the shape every engine change is gated on (docs/speed/harness/).

        

          
StreamsProse, tok/sSampled, T=0.7

          
            
138.4 · code 56.639.1-39.3

            
479-8175-78

            
694-95

            
8105-106

          
        

        
Same checkpoint, prompts and day, prose at T=0.7 (1 / 4 streams): this build 39.1 / 75-78, TensorFold's Python engine v0.6.0 29.6 / 68-71, the Zig v1.0.7 branch 21.3 / 30-31. Under greedy decoding that branch leads at 4 streams: 89-95 against 79-81.

      

      

        
Decode beside a long prefill

        
A long prompt is read in atomic 4,096-row chunks, about 2.5 s each; other streams decode between chunks.

        
PREFILL_SHARE=p makes each chunk owe the decoders part of its own time in decode rounds first. At the default 0.68, a chat beside a 50k-token prefill keeps 12.9 tok/s with a worst pause of 2.7 s (one chunk), and the prompt's first token comes about 47% later than at 1. Replies are byte-identical at any value.

        
Measure cold prefill with fresh prompt content: the prompt cache resumes a shared prefix and reports a rate that is not prefill (cached_tokens is 0 when nothing was resumed).

      

      

        
What the prefill tuning found

        
Tuned on 2 November 2026; the record is in the repo's RESULTS.md.

        

          
ChangeResult

          
            
ABSORB_MMA (on by default)+8.0% prefill with identical answers

            
The 4,096-row prompt chunk (on by default)+7% prefill, byte-identical replies

            
W8A8 (off)+10.5%, but rounds activations: one of ten answers wrong

            
q4 denseRejected on quality

            
The ConnectX-7 linkAbout 3% of a long prompt's time

          
        

        
At equal dense width the engine's prefill is at parity with the v1.0.7 build's.

      

    

  







  

    

      
Speed preview

      
What 47.8 tok/s feels like.

      
Sample text played at the measured decode speed, at about four characters a token: one request at 47.8 tok/s, or four at once at 22.6 tok/s each, 87.3 combined.

    

    

      

        
        TensorFold Zig · GLM-5.3-Flash · 2× DGX Spark
        
          
          
        
      

      
      
47.8 tok/sSample text, played at the measured speed

    

  







  

    
The engine on a Spark

    
Built for this pair.

    
The engine is TensorFold's native Zig/CUDA server at commit f350cd3e, compiled by the recipe's Dockerfile with Zig 0.17.0 for sm_121 — the engine's own default target for the GB10. The patches live in the repo's patches/engine/ and are applied at image build, in filename order.

    

      
RoCE across the cable
The engine's all-gather carries the decode and verify exchanges as one-shot RDMA writes over the Sparks' ConnectX-7 link: same bytes, same bits as NCCL, less time. Link discovery finds both PCIe twins of the cabled port and the RoCE v2 GID holding each side's address; a second rail is probed with a full-size ping before it is used.

      
Unified-memory budgeting
On a GB10, VRAM and system RAM are one pool. The engine budgets from /proc/meminfo plus the pages parked on the kernel's per-CPU lists — not from a card's free bytes — and the recipe reserves a floor of free memory on top (memory below).

      
Single-host tricks step aside
The two single-host optimisations in the patch stack, the peer-store all-gather and the copy-engine prompt exchange, step aside across hosts instead of failing. CE_EXCHANGE needs CUDA IPC between the ranks, which two Sparks do not have.

    

    

      

        
What is checked

        
    
          
  • All 105 patches apply to f350cd3e with no rejects; the image builds on aarch64 for sm_121 in about 6 minutes and reaches the worker by content.
    • 
          
  • NCCL and the engine's RDMA all-gather run over the cable.
    • 
          
  • The full 1,048,576-token window serves 8 streams beside a 3.6 GiB prompt cache.
    • 
          
  • Greedy replies are byte-stable across every change.
    • 
        

      

      

        
Not checked (yet)

        
    
          
  • Image input. The vision tower ships on and its prompt template is there, but image input is not verified in this build; the settings below can turn it off.
    • 
          
  • W8A8 is stashed on quality: it is +10.5% prefill but rounds activations.
    • 
        

      

    

  







  

    

      
Memory on a 128 GB unified Spark

      
One pool, spent in order.

      
The budget is MemAvailable plus parked pages, minus MEMORY_RESERVE_GIB. Per Spark on the served pair: model 83.3 GiB, KV pool 12.5 GiB (4,098 pages, about 2.1M tokens), prompt cache 3.6 GiB, and 9.5 GiB kept free. Any one request grows to the full 1,048,576-token window; requests draw from the one shared pool.

      
The page pool takes whole 1M windows before the prompt cache, so a larger room can silently leave the cache at zero and every chat turn re-reads its history. The 9.5 GiB reserve holds the pool at two windows. After a start, rank 0's startup line must show 4098 pages, a prompt cache: N GiB, and no cannot keep one state warning.

      
HOST_CACHE_GIB=0 here: on GB10 a pinned host tier takes the pool's own memory. The memory guard (MEMGUARD) stops the pair in order when a Spark stays under 2.5 GiB available. To free memory, lower PARALLEL, the DENSE/HEAD bits or VISION; on out-of-memory errors, raise MEMORY_RESERVE_GIB.

    

    

      
CONTEXT=auto

      
A window that fits, guaranteed.

      
The engine refuses a window that does not fit after loading, and names the bytes. CONTEXT=auto (the pair's setting) asks for CONTEXT_MAX — 1,048,576 — and, on a refusal with figures, restarts both ranks at the window those figures allow, times a safety margin, the tighter rank winning. It retries up to 3 times, and a window that served is kept in .state/context-fit.tsv for the same settings. CONTEXT_RETRY=0 turns the dance off.

      
A fixed CONTEXT works too: any token count up to 1,048,576, with the same shared pool behind it.

    

  







  

    
What you need

    
Two Sparks, one cable.

    

      
Two DGX Sparks, cabled
GB10, 128 GB, CUDA 13 driver, cabled to each other on their ConnectX-7 ports with an IPv4 address on each end of one subnet. One cable is two netdevs and two RDMA devices — the port's two PCIe links — and both are used.

      
Docker, ssh, RDMA
Key-based ssh from this Spark to the worker; your login in docker on both; Docker with the NVIDIA Container Toolkit and BuildKit; rdma-core — without it every exchange stays on NCCL, more slowly.

      
The weights
The checkpoint Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold at the pinned revision, about 164 GiB, downloaded on the first start. Both ranks read every shard, so the worker gets it over the link (WORKER_WEIGHTS=rsync) or mounts yours by NFS.

      
Disk and small tools
About 40 GiB for the image on each Spark and 180 GiB for the checkpoint on each Spark that holds it. curl, python3, flock, tar, gzip, sha256sum, nvidia-smi, ssh, rsync, ip, awk, and hf for downloads.

    

    
The default drafter, incoai's DFlash2 at 4 bits, is CC BY-NC-ND 4.0 — non-commercial use only. For commercial use set DRAFTER=mtp (the checkpoint's own head) or none.

  







  

    
Quick start

    
One command, on the head.

    

      

        1
        

          
Clone and point it at the worker

          
git clone https://github.com/MiaAI-Lab/GLM-5.3-Flash-DGX-Sparks-1104.git
cd GLM-5.3-Flash-DGX-Sparks-1104
cp scripts/local.sh.example scripts/local.sh   # set [email protected] in it

          
WORKER is the worker's ssh target; its address on the direct link is ideal — one hop, no LAN. The link, its RDMA devices and the RoCE v2 GID are found from the route to WORKER. The worker needs no copy of the repository. Settings layer: scripts/config.sh defaults, then .env, then scripts/local.sh, then the environment.

        

      

      

        2
        

          
Look, then start

          
DRY_RUN=1 ./start.sh   # the whole plan: the link, the docker build, both ranks' command lines
./start.sh

          
The first run checks both Sparks, builds the image once (about 6 minutes) and sends it to the worker by content, places the weights, starts rank 1 on the worker and rank 0 here, each PID 1 in its own container, then waits for /health, /v1/models and a greedy smoke test. A rank that dies stops the other. Later starts are quicker; the very first also hashes all 164 GiB once per rank, cached.

        

      

      

        3
        

          
Talk to it

          
Any OpenAI client works with base_url = "http://<head-address>:8020/v1" — the port is PORT in scripts/local.sh — and the model id GLM-5.3-Flash (SERVED_NAME). Thinking is off by default; a request can ask for it with reasoning_effort.

          
curl -s http://<head-address>:8020/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "GLM-5.3-Flash",
  "messages": [{"role": "user", "content": "Write a Python fibonacci function."}],
  "max_tokens": 2000
}'

          
./stop.sh   # rank 0 first (it drains for 20 s), then rank 1; logs archived to logs/glm53-spark-*.tar.gz

          
logs/current holds rank0.log and rank1.log, collected from each Spark's Docker output. stop.sh only ever stops containers labelled tf.recipe=glm53-spark. Every rank carries the recipe's TF_*/TENSORFOLD_*/NCCL_* settings; any other such variable you export reaches both ranks by name, TENSORFOLD_API_KEY included (never printed).

        

      

    

    

      
If something stops

      

        
SymptomWhat it means

        
          
[E_WORKER]ssh, Docker or the NVIDIA runtime is not ready on the worker; the message names which.

          
[E_FABRIC]No directly reached address, or no RoCE v2 GID holding an IPv4 on the link's port: cable the Sparks on one subnet, or set FABRIC_PEER / NCCL_IB_GID_INDEX.

          
[W_FABRIC] a rail left outThe second rail does not carry a full-size ping: check the cable, the twin's address and the MTU.

          
[E_MEMORY], [E_PORT]Less than MIN_AVAIL_GIB available (another rank, usually), or a port in use.

          
RankMismatch, RankNeverCame, NCCL timeoutsThe ranks were started differently, or rank 1 did not reach rank 0 within 300 s: check rank1.log and the ports; NCCL_DEBUG=INFO for detail.

          
vendor_err 0x81A rail without a path to its partner was set by hand through TF_ROCE_HCA.

          
A long prompt dies in a RoCE gatherROCE=off puts every exchange on NCCL.

          
--parallel N needs … GiB, [memguard]The window or stream count does not fit, or a Spark went under its floor: lower CONTEXT/PARALLEL or raise MEMORY_RESERVE_GIB.

          
No prompt cache in the startup lineThe page pool took a third window: raise MEMORY_RESERVE_GIB.

        
      

      
The full table, and everything else, is in the recipe's README.

    

  







  

    
Settings

    
The ones you might change.

    
Every setting lives in scripts/config.sh, with its default. Set one for a single run from the environment, or keep it in scripts/local.sh or a .env file. The served pair's own local.sh, for the record: PORT=8888, PARALLEL=8, CONTEXT=auto, KV=fp8, DENSE=q8, HEAD=q8, ABSORB_MMA=1, HOST_CACHE_GIB=0, MEMORY_RESERVE_GIB=9.5, MODEL_SOURCE=local, WORKER_WEIGHTS=check.

    

      

        
VariableDefaultMeaning

        
          
WORKERrequiredThe second Spark's ssh target on the direct link, e.g. [email protected]

          
TP2Two Sparks; other values are refused

          
PORT / NCCL_PORT8020 / 29583The API address, and rank 0's bootstrap port on the link

          
ROCE / ROCE_MAX_KBon / 4096The engine's one-shot RDMA all-gathers, and the largest shard they carry; off puts everything on NCCL

          
WORKER_WEIGHTSrsyncrsync the checkpoint over the link, nfs (the worker mounts it), or check (refuse unless present)

          
PARALLEL4Requests decoded together; auto serves up to 8. The pair runs 8: free at 1-4 streams, 94-95 tok/s at 6, 105-106 at 8

          
CONTEXT / CONTEXT_MAXauto / 1048576The window dance, or a fixed token count

          
KV / KV_PAGESfp8 / onOne shared pool of 512-row pages, CONTEXT as one request's cap

          
DENSE / HEADq8 / q8bf16, q8, q6 or q4 for the dense linears and the LM head

          
MEMORY_RESERVE_GIB / HOST_CACHE_GIB12 / 0The free-memory floor, and the pinned host tier (keep it at 0 on a Spark); the pair runs the reserve at 9.5

          
PREFILL_SHARE0.68How much of each prompt chunk's time is owed to the decoders first; above

          
DRAFTER / DRAFTER_BITSdflash2 / 4dflash2 (non-commercial license), mtp (the checkpoint's own head) or none

          
ABLIT01: serve the gated Ablit weights; needs HF_TOKEN

          
THINKING / REASONING_EFFORT0 / highRequest defaults; earlier turns' reasoning is dropped from the prompt

          
ABSORB_MMA / W8A81 / 0Prompt numerics: ABSORB_MMA is free (+8.0%, identical answers); W8A8 rounds activations and stays off

          
VISION / VISION_MAX_IMAGES1 / 50Image input on rank 0's tower (not yet verified in this build); 0 serves text only

          
MEMGUARD / MEMGUARD_FLOOR_GIB1 / 2.5The memory guard, and the available-memory floor it keeps

          
SM / ZIG_CPU121 / nativeGB10's target; baseline builds an image that runs elsewhere too

          
SERVED_NAMEGLM-5.3-FlashThe model id clients send

          
DRY_RUN01 prints the whole plan, and changes nothing

        
      

    

  







  

    
Ablit weights · ABLIT=1

    
The same model, abliterated.

    
ABLIT=1 serves Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold-Ablit, the same checkpoint abliterated, at the revision pinned in scripts/config.sh — everything else (drafts, the window, the memory budget) works the same. Thinking off, the recipe's default, is also how these weights give their best results.

    

      

        
Turn it on

        
The weights are gated on Hugging Face, so the server needs your token.

        
    
          
  1. Open the repo's page on Hugging Face, log in and agree to its terms.
    2. 
          
  2. With the same account, create a token with read access at huggingface.co/settings/tokens.
    3. 
          
  3. Set both in scripts/local.sh (or .env), then restart.
    4. 
        

        
ABLIT=1
HF_TOKEN=hf_...

      

      

        
Good to know

        
    
          
  • Checked before anything happens: start.sh verifies that HF_TOKEN reaches the repo before building or downloading. A 401 means Hugging Face doesn't accept the token; a 403, that its account hasn't agreed to the terms yet.
    • 
          
  • The first Ablit start downloads it, beside the published checkpoint, and the worker gets it like any other set.
    • 
          
  • ABLIT=0 goes back to the published checkpoint, which stays in the cache.
    • 
        

      

    

    
Access needs you to agree to a Responsible Use Agreement on Hugging Face: you must be 18 or older, and you take responsibility for adding your own safety filtering, review and access controls. The weights carry the upstream GLM-5.3-Flash license.

  







  

    
The patch stack

    
105 patches, each earned.

    
patches/engine/ holds 105 patches on TensorFold f350cd3e, applied in filename order at image build. 0001-0093 are my two-RTX-PRO-6000 recipe's, unchanged; 0094-0105 are this pair's own or ported from that recipe. None is GB10-specific: the arch-gated kernels test for compute capability 1200 and up, which sm_121 satisfies. Every default-on patch was A/B tested against the image before it: greedy replies identical, plus a measured gain.

    

      

        
PatchesWhat they do

        
          
0001-0012EXL3 experts at mixed bit widths, 3-bit prompt kernels; FP8 attention loads, the shared expert on a side stream; one expert call a prompt pass, no MTP pass beside DFlash2

          
0013-0016DFlash2 reads complete prompt taps at TP>1 (upstream #1623); no target MTP layer beside DFlash2; a pinned host prompt-state tier (off here)

          
0017-0033FP8 KV on shared pages; EXL3 load-ahead, fused MoE glue, KDA in one launch, one broadcast a follower step

          
0034-0035Peer-store all-gather and copy-engine prompt exchange, both stepping aside across hosts

          
0036-0048DFlash2 copy rounds, ABSORB_MMA (opt-in), "fits:" refusals, the device prompt-cache share

          
0049-0066Paired prompt chunks (upstream #1423) extended to DFlash2; W8A8 (opt-in)

          
0067-0091DSA select on many CTAs, audited programmatic dependent launches along the decode chain, glue fusions, EXL3 prompt and decode kernels; each keeps an off switch. Never ship 0086 or 0089 without 0090

          
0092-0093A JSON answer stays JSON beside offered tools; a reply grammar is freed when its stream cannot start

          
0094-0097This pair's. A 4,096-row prompt chunk with DFlash2's MAX_CTX to match (+7% prefill, byte-identical replies); the GB10's measured cost tables and draft-rule seed

          
0098-0102This pair's, env-gated, defaults behaviour-identical: the decode campaign's exact knobs, each measured neutral

          
0103-0104From the RTX recipe: a tool call after prose that quotes a tool-call opener is the call; warm turns, so a user message after an agent loop resumes past the loop

          
0105An untyped tool parameter takes the JSON object or array the model wrote; text stays text

        
      

    

  







  

    

      

        
The engine

        
Powered by TensorFold.

        
TensorFold, by Ash Hart, is an open-source engine that serves language models on Apple Silicon and NVIDIA GPUs through an OpenAI-compatible API. Each model family brings its own kernels and its own draft verification, tuned for that model instead of one generic path for all.

      

      

        
Exact decoding
Drafts make it fast without changing a word: a draft is accepted only when it equals the token the same engine would produce one at a time.

        
Apple Silicon and NVIDIA
MLX on Macs, and CUDA on NVIDIA GPUs from compute capability 8.9: Ada (RTX 40), Hopper and Blackwell, including the DGX Spark's GB10 and the RTX 50 series.

        
Many model families
GLM-5.3-Flash, Qwen3.8-27B, Qwen3.8 Flash Next, Nemotron 3.5, Gemma 4 and DeepSeek-V4-Flash, among others, each with its own kernels.

        
In this recipe
The native Zig server at pin f350cd3e, one rank per Spark over RoCE, with 105 patches: FP8 KV on shared pages, DFlash2 drafts, the 4,096-row prompt chunk, and the pair's own memory accounting.

      

      

        TensorFold on GitHub 
        More TensorFold recipes:
        GLM 5.3 Flash on two RTX PRO 6000s, Zig engine →
        GLM 5.3 Flash on two RTX PRO 6000s →
        GLM 5.3 Flash on four RTX PRO 6000s →
        The full GLM 5.3 on three Sparks →
        DeepSeek V4.1 Flash on four Sparks →
        Qwen3.8 Flash Next on two Sparks →
        Qwen3.8 Flash Next on one Spark →
        Qwen3.8 27B on one Spark →
        Nemotron 3.5 Bolt on one Spark →
        Qwen3.8 Flash Next on one RTX PRO 6000 →
      

    

  







  

    

      
Licenses

      
Free, with a few rules.

      
    
        
  • The recipe is Apache License 2.0. Its patches modify TensorFold at f350cd3e, whose code keeps TensorFold's licenses.
    • 
        
  • The checkpoint, Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold, is by Mia's AI Lab, from zai-org's GLM-5.3-Flash (MIT).
    • 
        
  • The DFlash2 drafter is CC BY-NC-ND 4.0, non-commercial use only. DRAFTER=mtp or none serves without it.
    • 
        
  • The Ablit weights are gated behind a Responsible Use Agreement and carry the upstream GLM-5.3-Flash license.
    • 
        
  • Downloaded software and weights keep their own terms; required notices are in the repo's NOTICE.md.
    • 
      

    

    

      
Credits

      
Built on the work of others.

      
      
The full list is in the recipe's CREDITS.md.

    

  







  

    

      
Run it on your pair.

      
Free and open, with every patch and every number published.

      

        Get the free recipe 
        See all recipes