DeepSeek-V4.1-Flash-1017-4x-DGX-Sparks-TensorFold

Performance

Measured, not promised.

One request65.8tok/s, prose

Eight at once165.3tok/s combined

Context1Mtokens a request, up to 16 at once

Prefill4,127tok/s, 32k prompt

Measured on 2026-10-25 on four Sparks on a switchless ring (no RoCE switch, ./start-switchless.sh), at the defaults: a 1,048,576-token window, 16 streams, DSpark drafts, CUDA graphs, side stream and L2 prefetch on. Median of three runs, from spark1. "Aggregate" is the total tok/s across every request at once, "per request" is one request's own share, and "first token" is the time to the first token. A switch removes the ring's one relayed hop between the two nodes that have no direct cable; these numbers are this fabric's, not a switch's.

Decode, prose

Non-greedy (T = 0.7), thinking on.

Requests	Aggregate	Per request	First token
1	65.8 tok/s	65.8 tok/s	186 ms
2	96.5 tok/s	48.9 tok/s	304 ms
4	130.5 tok/s	34.1 tok/s	499 ms
8	165.3 tok/s	21.7 tok/s	901 ms

Decode, prose at 128k

The same, with every session starting at 128k tokens of context.

Requests	Aggregate	Per request	First token
1	58.6 tok/s	58.6 tok/s	1.07 s
2	83.1 tok/s	44.3 tok/s	1.61 s
4	111.3 tok/s	31.0 tok/s	2.61 s
8	132.3 tok/s	19.2 tok/s	4.67 s

Decode, code

Greedy (T = 0), thinking off. Code is easier to draft, so it runs faster than prose.

Requests	Aggregate	Per request	First token
1	117.4 tok/s	117.4 tok/s	192 ms
2	158.2 tok/s	80.6 tok/s	262 ms
4	174.4 tok/s	46.2 tok/s	482 ms
8	183.7 tok/s	25.3 tok/s	833 ms

Prefill

How fast a cold prompt is read before the reply starts, one request.

Prompt	Prefill	First token
4,116 tokens	3,497.9 tok/s	1.33 s
8,210 tokens	3,851.9 tok/s	2.28 s
16,400 tokens	4,051.9 tok/s	4.20 s
32,787 tokens	4,127.2 tok/s	8.10 s
65,559 tokens	4,134.1 tok/s	16.01 s
131,093 tokens	4,058.4 tok/s	32.45 s
262,161 tokens	3,929.4 tok/s	66.87 s

Checked, not just timed

The recipe's tools/validate.py passes: text, thinking, a tool call and its result, an image, a video, a 21,875-token prompt, eight greedy replies sent at once equal to the same replies sent one at a time, and drafted replies equal to undrafted ones. The native kernels are checked against DeepSeek's reference arithmetic on real weights.

Needle in a haystack (tools/needle_bench.py): a random code hidden at 0, 25, 50, 75 and 100 % depth, at nine context lengths from 4k up to 997,530 tokens, effectively the model's full 1,048,576-token window. All 45 of 45 cells retrieved it correctly, including the longest, deepest one (997k tokens, needle at 100 % depth, 156 s). The recipe's prompt-sharding patches don't degrade long-context retrieval at any tested depth.

Opt-in extras

Three switches, off by default, add more with the same replies: TF_DS_EXPERT_EXIT=1 and TF_DS_SAMPLE_CAND=1 together give level at one request, +1.6 % to +2.8 % at 4 and +1.9 % to +8.2 % at 8. TF_DS_PREFILL_YIELD=1 keeps the other requests going while a long prompt is read: with 15 requests decoding, a 25,000-token prompt stalled them about 7.5 s, and about 1 s with it.

Speed preview

What 65.8 tok/s feels like.

Sample text played at the measured decode speed: one request at 65.8 tok/s, or four at once at 34.1 tok/s each (130.5 combined), at about four characters a token. Switch between them in the window.

TensorFold · DeepSeek-V4.1-Flash-1017 · 4× DGX Spark

65.8 tok/sSample text, played at the measured speed

How it is built

Four Sparks, one model, no switch.

The checkpoint is about 510 GB, so it needs four Sparks in tensor parallel (TP4). Rank 0, on spark1, also serves the API and runs the vision tower and the sampler. The other three Sparks read the checkpoint and their weight files from spark1 over NFS, so spark1 holds the only copy.

A ring or a switch

The recipe runs the four Sparks either way. On a switch (every Spark on one RoCE network): ./start.sh. As a ring without a switch: spark1, spark2, spark3, spark4 and back to spark1, one QSFP cable between neighbours: ./start-switchless.sh. scripts/nodes.sh works out which one you have from the addresses, and each script stops on the other kind and names the right one.

A 2 × 2 split

Each Spark is a node of a 2 × 2 grid, with a row partner and a column partner. On a ring, ORDER=auto places the nodes so every partner pair has a cable of its own: the partners' exchanges run over two-rank NCCL communicators, one per cable, and the all-gathers go round the ring.

Weights as published

DeepSeek's FP8 and MXFP4 checkpoint is read as it is, on new CUDA kernels, with no re-quantization. The native kernels are checked against DeepSeek's reference arithmetic on real weights. Each Spark takes about 80 GiB of weights.

DSpark drafts, checked

The checkpoint carries DSpark draft blocks: after each decode step they propose a block of 5 tokens, which the next step verifies in one pass. Verification is exact, so a drafted reply equals the undrafted one and drafts only change speed. On prose they give about 1.8 tokens a step. DRAFTS=0 turns them off.

The recipe adds 25 patches to TensorFold's DeepSeek-V4.1 engine, all Mia's AI Lab's except 0008 (Hector's): DeepSeek's FP8/MXFP4 checkpoint, the ring's node order and links, video input, tool-call arguments, and a long list of prefill and decode speed-ups. Each one, with its flag and its measurements, is in the recipe's CHANGELOG and CREDITS.

The engine

Powered by TensorFold.

TensorFold, by Ash Hart, is an open-source engine that serves language models on Apple Silicon and NVIDIA GPUs through an OpenAI-compatible API. Each model family brings its own kernels and its own draft verification, tuned for that model instead of one generic path for all.

The DeepSeek family

The DeepSeek-V4.1-Flash family on TensorFold's Zig engine (the four-node 2D split, Engram, DSpark drafting, the vision tower, the weight-file and token-map tools) and the kernel kit are BertholomusAI's (Albert Lee) work, in their TensorFold fork.

Apple Silicon and NVIDIA

MLX on Macs, and CUDA on NVIDIA GPUs from compute capability 8.9: Ada (RTX 40), Hopper and Blackwell, including the DGX Spark's GB10 and the RTX 50 series.

Many model families

Qwen3.8 Flash Next, Qwen3.8-27B, GLM-5.3-Flash, Nemotron 3.5, Gemma 4 and DeepSeek-V4, among others, each with its own kernels.

In this recipe

TensorFold's native Zig engine (Zig 0.17.0) at BertholomusAI's deepseek-v41-zig-tp4 branch, commit 5940208, plus this recipe's 25 patches: the checkpoint as published, the ring, video input and tool-call arguments.

TensorFold on GitHub More TensorFold recipes: Nemotron 3.5 Bolt on one Spark → Qwen3.8 Flash Next on one Spark → Qwen3.8 Flash Next on two Sparks → Qwen3.8 27B on one Spark → The full GLM 5.3 on three Sparks → GLM 5.3 Flash on two or three Sparks → GLM 5.3 Flash on four RTX PRO 6000s → GLM 5.3 Flash on two RTX PRO 6000s →

What you need

Four Sparks, and a cable between neighbours.

Four DGX Sparks

Linked by their ConnectX-7 ports, on one RoCE switch or as a ring (spark1 - spark2 - spark3 - spark4 - spark1, one QSFP cable between neighbours). About 100 GB of free memory on each Spark: the weights take about 80 GiB a node.

Docker and ssh

Docker with the NVIDIA runtime on every Spark, and key-based ssh from spark1 to the other three.

Disk, on spark1

About 510 GB for the checkpoint and about 320 GB for the four weight files, under the Hugging Face cache (HF_CACHE, default ~/.cache/huggingface).

An NFS export

spark1 exports that cache, read-only, to each worker's address on its link; scripts/prepare.sh makes a Docker volume on each worker, mounts it and checks that every file a rank reads is there, at the right size. The default path is /huggingface (NFS_PATH).

Quick start

One command on spark1.

1

Clone and point it at your Sparks

git clone https://github.com/MiaAI-Lab/DeepSeek-V4.1-Flash-1017-4x-DGX-Sparks-TensorFold.git
cd DeepSeek-V4.1-Flash-1017-4x-DGX-Sparks-TensorFold
cp scripts/local.sh.example scripts/local.sh

Set WORKER, WORKER2 and WORKER3 in scripts/local.sh to spark2, spark3 and spark4 (ssh targets; on a ring, in its order), and NFS_PATH if your export puts the cache somewhere other than /huggingface.

2

Start it

./start.sh                # on a switch
./start-switchless.sh     # on a ring, no switch

The first run builds the image, downloads the checkpoint, writes the four weight files (one per node, about 80 GB each) and the workers' NFS volumes, then starts the four ranks and runs a smoke test. Later runs start in about two and a half minutes.

3

Talk to it

Any OpenAI client works with base_url = "http://<spark1-address>:8888/v1" and the model DeepSeek-V4.1-Flash-1017.

curl -s http://<spark1-address>:8888/v1/chat/completions -H 'content-type: application/json' -d '{
  "model": "DeepSeek-V4.1-Flash-1017",
  "messages": [{"role": "user", "content": "Hello!"}],
  "chat_template_kwargs": {"enable_thinking": false}
}'

4

Run it day to day

./start.sh restart              # restart with changed settings (or ./start-switchless.sh restart)
./stop.sh                       # stop all four ranks
docker logs -f dsv41-1017-tf    # a Spark's server log

To use Blackfrost-AI's DERISKED FP8 checkpoint instead, set DERISKED=1 in scripts/local.sh (or run DERISKED=1 ./start-switchless.sh restart); DERISKED=0 returns to the original. The first switch downloads and prepares separate weights.

Images and video

Pictures and clips, as data URLs.

User turns and tool results take image_url and video_url parts as base64 data: URLs (remote URLs are off).

Images go to the checkpoint's vision tower on rank 0. PNG is read natively; other formats go through Pillow.
Video has no tower of its own in this model, so the server samples the clip into frames and sends them as images, each after a text line Video frame at <t> s:, in time order. PyAV in the image decodes MP4, WebM, MOV and the other formats FFmpeg reads.
Frames: TF_VIDEO_FPS a second (default 2), at most TF_VIDEO_MAX_FRAMES (default 32; a longer clip's frames are spread evenly over it), each scaled to at most TF_VIDEO_MAX_SIDE pixels a side (default 768). Every frame costs prompt tokens like an image.

Thinking and the API

Thinking on, by default.

The model thinks first unless a request sends "chat_template_kwargs": {"enable_thinking": false}. The effort a request names is used; with none, the server uses REASONING_EFFORT (max by default; low is 50, high 75, max 100). A request's own sampling settings win over the server's defaults (temperature 1.0, top_p 1.0, top_k 20).

OpenAI-compatible on port 8888, model id DeepSeek-V4.1-Flash-1017.
Text, images, video, tool calling, and thinking on or off.
DSpark drafting from the checkpoint's own draft blocks, on by default.

Memory and settings

A million tokens, sixteen at once.

The KV pool

The pool is the number of positions all requests share. The recipe's README runs it at its true ceiling, 16,777,216 positions (PARALLEL 16 × CONTEXT 1,048,576), so all 16 streams can each hold the model's full ~1M-token context at once, with nothing wasted past that. On the Sparks it was measured on, that leaves three of the four at about 15-18 GiB free and the fourth, which also runs other workloads, at about 11 GiB.

The shipped default for POOL is 2,097,152. Set it to 16,777,216 for the full 16 × 1M, and lower it if a Spark on your own setup runs other memory-heavy work too.

The ones you might change

Settings come from the environment, then scripts/local.sh, then ./.env, then the defaults in scripts/config.sh, which documents each one.

Variable	Default	Meaning
`CONTEXT`	1048576	A request's positions at most (the model's window)
`POOL`	2097152	The positions all requests share
`PARALLEL`	16	Requests at once
`THINKING`	1	Think first unless a request sends `enable_thinking: false`
`REASONING_EFFORT`	max	The effort when a request names none
`DRAFTS`	1	DSpark drafting; `0` turns it off
`GRAPHS` / `SIDE` / `PREFETCH`	1 / 1 / 1	CUDA graphs, the side stream, the paced L2 prefetch
`ABLIT`	0	`1` grafts the Keys abliteration overlay (above); gated, needs `HF_TOKEN`
`DERISKED`	0	`1` loads Blackfrost-AI's DERISKED FP8 checkpoint
`ENGRAM_PACKED`	0	Engram's packed rows (opt-in, below)
`PORT`	8888	Where the API listens

ABLIT (opt-in)

ABLIT=1 grafts drowzeys's Keys abliteration overlay onto the checkpoint before the rank caches are baked: a refusal-direction ablation of attn.wo_b in layers 10-35 only. Everything else stays stock: layers 0-9 and 36-39, all MTP, vision, Engram, every expert, embed and head. The overlay is gated and free: accept its terms while logged in on Hugging Face, then export HF_TOKEN=hf_... or run hf auth login with that account first.

ABLIT=1 ./start-switchless.sh restart

ABLIT=0 returns to the stock checkpoint. The first switch downloads the overlay's sidecar (about 1.1 GB) and rebuilds all four weight files from it, the usual rank-cache build (about 80 GB, a few minutes), once.

Checked on this cluster, not just by the overlay's author: the grafted wo_b tensors in the built checkpoint match the overlay's sidecar byte for byte, tools/validate.py all passed, and a needle spot check at 65k and 262k tokens (25 % and 75 % depth) passed at the same speed as the stock numbers above.

Engram's packed rows (opt-in)

Each decode round and prompt chunk reads Engram rows from two tables of about 384 million rows each (about 203 GB). ENGRAM_PACKED=1 writes packed shards once on every Spark's own disk, so a row is one read instead of two; with ENGRAM_OWNED=1 (the default with packing) each Spark keeps only its quarter of the rows, about 51 GB, and one exchange a layer fills in the rest.

The rows are the same bytes either way, so replies should not change, but this path is new and unmeasured: run tools/validate.py and compare replies with the default path before relying on it.

Licenses

Free, with a few rules.

The recipe's own scripts, patches and documentation are Apache-2.0; its NOTICE file lists the third-party work and its licenses.
The model weights are DeepSeek's, under the MIT license, and are not part of the recipe.
"DeepSeek" belongs to DeepSeek; "DGX Spark" and "GB10" belong to NVIDIA. This recipe is not affiliated with or endorsed by DeepSeek, NVIDIA, the TensorFold authors or BertholomusAI.

Credits

Built on the work of others.

DeepSeek: DeepSeek-V4.1-Flash-1017, its weights, its DSpark draft blocks and its reference inference code
TensorFold, by Ash Hart and the TensorFold contributors: the native Zig engine and its OpenAI-compatible server
BertholomusAI (Albert Lee): the DeepSeek-V4.1-Flash family on TensorFold's native engine, the kernel kit and the served settings this recipe starts with
drowzeys: the Keys abliteration overlay behind the optional ABLIT=1
Hector: patch 0008 (pair-sum gathers) and the prefill measurements behind it
The GLM 5.3 Flash TensorFold recipe's contributors, whose code the scripts keep: webzone (twin rails), Alexbob0 (workers by host name) and john (each rank's rendezvous address)
PyAV, Zig, NVIDIA's PyTorch container and huggingface_hub

The full list is in the recipe's CREDITS.md.

Run it on your Sparks.

Free and open, with every patch and every number published.

Get the free recipe See all recipes

DeepSeek V4.1 Flash on four DGX Sparks, with TensorFold.

Install this model