Nemotron 3.5 Bolt on one DGX Spark with TensorFold

Performance

Measured, not promised.

One request65.8tok/s, prose

Eight at once176.1tok/s combined

Context262ktokens a stream, up to 16 at once

Prefill1,375tok/s, 32k prompt

One DGX Spark, the recipe's defaults (16 streams, a 262,144-token window each, bf16 KV cache, vision and the LiLiCorr drafter on), measured with sparkDash against the running server. "Aggregate" is the tokens of every request at once; "first token" is the time to the first token.

Decode, prose

Aggregate across the concurrent requests, per request, and time to first token.

Requests	Aggregate	Per request	First token
1	65.8 tok/s	65.8 tok/s	62 ms
2	97.0 tok/s	49.6 tok/s	117 ms
4	131.2 tok/s	35.0 tok/s	219 ms
8	176.1 tok/s	24.2 tok/s	487 ms
16	171.2 tok/s	23.2 tok/s	6.20 s

Decode, code

The same server, with code prompts: the drafter guesses code more often.

Requests	Aggregate	Per request	First token
1	119.5 tok/s	119.5 tok/s	65 ms
2	159.4 tok/s	81.5 tok/s	107 ms
4	210.9 tok/s	58.0 tok/s	229 ms
8	278.4 tok/s	38.4 tok/s	597 ms
16	267.7 tok/s	36.0 tok/s	8.05 s

Prefill

How fast a cold prompt is read before the reply starts, one request.

Prompt	Prefill	First token
4,135 tokens	1,575.6 tok/s	2.80 s
8,231 tokens	1,536.8 tok/s	5.53 s
16,424 tokens	1,480.8 tok/s	11.27 s
32,809 tokens	1,375.1 tok/s	24.03 s
65,579 tokens	1,212.3 tok/s	54.27 s
131,115 tokens	986.8 tok/s	133.04 s

Room to spare

Loading the model with the defaults takes 51.1 GB of the Spark's unified memory: 16 streams, a 262,144-token window each, vision and drafter on. That leaves roughly 70 GB of the 121 GB for whatever else you run beside it.

A chat turn or a repeated system prompt reads only its new tail: the prompt cache keeps each stream's state at its marks, and replies are identical with and without it. A follow-up turn that repeats 50 images and 4 videos (30,500 tokens) answers in 1.7 s.

Speed preview

What 65.8 tok/s feels like.

Sample text played at the measured decode speed: one request at 65.8 tok/s, or four at once at 35.0 tok/s each (131.2 combined), at about four characters a token. Switch between them in the window.

TensorFold · Nemotron-3.5-Bolt · 1× DGX Spark

65.8 tok/sSample text, played at the measured speed

How it is built

A new backend, written for Bolt.

TensorFold's Nemotron family serves Lightning 30B-A3B. Bolt is a different shape: 88 blocks, an expert count that changes by layer, and experts packed at about 2 bits. So the recipe adds a Bolt backend to TensorFold's Zig engine. The HTTP server, chat template, thinking budget, round loop, stop handling and sampler stay TensorFold's own; the forward pass is the recipe's CUDA library, libbolt.so.

The checkpoint

74B parameters, 9B active, 9 of up to 496 experts a token. Of the 88 blocks, 40 are Mamba, 40 are MoE and 8 are attention. NVIDIA's 24GB pack is 22.45 GiB: IQ2_XXS experts, NVFP4 Mamba projections, Q8_0 attention, latent projections and head, and the C-RADIO v4-H vision tower.

Weights stay packed

At one request a token reads about 5 GB, not the 14.6 GB of an fp16 expansion. Each matrix is repacked at load so the dense GEMMs run on tensor cores, and the routed experts run on int8 tensor cores. A block of rows, embedding to head, replays as one CUDA graph.

Drafts, checked

The checkpoint has no MTP head, so the recipe serves it with NVIDIA's LiLiCorr drafter (about 1.3 GB). It proposes seven tokens a stream each round, for every stream in one pass, and TensorFold verifies them, choosing how many to check from costs measured at load. Rollback after a partial accept is a pointer flip.

Memory follows use

Every stream has the full 262,144-token window, but its attention rows are mapped 4,096 tokens at a time and given back when it ends. Only the 0.4 GiB of recurrent state a stream is held from load, so sixteen streams start small and grow with their conversations.

Three bugs had made the old engine's drafter accept 0 of 7 tokens; all three are fixed: the shared expert read the norm of the updated residual instead of the block input, Mamba's dt was clamped at 0.001 (the config's limit is 0), and the drafter applied its grouped convolution before o_proj instead of after. More in the recipe's engine notes and CHANGELOG.

The engine

Powered by TensorFold.

TensorFold, by Ash Hart, is an open-source engine that serves language models on Apple Silicon and NVIDIA GPUs through an OpenAI-compatible API. Each model family brings its own kernels and its own draft verification, tuned for that model instead of one generic path for all.

Lane backends

A model can bring its own forward pass and keep everything else: TensorFold's server, chat template, sampler and speculation loop drive it through verify, keep and draft. That is how Bolt plugs in.

Apple Silicon and NVIDIA

MLX on Macs, and CUDA on NVIDIA GPUs from compute capability 8.9: Ada (RTX 40), Hopper and Blackwell, including the DGX Spark's GB10 and the RTX 50 series.

Many model families

Qwen3.8 Flash Next, Qwen3.8-27B, GLM-5.3-Flash, Nemotron 3.5 Lightning, Gemma 4 and DeepSeek-V4-Flash, among others, each with its own kernels.

In this recipe

TensorFold's Zig engine, tensorfold-native, at a pinned 1.0.12 commit, with its RADIO image tower and video decoder. The recipe's patches register the Bolt family and its video input, and libbolt.so runs the model.

TensorFold on GitHub More TensorFold recipes: Qwen3.8 Flash Next on one Spark → Qwen3.8 Flash Next on two Sparks → Qwen3.8 27B on one Spark → The full GLM 5.3 on three Sparks → GLM 5.3 Flash on two or three Sparks → GLM 5.3 Flash on four RTX PRO 6000s → GLM 5.3 Flash on two RTX PRO 6000s →

What you need

One Spark, half of it.

A DGX Spark

Or another GB10 system with 128 GB unified memory. Sixteen streams need about 55 GiB free when the server starts: weights, the vision tower, recurrent state, a working cache, the prompt cache and a 10 GiB floor. start.sh stops before launching below that.

Docker

Docker with the NVIDIA container runtime, and your user in the docker group.

Disk

About 23 GiB for the checkpoint under ~/.cache/huggingface, 1.3 GB for the drafter, plus the image under Docker's root; scripts/prepare.sh checks both.

Optional

The hf CLI and a Hugging Face token in ~/.cache/huggingface/token or HF_TOKEN. The checkpoint is public, so neither is required.

Quick start

Three lines, that's all.

1

Clone and start

git clone https://github.com/MiaAI-Lab/Nemotron-3.5-Bolt-TensorFold-DGX-Spark.git
cd Nemotron-3.5-Bolt-TensorFold-DGX-Spark
./start.sh

The first run builds the image (TensorFold at the pinned commit, the recipe's patches and its engine), downloads the checkpoint (23 GiB) and the LiLiCorr drafter (1.3 GB), checks them against the recipe's contract, starts the server and sends a smoke request. Later runs reuse the image and the files, and load in about 45 seconds.

2

Talk to it

Any OpenAI client works with base_url = "http://<spark-address>:8888/v1" and the model Nemotron-3.5-Bolt. The model thinks before it answers, and the reasoning and the answer share max_tokens, so give replies enough; a request without one gets 32,768.

curl -s http://<spark-address>:8888/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "Nemotron-3.5-Bolt",
  "messages": [{"role": "user", "content": "Write a Python fibonacci function."}],
  "max_tokens": 1000
}'

3

Run it day to day

./start.sh restart                            # restart it, e.g. after changing a setting
./stop.sh                                     # stop the server
docker logs -f nemotron35-bolt-tf             # server log
DRY_RUN=1 ./start.sh                          # print the docker command, change nothing

Images and video

Up to 64 images a request.

The checkpoint's own C-RADIO tower (1.7 GB) turns pictures and video frames into tokens, through TensorFold's image tower. Send them as OpenAI-style content parts: an image_url part, or a video_url part for a clip.

	Images	Videos
Formats	PNG and JPEG, up to 10 MB, 8,192 px a side and 16 megapixels	MP4 or WebM (H.264, HEVC, VP8, VP9, MPEG-4), up to 16 MB a clip, an hour long
Per request	Up to 64	Up to 4, 20 MB in all
Tokens	256 to 3,328 an image (a 512 px square about 260)	2 frames a second, at most 256; a 6 s clip is 1,700 tokens, a 2-minute 720p clip 36,500

Tested: 50 images of 2560×1920 (165,600 tokens) in one chat, every one read correctly. Data URLs by default; VISION_URLS=1 also fetches public HTTPS links. VISION=0 serves text only.

Thinking and the API

Thinking on, by default.

The model thinks before it answers. "chat_template_kwargs": {"enable_thinking": false} answers directly, and {"enable_thinking": true, "medium_effort": true} is the model card's medium effort. A request's own temperature, top_p and seed override the defaults (1.0 and 0.95; temperature: 0 is greedy).

OpenAI-compatible on port 8888, model id Nemotron-3.5-Bolt.
Images on the chat route, the Anthropic route (image blocks) and the Responses route (input_image).
A prompt that shares a prefix with an earlier one, including the same pictures, is read from the prompt cache.
Not served: logprobs, grammar-constrained decoding, int8 or int4 KV caches, WebP, GIF and BMP pictures.

Memory and settings

Sixteen streams in 51 GB.

Where the memory goes

The weights are 22.45 GiB on disk. Each stream holds 0.4 GiB from load: two copies of its Mamba state, so a draft window can be undone, plus the drafter's context. Its attention rows grow with the conversation, 8 KB a token in bf16, and are given back when it ends. The 8 GiB prompt cache fills with use.

Window	bf16 attention cache
32,768 tokens	0.25 GiB
131,072 tokens	1.00 GiB
262,144 tokens	2.00 GiB (1.03 GiB with `KV_DTYPE=fp8`)

For a fixed footprint, KV_PIN=1 reserves every stream's whole window at load (16 GiB for 8 × 262,144 in bf16), so a start that cannot have it fails at once. Keep at least 10 GiB free under load: on this machine, running out of unified memory freezes it.

The ones you might change

Every setting lives in scripts/config.sh; override one from the environment (PARALLEL=2 ./start.sh) or a .env file. The full list is in the README.

Variable	Default	Meaning
`PARALLEL`	16	Streams decoded together (1 to 16); more requests queue
`CONTEXT`	262144	Prompt + reply window of every stream, the model's maximum
`KV_DTYPE`	bf16	`fp8` halves the cache, at no speed cost
`KV_PIN`	0	`1` reserves every stream's whole window at load
`PROMPT_CACHE_GIB`	8	Kept prompt states; `0` turns the cache off
`DRAFTS`	1	LiLiCorr drafts; `0` serves one row at a time
`VISION`	1	Image and video input; `0` serves text only
`THINKING`	1	Think before answering; `0` answers directly
`MAX_TOKENS`	32768	Reply length for a request without `max_tokens`
`PORT`	8888	Where the API listens

Licenses

Free, with a few rules.

The recipe is MIT.
The model weights, downloaded from Hugging Face and not part of the recipe, are under NVIDIA's OpenMDW License Agreement, version 1.1.
The image is based on NVIDIA's PyTorch container; by pulling or running it you accept NVIDIA's terms, which the container prints at every start. It also holds FFmpeg (decoders only, LGPL), libjpeg-turbo and ggml's IQ2_XXS tables (MIT).

Credits

Built on the work of others.

TensorFold and its Zig engine, image tower and video decoder, by Ash Hart and the TensorFold contributors
Nemotron 3.5 Bolt 74B-A9B 24GB, its C-RADIO vision tower and the LiLiCorr drafter, by NVIDIA
ggml and llama.cpp, for the IQ2_XXS and Q8_0 formats the checkpoint is packed in
FFmpeg, libjpeg-turbo and Zig

The full list is in the recipe's CREDITS.md.

Run it on your Spark.

Free and open, with every patch and every number published.

Get the free recipe See all recipes

Nemotron 3.5 Bolt on one DGX Spark, with TensorFold.

Install this model