The frontier of local intelligence: free, open-source recipes for NVIDIA DGX Sparks and RTX GPUs.
Each recipe runs on my own Sparks, and the benchmarks, patches and launch scripts are all published on GitHub. Clone one and serve a frontier-class model on your own hardware.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve DeepSeek V4.1 Flash 1017 (deepseek-ai/DeepSeek-V4.1-Flash-1017, FP8 / MXFP4 as published, no re-quantization) on TensorFold's Zig engine, tensor-parallel across four DGX Sparks, with DSpark drafting from the checkpoint's own draft blocks using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/DeepSeek-V4.1-Flash-1017-4x-DGX-Sparks-TensorFold
This recipe needs four DGX Sparks (GB10, 128 GB each) linked by ConnectX-7 either on one RoCE switch (./start.sh) or as a switchless ring, spark1-spark2-spark3-spark4-spark1 with one QSFP cable between neighbours (./start-switchless.sh, with the workers listed in the ring's order); Docker with the NVIDIA runtime on every Spark; key-based ssh from spark1 (the head) to the other three; about 100 GB of free memory on each Spark, so nothing else large can run on them. You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (about 510 GB for the checkpoint plus about 320 GB for the four weight files on spark1, under the Hugging Face cache; the other three Sparks read the checkpoint and their weight file from spark1 over NFS (read-only export)); and anything already using GPU memory or port 8888. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. Copy scripts/local.sh.example to scripts/local.sh and set WORKER, WORKER2 and WORKER3 to the other three Sparks (on a ring, in the ring's order), and NFS_PATH if my export puts the cache somewhere other than /huggingface. The NFS export of the Hugging Face cache needs sudo on spark1, so ask me first. Keep the defaults (16 requests at once, DSpark drafts on, CUDA graphs on); leave the opt-in Engram packing, RANK_LOCAL and the TF_DS_* switches at their defaults. Do not set NVFP4=1, DERISKED=1 or ABLIT=1 unless I ask for them: NVFP4=1 and DERISKED=1 each download another checkpoint (NVFP4=1 about 430 GB, with about 750 GB free on spark1 and 110 GB on each worker), and ABLIT=1 grafts a gated abliteration overlay and needs my Hugging Face login or HF_TOKEN. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh on a switch, or ./start-switchless.sh on a ring, on spark1. The first run builds the image, downloads the checkpoint, writes the four weight files and the workers' NFS volumes, then starts the four ranks and runs a smoke test; later starts take about two and a half minutes. The API should come up on port 8888 with the model id DeepSeek-V4.1-Flash-1017. The model thinks by default, so give test requests enough max_tokens or send chat_template_kwargs enable_thinking false. ./stop.sh stops all four ranks. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve GLM 5.3 Flash (EXL3 4-bit, Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold) on TensorFold's native Zig engine, across two DGX Sparks, one rank on each using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/GLM-5.3-Flash-DGX-Sparks-1104
This recipe needs two DGX Sparks (GB10, 128 GB) cabled to each other on their ConnectX-7 ports, with IPv4 addresses on one private subnet, key-based ssh from the first Spark (the head) to the second (the worker), your login in the docker group on both, Docker with the NVIDIA Container Toolkit and BuildKit, and rdma-core on both. You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (about 40 GiB for the Docker image on each Spark, and about 180 GiB for the checkpoint on each Spark that holds it (both ranks read every shard; WORKER_WEIGHTS=nfs lets the worker mount the head's copy instead)); and anything already using GPU memory or port 8020. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. On the head, copy scripts/local.sh.example to scripts/local.sh and set WORKER=user@<worker address> in it, ideally the address on the direct link. The link, its RDMA devices and the RoCE v2 GID are found automatically; a second rail is probed before use. Keep the other defaults (PARALLEL=4, CONTEXT=auto, KV=fp8, q8 dense and head, DFlash2). The DFlash2 drafter is non-commercial only (DRAFTER=mtp avoids it). Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh on the head: it checks both Sparks, builds the image once (about 6 minutes) and sends it to the worker, fetches the checkpoint when missing, starts rank 1 then rank 0, and smoke-tests. DRY_RUN=1 ./start.sh prints the whole plan first. ./stop.sh stops both ranks. The API answers on PORT (8020 in scripts/local.sh.example) with the model id GLM-5.3-Flash. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark. Install and serve Qwen3.8 Flash Next (NVIDIA's FP4 QAD checkpoint by default, INT4-AutoRound optional, with MTP drafts) on TensorFold's Zig engine, on a single Spark using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold
This recipe needs one DGX Spark (or another GB10 system with 128 GB) with nothing else large on the GPU: the default window needs about 108 GiB free (MemAvailable) when the server starts with the default FP4 checkpoint, or about 93 GiB with CHECKPOINT=int4. You're running directly on that Spark.
Work through these steps in order:
1. Check my setup before changing anything. Report the GPU, driver and CUDA version; Docker and the NVIDIA container runtime; free disk space (about 160 GB free on a fresh machine: the checkpoint under ~/.cache/huggingface (about 96 GiB for the default FP4 one, about 122 GiB for INT4) plus the Docker image (scripts/prepare.sh checks both); keep at least 100 GB free); free memory; and anything already using GPU memory or port 8888. If something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the paths and options to match what you found in step 1. It needs Docker with the NVIDIA container runtime and my user in the docker group. There's no .env to create: every setting lives in scripts/config.sh and can be overridden from the environment or a .env file. Keep the defaults (CHECKPOINT=nvfp4, NVIDIA's FP4 QAD checkpoint; up to 5 requests at once, a 262,144-token window, FP8 KV cache, image and video input on); CHECKPOINT=int4 serves the faster INT4-AutoRound checkpoint instead, with lower general capability per its model card, so only switch if I ask. A request that doesn't fit the KV pool is refused, not queued. The API listens on every interface (HOST 0.0.0.0), so keep it on my local network. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh. The first run pulls the prebuilt engine image (or builds it) and downloads the checkpoint; later starts just load the weights. It runs a smoke test and prints "Qwen3.8-Flash-Next is now LIVE! on port 8888", with the model id Qwen3.8-Flash-Next. The model thinks before it answers, so give test requests enough max_tokens (a request without one gets 32,768). ./stop.sh stops it and frees the memory. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark. Install and serve Qwen3.8 27B (MLX 4-bit, with the DFlash2 drafter) on TensorFold, on a single Spark using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-27B-DGX-Spark-TensorFold
This recipe needs one DGX Spark (or another GB10 system with 128 GB unified memory); the startup estimate is 45 GiB at the defaults, and it pins up to 78 GiB of attention cache at startup, sized from the free memory (KV_POOL_GB=auto). You're running directly on that Spark.
Work through these steps in order:
1. Check my setup before changing anything. Report the GPU, driver and CUDA version; Docker and the NVIDIA container runtime; free disk space (about 60 GB free: about 19 GB for the checkpoints under ~/.cache/huggingface and 25-35 GB for the Docker image); free memory; and anything already using GPU memory or port 8888. If something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the paths and options to match what you found in step 1. It needs Docker with the NVIDIA container runtime and my user in the docker group. There's no .env to create: every setting lives in scripts/config.sh and can be overridden from the environment or a .env file next to start.sh. Keep the defaults (8 streams of 262,144 tokens, FP8 KV cache, the pinned KV pool sized automatically). Only if I ask for a 1M-token window, set YARN_FACTOR=4. The API is on port 8888; keep it on my local network. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh. The first run pulls the prebuilt image (or builds it locally, a few minutes), downloads the checkpoints and compiles the CUDA kernels once; later starts take about a minute. It runs a smoke test and prints the endpoint, with the model id Qwen3.8-27B. The model thinks before it answers, so give test requests enough max_tokens. ./stop.sh stops it and frees the memory. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my Linux server with one RTX PRO 6000 GPU. Install and serve Qwen3.8 Flash Next (NVIDIA's FP4 QAD checkpoint, nvidia/Qwen3.8-Flash-Next-FP4-Local, with MTP drafts) on TensorFold's Zig engine, on one RTX PRO 6000 GPU using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-Flash-Next-RTX-PRO-6000-TernsorFold
This recipe needs a Linux x86_64 host with one 96 GB RTX PRO 6000 Blackwell and nothing else large on it (a start needs 96,845 MiB free on the card), a driver for CUDA 13, Docker with the NVIDIA container runtime and my user in the docker group; it was developed on 89 GB of RAM, and the prompt cache pins up to 16 GiB of it. You're running directly on that machine.
Work through these steps in order:
1. Check my setup before changing anything. Report my operating system; the GPU model, VRAM and driver version (nvidia-smi); Docker and the NVIDIA Container Toolkit if the recipe uses Docker; the Python version if it builds its own Python environment; free disk space (about 160 GB free, with the checkpoint (about 96 GiB) on an NVMe drive); free RAM; and anything already using the GPU or port 8888. If my GPU doesn't match what the recipe needs, or something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Follow the README's setup for my operating system and GPU. Settings come from scripts/config.sh and can be overridden from the environment or a .env file next to start.sh. Keep the defaults: NVIDIA's FP4 checkpoint at its pinned revision, 3 requests at once, a 262,144-token window, FP8 KV cache, MTP drafts, image and video input on. GPU picks the card by its nvidia-smi index: on a machine with two cards, take the one without the monitors. The published prefill numbers are at the card's default 600 W and the decode numbers at 400 W, and speed depends on the limit. The scripts never set the power limit, so check it with nvidia-smi, tell me what it is, and don't change it without asking me. YARN=1 serves one request at a time with a window of up to 1,000,000 tokens; it changes replies on short prompts too and has not been run on NVIDIA's FP4 yet, so don't set it unless I ask. Don't set ABLIT=1 unless I ask: it has never been started on this card, and the gated overlay needs an HF_TOKEN from my account after I accept its terms on Hugging Face, so ask me for it and never print it. DRY_RUN=1 ./start.sh shows what a start would do and changes nothing. The API listens on every interface (HOST 0.0.0.0), so keep it on my local network. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh. The first run builds the engine (about 5 minutes) and downloads the checkpoint; later starts take about 2.5 minutes. It runs a smoke test and prints "Qwen3.8-Flash-Next is now LIVE! on port 8888", with the model id Qwen3.8-Flash-Next. The model thinks before it answers, so give test requests enough max_tokens. The README's decode numbers are prose at temperature 0.7 with thinking on, so compare like with like. ./stop.sh stops it; docker logs -f qwen38-flash-next-tf shows the logs. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my Linux server with two RTX PRO 6000 GPUs. Install and serve GLM 5.3 Flash (EXL3 4-bit, Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold) on TensorFold's native Zig engine, across two RTX PRO 6000 GPUs in one Linux host using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/GLM-5.3-Flash-2x-RTX-PRO-6000-TensorFold
This recipe needs one Linux x86_64 host with two 96 GB RTX PRO 6000 Blackwell GPUs and a driver for CUDA 13, Docker with the NVIDIA Container Toolkit and BuildKit (my user in the docker group), curl, python3, flock, tar, gzip, sha256sum and nvidia-smi, the hf command (huggingface_hub[cli]) for the downloads, and at least 88,000 MiB free on each GPU at launch. You're running directly on that machine.
Work through these steps in order:
1. Check my setup before changing anything. Report my operating system; the GPU model, VRAM and driver version (nvidia-smi); Docker and the NVIDIA Container Toolkit if the recipe uses Docker; the Python version if it builds its own Python environment; free disk space (about 230 GiB: 180 GiB free for the checkpoint (about 164 GiB), 10 GiB for the DFlash2 drafter and 40 GiB for the image build); free RAM; and anything already using the GPU or port 8020. If my GPU doesn't match what the recipe needs, or something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Follow the README's setup for my operating system and GPU. Settings come from the defaults in scripts/config.sh, then .env, then scripts/local.sh (copy it from scripts/local.sh.example), then exported variables. Keep the defaults: the pinned 4 bpw checkpoint downloaded on the first start, 4 streams at once, CONTEXT=auto (it finds the largest window that fits, and may restart to do it), FP8 KV cache on shared pages, q8 dense layers and the DFlash2 drafter. If fewer than four streams come up, tell me before changing MEMORY_RESERVE_GIB (the README's settings table says 3.5 fits four streams on pages with this checkpoint). GPUS lists the GPUs in rank order and rank 0 is the heavier one: keep it on the GPU without a desktop (with the monitors on GPU 0, GPUS=1,0). The published numbers were measured with both cards capped at 400 W. The scripts never set the power limit, so check it with nvidia-smi, tell me if it differs, and don't change it without asking me. The DFlash2 drafter is non-commercial only: DRAFTER=mtp or DRAFTER=none avoids it. Don't set ABLIT=1 unless I ask for the gated Ablit weights: they need an HF_TOKEN from my account after I accept their terms on Hugging Face, so ask me for it and never print it. DRY_RUN=1 ./start.sh prints the whole plan without changing anything. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh with the GPUs idle. The first start builds the image (a few minutes with BuildKit's caches), downloads the checkpoint and the drafter (an interrupted download resumes on the next start), and has each rank hash every checkpoint shard once, which is then cached; WAIT_TIMEOUT (2400 s) covers the hashing and loading. It smoke-tests the API and prints "GLM-5.3-Flash-EXL3 is now LIVE! on port 8020": the API is http://127.0.0.1:8020/v1 with the model id GLM-5.3-Flash. Thinking is off by default; a request turns it on with chat_template_kwargs.enable_thinking. The README's decode numbers are prose at temperature 0.7 with thinking on, so compare like with like. ./stop.sh stops it and archives the logs; ./start.sh restart applies changed settings. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve DeepSeek V4 Flash (Vision-Exp) with DSpark speculative decoding using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark
This recipe needs 2 Sparks connected over ConnectX-7 (a third Spark is optional and uses its own launcher). You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (about 157 GiB for the checkpoint on the head node, plus a second copy on the worker unless you turn on the README's NFS option (DSPARK_WORKER_HF_NFS=1)); and anything already using GPU memory or port 8888. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. The example file is .env.dspark.example: set WORKER_HOST, MASTER_ADDR, NCCL_IB_HCA, NCCL_SOCKET_IFNAME (and the matching TP_/GLOO_ interface names), VLLM_HOST_IP, WORKER_VLLM_HOST_IP and HF_CACHE for my cluster; the example addresses are the author's. Leave the serving knobs at their defaults and never set DSPARK_MODEL or GPU_MEMORY_UTILIZATION by hand. For 3 Sparks use ./start-tp3.sh rather than editing .env. The README says to disable earlyoom on both nodes, or it can kill vLLM under long contexts; ask me before doing that. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Pull the pinned image (ghcr.io/anemll/dspark-vllm-gx10:0.1.1) on both nodes, run ./prepare-dspark-model-cache.sh --official, then ./start-deepseek-v4-flash-dspark.sh, which starts the worker and then the head. The API should come up on port 8888 with the model id deepseek-v4-flash-vision-exp and a max_model_len of 1048576; ./smoke-deepseek-v4-flash-dspark.sh checks it. The API listens on every interface by default, so keep it on my local network. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve GLM 5.3 Flash (EXL3 4-bit) using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks
This recipe needs 3 or 4 Sparks connected over ConnectX-7. You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (the weights are about 164 GiB on the head node; the README explains how the workers get them (NFS share or rsync)); and anything already using GPU memory or port 8888. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. Pick the launcher for my node count: ./start-tp3.sh for 3 Sparks and ./start-tp4.sh for 4. Don't change TP in .env. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. The API should come up on port 8888 with the model id GLM-5.3-Flash-EXL3. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark. Install and serve Qwen3.8 Flash Next (NVFP4) on a single Spark using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark
This recipe needs one DGX Spark. You're running directly on that Spark.
Work through these steps in order:
1. Check my setup before changing anything. Report the GPU, driver and CUDA version; Docker and the NVIDIA container runtime; free disk space (about 130 GiB: 99 GiB for the checkpoint plus the ~27 GiB PLE table built on first launch); free memory; and anything already using GPU memory or port 8888. If something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the paths and options to match what you found in step 1. Keep the shipped memory budget (HOST_RESERVE_GIB and the KV target): the README's safety rules explain why the host must keep about 10 GiB free under load, and running out of unified memory hangs the machine. Never set PLE_OFFLOAD=false, and don't raise the YaRN ceiling past what the README allows. The API listens on every network interface by default, so set an API_KEY in .env (or BIND=127.0.0.1) before starting. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Fetch the checkpoint with ./download.sh, then run ./start.sh. It takes about 10-12 minutes to become healthy on port 8888, with the model id qwen3.8-flash-next. This build returns its reasoning before the answer, so give test requests at least 400 max_tokens. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my PC with an NVIDIA GPU. Install and serve Qwen3.8 27B (EXL3) with the one-click kit using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install
This recipe needs one NVIDIA GPU with 16–32 GB of VRAM (12 GB works with the smallest quant) and driver 570 or newer, on Windows or Linux. You're running directly on that machine.
Work through these steps in order:
1. Check my setup before changing anything. Report my operating system; the GPU model, VRAM and driver version (nvidia-smi); Docker and the NVIDIA Container Toolkit if the recipe uses Docker; the Python version if it builds its own Python environment; free disk space (9.7–22.9 GB for the model depending on the quant it picks, plus several GB for the Python environment); free RAM; and anything already using the GPU or port 8888. If my GPU doesn't match what the recipe needs, or something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Follow the README's setup for my operating system and GPU. Run the kit's setup: windows\START-HERE.bat on Windows or ./linux/setup.sh on Linux. It creates .env and asks which quant to use; take the recommended option for my VRAM unless I say otherwise. It needs Python 3.11+ (64-bit); Node 22.19+ is only needed for the chat UI. Nothing needs administrator rights. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Start it with windows\start.bat or ./linux/start.sh. The OpenAI-compatible API comes up on http://localhost:8888/v1 and the chat UI on http://127.0.0.1:3080/. If loading fails with "Insufficient VRAM", lower CONTEXT_SIZE or GPU_MEM_GB in .env as the README describes. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark. Install and serve Qwen3.8 27B (NVFP4) on SGLang using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-27B-SGLang-DGX-Spark
This recipe needs one DGX Spark. You're running directly on that Spark.
Work through these steps in order:
1. Check my setup before changing anything. Report the GPU, driver and CUDA version; Docker and the NVIDIA container runtime; free disk space (about 24 GB for the NVFP4 checkpoint plus about 2.7 GB for the DSpark drafter, downloaded into ./.cache/huggingface on first start); free memory; and anything already using GPU memory or port 8888. If something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the paths and options to match what you found in step 1. Copy .env.sample to .env: it ships 262K context, YaRN off and 10 concurrent requests. Keep those defaults unless I ask for more context (YaRN up to 1M, as the README explains). If I have a Hugging Face token, put it in HF_TOKEN. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Start it with ./start-dspark.sh, the README's default for chat, agents and code (./start-dflash.sh is the alternative that's better for long-form writing). The API comes up on port 8888; check it with curl http://127.0.0.1:8888/v1/models. Thinking mode is on by default. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve Qwen3.8 Flash Next (NVFP4) using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-Flash-Next-Dual-DGX-Sparks
This recipe needs exactly 2 Sparks connected over ConnectX-7. You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (about 126 GiB free on each node for the checkpoint (the worker needs less if you use the README's NFS weight sharing)); and anything already using GPU memory or port 8888. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. Set the network interface and IB_HCA values for each node exactly, using the worker overrides if the two nodes are cabled differently. Edit .env rather than passing environment variables, because .env wins over most of them. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Download with ./download.sh, run ./check-weights.sh --verify, then ./start.sh --no-download. The API should come up on port 8888, and the README shows how to confirm the KV cache size in the logs. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve DeepSeek V4.1 Flash 1017 (Mia-AiLab/DeepSeek-V4.1-Flash-1017-EXL3-EAQ-v1-2.9-bpw, EXL3 2.9 bpw; experimental, its model card has no perplexity or KL measurement) on TensorFold's Zig engine, across two DGX Sparks using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks
This recipe needs two DGX Sparks (or two GB10 systems with 128 GB) linked directly by a QSFP cable between their ConnectX-7 ports (an IPv4 address on each end in one private subnet, and the same GPU clock ceiling on both: nvidia-smi -lgc 0,2200), key-based ssh from the first Spark (the head) to the second (the worker), Docker with the NVIDIA container runtime and my user in the docker group on both, git, python3 and the hf CLI on the head. start.sh refuses to start unless about 109 GiB of memory is available on each Spark, so nothing else large can run on them. You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (about 505 GB on the head (the checkpoint 196 GiB, Engram's two tables about 203 GB, rank 0's weight file about 106 GB) plus about 25 GB for the Docker image; about 310 GB on the worker (Engram's tables and rank 1's weight file). Engram's tables must be on local disk on both Sparks: read over NFS, prefill drops to a third); and anything already using GPU memory or port 8888. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. On the head, copy scripts/local.sh.example to scripts/local.sh and set WORKER=user@<worker address> (and FABRIC_PEER to the worker's CX7 address if that is not the link's address). The worker mounts the head's Hugging Face cache read-only over NFS, which needs sudo on the head once (apt install nfs-kernel-server, an /etc/exports line, exportfs -ra), so ask me first. The README's clone command names a repository ending in -TensorFold; clone MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks instead. Keep the defaults (4 requests at once, DSpark drafts on, images on). The default window is 1,048,576 tokens, but a head that also runs a desktop is left with only 3 GiB free at the worst point, so use CONTEXT=524288 if mine does. Never lower the engine's memory reserve: running out of memory freezes a Spark. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh on the head (DRY_RUN=1 ./start.sh first prints what it would do). The first run sets up both Sparks (two Zig builds of about 6 minutes, the checkpoint and Engram's tables downloaded on the head, both ranks' weight files written); later starts load about 98 GiB of weights on each Spark. It runs a smoke test through both ranks and prints "DeepSeek-V4.1-Flash-1017-EXL3 is now LIVE! on port 8888", with the model id DeepSeek-V4.1-Flash-1017-EXL3. The model thinks before it answers, so give test requests enough max_tokens. ./stop.sh stops both ranks. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve DeepSeek V4.1 Flash (EXL3 2.9 bpw) using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-EXL3-2x-DGX-Sparks
This recipe needs exactly 2 Sparks connected over ConnectX-7. You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (about 387 GiB of downloads on the head node (197 GiB of EXL3 weights plus 190 GiB of Engram source shards); the worker reads them over the NFS share); and anything already using GPU memory or port 8888. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. The README's quick start assumes the head is 10.0.0.1; use my addresses. Keep the shipped memory defaults (KV pool, MAX_MODEL_LEN, batch sizes) and don't pass --kv-cache-dtype. If a memory setting has to change, change one at a time and check MemAvailable after a long prompt, as the README describes. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh on the head: it downloads the weights (it needs the Hugging Face CLI, so ask me before installing it), pulls the image, sets up the NFS share and serves on port 8888 with the model id DeepSeek-v4.1-Flash-EXL3. Thinking is on by default, so pass chat_template_kwargs as the README shows when you test. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my PC with an NVIDIA GPU. Install and serve Qwen3.8 27B (NVFP4) on vLLM using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-27B-NVFP4-RTX-5090
This recipe needs an NVIDIA RTX 5090 (32 GB, sm_120) on Linux. You're running directly on that machine.
Work through these steps in order:
1. Check my setup before changing anything. Report my operating system; the GPU model, VRAM and driver version (nvidia-smi); Docker and the NVIDIA Container Toolkit if the recipe uses Docker; the Python version if it builds its own Python environment; free disk space (about 22 GiB for the model weights plus about 8 GiB for the Python environment); free RAM; and anything already using the GPU or port 8888. If my GPU doesn't match what the recipe needs, or something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Follow the README's setup for my operating system and GPU. There's no .env to create: settings are environment variables with sensible defaults (PORT, MODEL_DIR, GPU_UTIL). It needs curl, python3, nvidia-smi and gcc. Keep concurrency at 1 unless I ask; the README explains why MTP with this KV cache can't batch. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh: it creates the venv, installs the pinned vLLM, applies the PR #40914 patch, downloads the model and serves on port 8888 with the model id qwen38-nvfp4, then warms up. It binds to 0.0.0.0, so keep it on my local network. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my machine. Install and serve Laguna S 2.1 (NVFP4) with DFlash speculative decoding using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Laguna-S-2.1-DGX-Spark-RTX-6000-PRO
This recipe needs one DGX Spark, or an RTX PRO 6000 (96 GB) with the README's RTX settings. You're running directly on the machine that will serve it; the recipe supports 1 Spark or RTX PRO 6000.
Work through these steps in order:
1. Check my setup before changing anything. Report which of the supported machines this is; the GPU, VRAM (or unified memory) and driver/CUDA version; Docker and the NVIDIA container runtime or the Python version, whichever the recipe uses; free disk space (about 150 GB free for the model cache (the ~71 GB model plus its draft model)); free RAM; and anything already using the GPU or port 8888. If my hardware isn't one the recipe supports, or something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Follow the README's setup, including its per-platform settings for my hardware. If the model is gated, I need a Hugging Face token with access to poolside/Laguna-S-2.1-NVFP4 in HF_TOKEN. Keep --max-num-seqs at 4 (DFlash crashes at the default), don't add --linear-backend flashinfer_b12x, and don't put min_p in the generation config. On an RTX PRO 6000, use the README's RTX column: CUTE_DSL_ARCH for sm_120 and --gpu-memory-utilization 0.78 to start. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Download both models with ./start.sh --download-only, then run ./start.sh. The first start takes about 15 minutes (weight loading, kernel compilation, CUDA graphs), so follow the logs. The API comes up on port 8888 with the model id poolside/Laguna-S-2.1-NVFP4. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve GLM-5.3, Z.ai's full model (EXL3 2.75-bit, Mia-AiLab/GLM-5.3-EXL3-2.75bpw-TensorFold), on TensorFold across three DGX Sparks using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/GLM-5.3-EXL3-3x-DGX-Sparks-TensorFold
This recipe needs three DGX Sparks (GB10, 128 GB each) joined by a triangle of direct ConnectX-7 cables (one QSFP cable per pair of Sparks, each cable its own subnet, RoCE v2), one network all three share for ssh, key-based ssh from the head (rank 0) to both workers, Docker with the NVIDIA container runtime on each, and nothing else large on their GPUs (each rank holds about 86 GiB of weights). You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (on the head, about 273 GiB for the checkpoint and 2.4 GiB for the DSpark drafter in its Hugging Face cache, with 100 GB left free; the workers keep no copy of the weights (they read them from the head over a read-only NFS export), only the image (about 25 GB)); and anything already using GPU memory or port 8888. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. On the head, export ~/.cache/huggingface read-only over NFS to both workers' cable subnets as the README's "Weights over NFS" section shows (this needs sudo on the head, so ask me first). Copy scripts/local.sh.example to scripts/local.sh and set WORKER=user@<rank 1> and WORKER2=user@<rank 2> in it (plus FABRIC_PEER / FABRIC_PEER2 if a worker is reached over another network than its cable). Keep the defaults in scripts/config.sh (CP=1, the 499,712-token window, fp4 KV cache, the DSpark drafter). Run DRY_RUN=1 ./start.sh first: it prints the memory plan and every rank's command without changing anything. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh on the head. The first run sets up all three Sparks (image, checkpoint, NFS mounts, kernels); later starts take 3 to 5 minutes. It runs a smoke test through all three ranks and prints "GLM-5.3-EXL3 is now LIVE! on port 8888", with the model id GLM-5.3-EXL3. The model thinks before it answers, so give test requests enough max_tokens. ./stop.sh stops all three ranks. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark. Install and serve Unsloth Qwen3.6 35B-A3B (NVFP4) using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Unsloth-Qwen3.6-35b-NVFP4-DGX-Spark
This recipe needs one DGX Spark. You're running directly on that Spark.
Work through these steps in order:
1. Check my setup before changing anything. Report the GPU, driver and CUDA version; Docker and the NVIDIA container runtime; free disk space (about 50 GB for the weights and caches); free memory; and anything already using GPU memory or port 8888. If something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the paths and options to match what you found in step 1. There's no .env to create: start.sh has sensible defaults. If I have a Hugging Face token, export it as HF_TOKEN before starting. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh: it pulls the ghcr.io/miaai-lab/mia-vllm-gb10-linear-b12x image, downloads the model if needed and serves on port 8888 with the model id unsloth/Qwen3.6-35B-A3B-NVFP4. The first load takes several minutes. It listens on every interface, so keep it on my local network. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my PC with an NVIDIA GPU. Install and serve Qwen3.8 27B (NVFP4) on SGLang with DFlash 2 using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-27B-RTX-6000-PRO-SGLang-DSpark
This recipe needs an NVIDIA RTX PRO 6000 with 96 GB (Blackwell, SM120); the recipe needs the full 96 GB. You're running directly on that machine.
Work through these steps in order:
1. Check my setup before changing anything. Report my operating system; the GPU model, VRAM and driver version (nvidia-smi); Docker and the NVIDIA Container Toolkit if the recipe uses Docker; the Python version if it builds its own Python environment; free disk space (about 28 GB: the ~24 GB NVFP4 checkpoint plus the ~3.8 GB DFlash 2 drafter, pulled into ./.cache/huggingface on first start); free RAM; and anything already using the GPU or port 8888. If my GPU doesn't match what the recipe needs, or something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Follow the README's setup for my operating system and GPU. It runs in Docker with the NVIDIA Container Toolkit (docker run --gpus all must work). Keep the defaults: full 256K context, 8 concurrent requests and FP8 KV. If I have a Hugging Face token, put it in HF_TOKEN. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh: it pulls the weights on first run and waits until the API is ready on port 8888. Check it with curl http://127.0.0.1:8888/v1/models. Thinking mode is on by default. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve Qwen3.8 Flash Next (NVIDIA's NVFP4 checkpoint by default) on TensorFold's Zig engine, across two DGX Sparks using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-Flash-Dual-DGX-Sparks-TensorFold
This recipe needs two DGX Sparks (or two GB10 systems with 128 GB) linked by a QSFP cable between their ConnectX-7 ports, each with an IPv4 address in one private subnet, and key-based ssh from the first Spark (the head) to the second (the worker). Each needs about 102 GiB free memory at start for the default 1,048,576-token window. You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (about 150 GB on each Spark: the checkpoint (about 133 GB) under ~/.cache/huggingface and the Docker image (scripts/prepare.sh checks both); with WORKER_WEIGHTS=nfs the worker needs only the image); and anything already using GPU memory or port 8888. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. It needs Docker with the NVIDIA container runtime, my user in the docker group, and rsync, on both Sparks. Copy scripts/local.sh.example to scripts/local.sh and set WORKER to the worker's ssh target (and FABRIC_PEER to its ConnectX-7 address if WORKER is on another network). The example uses WORKER_WEIGHTS=nfs, which needs the head to export its Hugging Face cache to the worker once over NFS (this needs sudo, so ask me first); WORKER_WEIGHTS=copy copies the checkpoint to the worker instead. Keep the defaults (QUANT=nvfp4, 16 requests at once, FP8 KV cache, 1,048,576-token window); QUANT=int4ar is the faster checkpoint, with lower general capability per its model card. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh on the head. The first run sets up both Sparks (the prebuilt image on each, the checkpoint downloaded on the head); later starts take about two minutes. It runs a smoke test through both ranks and prints "Qwen3.8-Flash-Next is now LIVE! on port 8888", with the model id Qwen3.8-Flash-Next. The model thinks before it answers, so give test requests enough max_tokens. ./stop.sh stops both ranks. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve MiMo V2.6 Flash using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/MiMo-V2.6-Flash-2x-DGX-Sparks
This recipe needs exactly 2 Sparks connected over ConnectX-7 (it cannot run on one). You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (about 171 GiB free on the head node; the worker reads the checkpoint over NFS); and anything already using GPU memory or port 8888. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. The README's quick start contains the author's own paths and 10.0.0.x addresses; replace them with mine. Use the SGLang base image the README pins, or newer: older engines expand the MXFP4 experts and can take a node down. Keep EAGLE MTP as the drafter unless I say the server is mainly for code, where the README recommends DFlash. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh doctor and fix anything it flags, then download, build, share and serve as the README describes. The API should come up on port 8888 with the model id MiMo-v2.6-Flash. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my machine. Install and serve NVIDIA Nemotron 3.5 Lightning 30B-A3B (NVFP4) with DSpark using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Nemotron3.5-Lightning-DGX-Spark-RTX-5090-6000-PRO
This recipe needs one DGX Spark, an RTX 5090 (32 GB) or an RTX 6000-class card. You're running directly on the machine that will serve it; the recipe supports 1 Spark, RTX 5090 or RTX PRO 6000.
Work through these steps in order:
1. Check my setup before changing anything. Report which of the supported machines this is; the GPU, VRAM (or unified memory) and driver/CUDA version; Docker and the NVIDIA container runtime or the Python version, whichever the recipe uses; free disk space (at least 30 GiB for the Docker image and both weight sets); free RAM; and anything already using the GPU or port 8888. If my hardware isn't one the recipe supports, or something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Follow the README's setup, including its per-platform settings for my hardware. Settings are environment variables set before ./start.sh. The default MEM_FRACTION_STATIC=0.78 is for a DGX Spark only; on other cards use the README's GPU Memory Quick Reference (0.30–0.45 on an RTX 5090, 0.60–0.70 on an RTX PRO 6000 Blackwell) and set CUDA_GRAPH_MAX_BS_DECODE=2 on the 5090. The script requires at least 80 GiB of host RAM; if I have less, tell me before changing anything. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh: it checks the prerequisites, downloads the model and DSpark draft, pulls lmsysorg/sglang:dev-nemotron3-5-lightning and serves on port 8888. It binds to all interfaces by default, so keep it on my local network. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark. Install and serve Ornith 1.5 122B (EXL3 4-bit, with the DFlash2 drafter) on TensorFold's Zig engine, on a single Spark using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Ornith-1.5-122B-TensorFold
This recipe needs one DGX Spark (or another GB10 system with 128 GB unified memory) with nothing else large on the GPU: about 92 GiB of MemAvailable when the server starts (the model and scratch take about 69 GiB on the device). You're running directly on that Spark.
Work through these steps in order:
1. Check my setup before changing anything. Report the GPU, driver and CUDA version; Docker and the NVIDIA container runtime; free disk space (about 112 GB free on a fresh machine: 62 GiB for the checkpoint and 2 GB for the drafter under ~/.cache/huggingface, plus the Docker image (scripts/prepare.sh checks both)); free memory; and anything already using GPU memory or port 8888. If something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the paths and options to match what you found in step 1. It needs Docker with the NVIDIA container runtime and my user in the docker group. There's no .env to create: every setting lives in scripts/config.sh and can be overridden from the environment or a .env file next to start.sh. Keep the defaults (4 requests at once, a 262,144-token window, FP8 KV cache, the DFlash2 drafter on, image input on). This recipe needs the patched TensorFold that ships in its patches/ folder; start.sh applies it. Keep at least 10 GiB of memory free under load: running out of unified memory freezes the Spark. The API listens on every interface (HOST 0.0.0.0), so keep it on my local network. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh. The first run builds the engine (it tries a prebuilt image first), downloads the checkpoint and the drafter; later starts load in 80-95 seconds. It runs a smoke test and prints LIVE, with the model id Ornith-1.5-122B on port 8888. The model thinks before it answers, so give test requests enough max_tokens. ./stop.sh stops it. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark. Install and serve NVIDIA Nemotron 3.5 Bolt 74B-A9B (NVIDIA's 24GB checkpoint, with the LiLiCorr drafter) on TensorFold's Zig engine, on a single Spark using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Nemotron-3.5-Bolt-TensorFold-DGX-Spark
This recipe needs one DGX Spark (or another GB10 system with 128 GB): the default 16 streams need about 55 GiB free (MemAvailable) when the server starts, and loading takes about 51 GB, so the rest of the Spark stays free for other work. You're running directly on that Spark.
Work through these steps in order:
1. Check my setup before changing anything. Report the GPU, driver and CUDA version; Docker and the NVIDIA container runtime; free disk space (about 23 GiB for the checkpoint under ~/.cache/huggingface plus the Docker image (scripts/prepare.sh checks both)); free memory; and anything already using GPU memory or port 8888. If something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the paths and options to match what you found in step 1. It needs Docker with the NVIDIA container runtime and my user in the docker group. There's no .env to create: every setting lives in scripts/config.sh and can be overridden from the environment or a .env file. Keep the defaults (up to 16 streams at once, a 262,144-token window each, bf16 KV cache, image and video input on, the LiLiCorr drafter on). Keep at least 10 GiB of memory free under load: running out of unified memory freezes the Spark. The API listens on every interface (HOST 0.0.0.0), so keep it on my local network. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh. No prebuilt image is published yet, so the first run builds TensorFold with the recipe's patches and engine, then downloads the checkpoint (23 GiB) and the drafter (1.3 GB); later starts reuse them and load in about 45 seconds. It runs a smoke test and prints LIVE, with the model id Nemotron-3.5-Bolt on port 8888. The model thinks before it answers, so give test requests enough max_tokens (a request without one gets 32,768). ./stop.sh stops it. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark. Install and serve Ling 3.0 Flash (INT4) with the DSpark draft using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Ling-3.0-Flash-SGLang-DSpark-DGX-Spark
This recipe needs one DGX Spark. You're running directly on that Spark.
Work through these steps in order:
1. Check my setup before changing anything. Report the GPU, driver and CUDA version; Docker and the NVIDIA container runtime; free disk space (25–30 GB for the weights and the runtime image (the script wants 40 GiB or more free)); free memory; and anything already using GPU memory or port 8888. If something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the paths and options to match what you found in step 1. Settings are environment variables with safe defaults (SPEC_ALGO=dspark, 256k context, 6 concurrent requests), so there's no .env to edit. Keep the defaults unless I ask; the README's MEM_FRACTION_STATIC table shows the memory trade-off. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Download with ./start.sh --download-only, then run ./start.sh. It pulls the public lmsysorg/sglang:dev-Ling-3.0-flash image; don't try to clone inclusionAI's private sglang repo. The API comes up on port 8888 (the model name can be "auto"), and thinking is on by default. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my machine. Install and serve Meta Muse Glimmer 30B (NVFP4) with the DFlash drafter using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Muse-Glimmer-30B-DGX-Spark-RTX-5090-6000-PRO
This recipe needs one DGX Spark, an RTX 5090 (32 GB) or an RTX PRO 6000 (96 GB). You're running directly on the machine that will serve it; the recipe supports 1 Spark, RTX 5090 or RTX PRO 6000.
Work through these steps in order:
1. Check my setup before changing anything. Report which of the supported machines this is; the GPU, VRAM (or unified memory) and driver/CUDA version; Docker and the NVIDIA container runtime or the Python version, whichever the recipe uses; free disk space (about 19 GB of weights plus 5.1 GB for the DFlash drafter and about 10 GB for the Python environment); free RAM; and anything already using the GPU or port 8888. If my hardware isn't one the recipe supports, or something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Follow the README's setup, including its per-platform settings for my hardware. It needs Python 3.12 (not 3.13 or newer) and CUDA 13 (driver 570+). Set GPU_MEM_UTIL for my hardware from the README's platform table: 0.42 on a DGX Spark (the default), about 0.46 on an RTX PRO 6000, and about 0.93–0.95 on a dedicated RTX 5090. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh: it builds the venv, installs the pinned vLLM fork with precompiled kernels, downloads the weights and drafter, and serves on port 8888 with the model id muse-glimmer-30b. Stop it with ./stop.sh, because orphaned engine processes keep memory pinned. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my Linux server with four RTX PRO 6000 GPUs. Install and serve GLM 5.3 Flash (EXL3 4-bit, Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold) on TensorFold v0.6.5, across four RTX PRO 6000 GPUs in one Linux host using the Aevonix Research recipe, made with Mia's AI Lab:
https://github.com/Aevonix/GLM-5.3-Flash-EXL3-4x-RTX-PRO-6000-TensorFold
This recipe needs one Linux x86_64 host with four 96 GB RTX PRO 6000 Blackwell GPUs and CUDA peer access between every pair, Docker with the NVIDIA Container Toolkit (my user in the docker group), Bash, Python 3, Git, GNU patch, curl, gzip and flock, and at least 90,000 MiB free on each GPU at launch. You're running directly on that machine.
Work through these steps in order:
1. Check my setup before changing anything. Report my operating system; the GPU model, VRAM and driver version (nvidia-smi); Docker and the NVIDIA Container Toolkit if the recipe uses Docker; the Python version if it builds its own Python environment; free disk space (about 240 GiB: 180 GiB for the checkpoint, 10 GiB for the DFlash2 drafter, 40 GiB for the image and build, and 10 GiB for kernels); free RAM; and anything already using the GPU or port 8020. If my GPU doesn't match what the recipe needs, or something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Follow the README's setup for my operating system and GPU. Settings come from the environment, then scripts/local.sh, then .env, then the defaults in scripts/config.sh. Keep the defaults: 40 concurrent requests, the 1,048,576-token window, FP8 dense layers, FP8 KV cache and the DFlash2 drafter. The published numbers were measured with each GPU at 250 W; the script warns when the power limit differs and never changes it, so don't change it either without asking me. The DFlash2 drafter is non-commercial only: DRAFTER=mtp ./start.sh on the first run avoids it (one request at a time). DRY_RUN=1 ./start.sh previews every step without Docker or GPUs. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh with the GPUs idle. The first run checks the host, builds TensorFold with all 86 patches, downloads the checkpoint and drafter, compiles kernels and autotunes the launch table; later runs reuse what is ready. It sends a smoke request and prints "GLM-5.3-Flash-EXL3 is now LIVE! on port 8020": the API is http://127.0.0.1:8020/v1 with the model id glm-5.3-flash. Thinking is off by default. ./stop.sh stops it and archives the logs; ./start.sh restart restarts it. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my Linux server with two RTX PRO 6000 GPUs. Install and serve GLM 5.3 Flash (EXL3 4-bit, Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold) on TensorFold v0.6.5, across two RTX PRO 6000 GPUs in one Linux host using the Aevonix Research recipe, made with Mia's AI Lab:
https://github.com/Aevonix/GLM-5.3-Flash-EXL3-2x-RTX-PRO-6000-TensorFold
This recipe needs one Linux x86_64 host with two 96 GB RTX PRO 6000 Blackwell GPUs and CUDA peer access between them, Docker with the NVIDIA Container Toolkit (my user in the docker group), Bash, Python 3, Git, GNU patch, curl, gzip, GNU coreutils and flock, and at least 90,000 MiB free on each GPU at launch. You're running directly on that machine.
Work through these steps in order:
1. Check my setup before changing anything. Report my operating system; the GPU model, VRAM and driver version (nvidia-smi); Docker and the NVIDIA Container Toolkit if the recipe uses Docker; the Python version if it builds its own Python environment; free disk space (about 240 GiB: 180 GiB for the checkpoint, 10 GiB for the DFlash2 drafter, 40 GiB for the image and build, and 10 GiB for kernels); free RAM; and anything already using the GPU or port 8022. If my GPU doesn't match what the recipe needs, or something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Follow the README's setup for my operating system and GPU. Settings come from the environment, then scripts/local.sh, then .env, then the defaults in scripts/config.sh. Keep the defaults: GPUS=0,1, 4 concurrent requests, the 524,288-token window, 4-bit dense layers, FP8 KV cache and the DFlash2 drafter. The published numbers were measured with each GPU at 300 W; the script warns when the power limit differs and never changes it, so don't change it either without asking me. The DFlash2 drafter is non-commercial only: DRAFTER=mtp ./start.sh on the first run avoids it (one request at a time). DRY_RUN=1 ./start.sh previews every step without Docker or GPUs. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh with the GPUs idle. The first run checks the host, builds TensorFold with all 86 patches, downloads the checkpoint and drafter, compiles kernels and runs the one-time two-GPU launch-table autotune; later runs reuse what is ready. It sends a smoke request and prints "GLM-5.3-Flash-EXL3 is now LIVE! on port 8022": the API is http://127.0.0.1:8022/v1 with the model id glm-5.3-flash. Thinking is off by default. ./stop.sh stops it and archives the logs; ./start.sh restart restarts it. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
No recipe for that setup yet. More are on the way.
Decode speeds are single-stream figures published in each recipe's README (prose prompts wherever the README separates them); many were measured with sparkDash, my open-source benchmark dashboard. The meters run from 0 to 60 tok/s. GitHub numbers update every 15 minutes.
Lab tools
The dashboard behind every number.
For DGX Sparks & NVIDIA GPUs
sparkDash
A real-time dashboard for one or many DGX Sparks and NVIDIA GPU machines, and the benchmark behind the speeds on this page.
Monitor one or many DGX Sparks and NVIDIA GPU machines in one window
Live GPU, unified memory, storage and network metrics
Detects your local LLM server (vLLM, SGLang, llama.cpp and more) and shows live tok/s
Built-in decode and prefill benchmarks, the ones behind the numbers above
Prompt Showcase: stream up to 32 prompts side by side
You are installing sparkDash, the open-source monitoring and benchmark dashboard from Mia's AI Lab, for NVIDIA DGX Sparks and NVIDIA GPU machines:
https://github.com/MiaAI-Lab/sparkDash
Work through these steps in order:
1. Check my setup before changing anything. Report which machine this is (a DGX Spark or a Linux PC with an NVIDIA GPU); Docker with Compose; the NVIDIA driver (nvidia-smi); free disk space; and whether port 5555 is free. Ask me which other Sparks or GPU machines I want to monitor and their IPs, and check that this machine can SSH into each of them with a key. If something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the Quick start, Configuration and Security sections. Treat the security notes as hard rules.
3. Start it with Docker as the README describes (docker compose up --build -d). It listens on 127.0.0.1:5555 by default; keep it that way. If I want it from another computer, set up the SSH tunnel the README shows, or ask me before binding it to the network, which also requires setting SPARKDASH_TOKEN.
4. Mount my SSH key into the container as the README explains, so it can reach the other machines. Prefer keys over passwords, and never delete config/.secrets-key.
5. Open the dashboard, add each machine with the right role (head, worker, standalone, or a dedicated GPU host), and confirm that GPU, memory and network metrics are streaming. If an LLM server is running, check that the LLM card detects it and shows tok/s.
6. Finish with a summary: the dashboard URL, how to open it from my laptop, the machines you added, and the commands to update, stop and restart it.
Ground rules: ask me before installing system packages, using sudo, or changing SSH settings on any machine. Don't expose the dashboard to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
Tier list
Model tier list
Every model I've tried, cloud or local, ranked by how well it worked for me.
