The frontier of local intelligence: free, open-source recipes for NVIDIA DGX Sparks and RTX GPUs.
Each recipe runs on my own Sparks, and the benchmarks, patches and launch scripts are all published on GitHub. Clone one and serve a frontier-class model on your own hardware.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve GLM 5.3 Flash (EXL3 4-bit, Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold) on TensorFold, across two DGX Sparks (three is experimental: ./start-tp3.sh; four is experimental and lives on the repository's tp4 branch: git checkout tp4, then ./start-tp4-switchless.sh for a ring of direct cables or ./start-tp4.sh on a switch, following that branch's README) using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold
This recipe needs two DGX Sparks (or two GB10 systems with 128 GB) linked directly by a QSFP cable between their ConnectX-7 ports, with an IPv4 address on each end in one private subnet, key-based ssh from the first Spark (the head) to the second (the worker), Docker with the NVIDIA container runtime and rsync on both, and about 110 GiB free GPU memory on each when the server starts. You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (about 205 GB free on each Spark: about 176 GB for the checkpoint and 2.3 GB for the DFlash2 drafter under ~/.cache/huggingface, and about 25 GB for the Docker image; the head downloads them and copies them to the worker); and anything already using GPU memory or port 8888. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. On the head, copy scripts/local.sh.example to scripts/local.sh and set WORKER=user@<worker address> in it; if that address isn't on the ConnectX-7 link, also set FABRIC_PEER to the worker's CX7 address. Every other setting lives in scripts/config.sh; keep the defaults (4 requests, 1M window, FP8 KV cache, DFlash2). The worker needs no copy of the repository. Note the DFlash2 drafter is non-commercial only (DRAFTER=mtp avoids it). Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh on the head. The first run sets up both Sparks (image, checkpoint, kernels); later starts take 2 to 6 minutes. It runs a smoke test through both ranks and prints "GLM-5.3-Flash-EXL3 is now LIVE! on port 8888", with the model id GLM-5.3-Flash-EXL3. The model thinks before it answers, so give test requests enough max_tokens. ./stop.sh stops both ranks. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark. Install and serve Qwen3.8 Flash Next (azampatti's INT4-AutoRound checkpoint, with MTP drafts) on TensorFold's Zig engine, on a single Spark using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold
This recipe needs one DGX Spark (or another GB10 system with 128 GB) with nothing else large on the GPU: the default window needs about 92 GiB free (MemAvailable) when the server starts. You're running directly on that Spark.
Work through these steps in order:
1. Check my setup before changing anything. Report the GPU, driver and CUDA version; Docker and the NVIDIA container runtime; free disk space (about 160 GB free on a fresh machine: about 122 GiB for the checkpoint under ~/.cache/huggingface plus the Docker image (scripts/prepare.sh checks both); keep at least 100 GB free); free memory; and anything already using GPU memory or port 8888. If something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the paths and options to match what you found in step 1. It needs Docker with the NVIDIA container runtime and my user in the docker group. There's no .env to create: every setting lives in scripts/config.sh and can be overridden from the environment or a .env file. Keep the defaults (up to 8 requests at once, a 262,144-token window, FP8 KV cache, image and video input on); a request that doesn't fit the KV pool is refused, not queued. The API listens on every interface (HOST 0.0.0.0), so keep it on my local network. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh. The first run pulls the prebuilt engine image (or builds it) and downloads the checkpoint; later starts just load the weights. It runs a smoke test and prints "Qwen3.8-Flash-Next is now LIVE! on port 8888", with the model id Qwen3.8-Flash-Next. The model thinks before it answers, so give test requests enough max_tokens (a request without one gets 32,768). ./stop.sh stops it and frees the memory. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark. Install and serve Qwen3.8 27B (MLX 4-bit, with the DFlash2 drafter) on TensorFold, on a single Spark using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-27B-DGX-Spark-TensorFold
This recipe needs one DGX Spark (or another GB10 system with 128 GB unified memory); the startup estimate is 45 GiB at the defaults, and it pins up to 78 GiB of attention cache at startup, sized from the free memory (KV_POOL_GB=auto). You're running directly on that Spark.
Work through these steps in order:
1. Check my setup before changing anything. Report the GPU, driver and CUDA version; Docker and the NVIDIA container runtime; free disk space (about 60 GB free: about 19 GB for the checkpoints under ~/.cache/huggingface and 25-35 GB for the Docker image); free memory; and anything already using GPU memory or port 8888. If something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the paths and options to match what you found in step 1. It needs Docker with the NVIDIA container runtime and my user in the docker group. There's no .env to create: every setting lives in scripts/config.sh and can be overridden from the environment or a .env file next to start.sh. Keep the defaults (8 streams of 262,144 tokens, FP8 KV cache, the pinned KV pool sized automatically). Only if I ask for a 1M-token window, set YARN_FACTOR=4. The API is on port 8888; keep it on my local network. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh. The first run pulls the prebuilt image (or builds it locally, a few minutes), downloads the checkpoints and compiles the CUDA kernels once; later starts take about a minute. It runs a smoke test and prints the endpoint, with the model id Qwen3.8-27B. The model thinks before it answers, so give test requests enough max_tokens. ./stop.sh stops it and frees the memory. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve DeepSeek V4 Flash (Vision-Exp) with DSpark speculative decoding using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark
This recipe needs 2 Sparks connected over ConnectX-7 (a third Spark is optional and uses its own launcher). You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (about 157 GiB for the checkpoint on the head node, plus a second copy on the worker unless you turn on the README's NFS option (DSPARK_WORKER_HF_NFS=1)); and anything already using GPU memory or port 8888. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. The example file is .env.dspark.example: set WORKER_HOST, MASTER_ADDR, NCCL_IB_HCA, NCCL_SOCKET_IFNAME (and the matching TP_/GLOO_ interface names), VLLM_HOST_IP, WORKER_VLLM_HOST_IP and HF_CACHE for my cluster; the example addresses are the author's. Leave the serving knobs at their defaults and never set DSPARK_MODEL or GPU_MEMORY_UTILIZATION by hand. For 3 Sparks use ./start-tp3.sh rather than editing .env. The README says to disable earlyoom on both nodes, or it can kill vLLM under long contexts; ask me before doing that. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Pull the pinned image (ghcr.io/anemll/dspark-vllm-gx10:0.1.1) on both nodes, run ./prepare-dspark-model-cache.sh --official, then ./start-deepseek-v4-flash-dspark.sh, which starts the worker and then the head. The API should come up on port 8888 with the model id deepseek-v4-flash-vision-exp and a max_model_len of 1048576; ./smoke-deepseek-v4-flash-dspark.sh checks it. The API listens on every interface by default, so keep it on my local network. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve GLM 5.3 Flash (EXL3 4-bit) using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks
This recipe needs 3 or 4 Sparks connected over ConnectX-7. You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (the weights are about 164 GiB on the head node; the README explains how the workers get them (NFS share or rsync)); and anything already using GPU memory or port 8888. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. Pick the launcher for my node count: ./start-tp3.sh for 3 Sparks and ./start-tp4.sh for 4. Don't change TP in .env. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. The API should come up on port 8888 with the model id GLM-5.3-Flash-EXL3. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark. Install and serve Qwen3.8 Flash Next (NVFP4) on a single Spark using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark
This recipe needs one DGX Spark. You're running directly on that Spark.
Work through these steps in order:
1. Check my setup before changing anything. Report the GPU, driver and CUDA version; Docker and the NVIDIA container runtime; free disk space (about 130 GiB: 99 GiB for the checkpoint plus the ~27 GiB PLE table built on first launch); free memory; and anything already using GPU memory or port 8888. If something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the paths and options to match what you found in step 1. Keep the shipped memory budget (HOST_RESERVE_GIB and the KV target): the README's safety rules explain why the host must keep about 10 GiB free under load, and running out of unified memory hangs the machine. Never set PLE_OFFLOAD=false, and don't raise the YaRN ceiling past what the README allows. The API listens on every network interface by default, so set an API_KEY in .env (or BIND=127.0.0.1) before starting. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Fetch the checkpoint with ./download.sh, then run ./start.sh. It takes about 10-12 minutes to become healthy on port 8888, with the model id qwen3.8-flash-next. This build returns its reasoning before the answer, so give test requests at least 400 max_tokens. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my PC with an NVIDIA GPU. Install and serve Qwen3.8 27B (EXL3) with the one-click kit using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install
This recipe needs one NVIDIA GPU with 16–32 GB of VRAM (12 GB works with the smallest quant) and driver 570 or newer, on Windows or Linux. You're running directly on that machine.
Work through these steps in order:
1. Check my setup before changing anything. Report my operating system; the GPU model, VRAM and driver version (nvidia-smi); Docker and the NVIDIA Container Toolkit if the recipe uses Docker; the Python version if it builds its own Python environment; free disk space (9.7–22.9 GB for the model depending on the quant it picks, plus several GB for the Python environment); free RAM; and anything already using the GPU or port 8888. If my GPU doesn't match what the recipe needs, or something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Follow the README's setup for my operating system and GPU. Run the kit's setup: windows\START-HERE.bat on Windows or ./linux/setup.sh on Linux. It creates .env and asks which quant to use; take the recommended option for my VRAM unless I say otherwise. It needs Python 3.11+ (64-bit); Node 22.19+ is only needed for the chat UI. Nothing needs administrator rights. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Start it with windows\start.bat or ./linux/start.sh. The OpenAI-compatible API comes up on http://localhost:8888/v1 and the chat UI on http://127.0.0.1:3080/. If loading fails with "Insufficient VRAM", lower CONTEXT_SIZE or GPU_MEM_GB in .env as the README describes. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark. Install and serve Qwen3.8 27B (NVFP4) on SGLang using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-27B-SGLang-DGX-Spark
This recipe needs one DGX Spark. You're running directly on that Spark.
Work through these steps in order:
1. Check my setup before changing anything. Report the GPU, driver and CUDA version; Docker and the NVIDIA container runtime; free disk space (about 24 GB for the NVFP4 checkpoint plus about 2.7 GB for the DSpark drafter, downloaded into ./.cache/huggingface on first start); free memory; and anything already using GPU memory or port 8888. If something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the paths and options to match what you found in step 1. Copy .env.sample to .env: it ships 262K context, YaRN off and 10 concurrent requests. Keep those defaults unless I ask for more context (YaRN up to 1M, as the README explains). If I have a Hugging Face token, put it in HF_TOKEN. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Start it with ./start-dspark.sh, the README's default for chat, agents and code (./start-dflash.sh is the alternative that's better for long-form writing). The API comes up on port 8888; check it with curl http://127.0.0.1:8888/v1/models. Thinking mode is on by default. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve Qwen3.8 Flash Next (NVFP4) using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-Flash-Next-Dual-DGX-Sparks
This recipe needs exactly 2 Sparks connected over ConnectX-7. You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (about 126 GiB free on each node for the checkpoint (the worker needs less if you use the README's NFS weight sharing)); and anything already using GPU memory or port 8888. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. Set the network interface and IB_HCA values for each node exactly, using the worker overrides if the two nodes are cabled differently. Edit .env rather than passing environment variables, because .env wins over most of them. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Download with ./download.sh, run ./check-weights.sh --verify, then ./start.sh --no-download. The API should come up on port 8888, and the README shows how to confirm the KV cache size in the logs. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve DeepSeek V4.1 Flash (EXL3 2.9 bpw) using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-EXL3-2x-DGX-Sparks
This recipe needs exactly 2 Sparks connected over ConnectX-7. You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (about 387 GiB of downloads on the head node (197 GiB of EXL3 weights plus 190 GiB of Engram source shards); the worker reads them over the NFS share); and anything already using GPU memory or port 8888. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. The README's quick start assumes the head is 10.0.0.1; use my addresses. Keep the shipped memory defaults (KV pool, MAX_MODEL_LEN, batch sizes) and don't pass --kv-cache-dtype. If a memory setting has to change, change one at a time and check MemAvailable after a long prompt, as the README describes. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh on the head: it downloads the weights (it needs the Hugging Face CLI, so ask me before installing it), pulls the image, sets up the NFS share and serves on port 8888 with the model id DeepSeek-v4.1-Flash-EXL3. Thinking is on by default, so pass chat_template_kwargs as the README shows when you test. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve DeepSeek V4.1 Flash using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-DGX-Sparks
This recipe needs 3 or 4 Sparks connected over ConnectX-7 (it does not fit on 2). You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (the checkpoint is about 476 GiB on the head node, plus about 63 GiB of Engram tables on each node's NVMe); and anything already using GPU memory or port 8888. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. Use the 3-node profile (./start.sh with .env) for 3 Sparks, or ./start-tp4.sh with .env.tp4 for 4 Sparks. The README's quick start contains the author's own paths, hostnames and IPs; replace them with mine. Never set OFFLOAD_MODE=ram. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh doctor first and fix anything it flags, then build, share, pack and serve as the README describes. The API should come up on port 8888 with the model id deepseek-v4.1-flash. A full boot takes 12-13 minutes. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my PC with an NVIDIA GPU. Install and serve Qwen3.8 27B (NVFP4) on vLLM using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-27B-NVFP4-RTX-5090
This recipe needs an NVIDIA RTX 5090 (32 GB, sm_120) on Linux. You're running directly on that machine.
Work through these steps in order:
1. Check my setup before changing anything. Report my operating system; the GPU model, VRAM and driver version (nvidia-smi); Docker and the NVIDIA Container Toolkit if the recipe uses Docker; the Python version if it builds its own Python environment; free disk space (about 22 GiB for the model weights plus about 8 GiB for the Python environment); free RAM; and anything already using the GPU or port 8888. If my GPU doesn't match what the recipe needs, or something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Follow the README's setup for my operating system and GPU. There's no .env to create: settings are environment variables with sensible defaults (PORT, MODEL_DIR, GPU_UTIL). It needs curl, python3, nvidia-smi and gcc. Keep concurrency at 1 unless I ask; the README explains why MTP with this KV cache can't batch. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh: it creates the venv, installs the pinned vLLM, applies the PR #40914 patch, downloads the model and serves on port 8888 with the model id qwen38-nvfp4, then warms up. It binds to 0.0.0.0, so keep it on my local network. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my machine. Install and serve Laguna S 2.1 (NVFP4) with DFlash speculative decoding using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Laguna-S-2.1-DGX-Spark-RTX-6000-PRO
This recipe needs one DGX Spark, or an RTX PRO 6000 (96 GB) with the README's RTX settings. You're running directly on the machine that will serve it; the recipe supports 1 Spark or RTX PRO 6000.
Work through these steps in order:
1. Check my setup before changing anything. Report which of the supported machines this is; the GPU, VRAM (or unified memory) and driver/CUDA version; Docker and the NVIDIA container runtime or the Python version, whichever the recipe uses; free disk space (about 150 GB free for the model cache (the ~71 GB model plus its draft model)); free RAM; and anything already using the GPU or port 8888. If my hardware isn't one the recipe supports, or something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Follow the README's setup, including its per-platform settings for my hardware. If the model is gated, I need a Hugging Face token with access to poolside/Laguna-S-2.1-NVFP4 in HF_TOKEN. Keep --max-num-seqs at 4 (DFlash crashes at the default), don't add --linear-backend flashinfer_b12x, and don't put min_p in the generation config. On an RTX PRO 6000, use the README's RTX column: CUTE_DSL_ARCH for sm_120 and --gpu-memory-utilization 0.78 to start. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Download both models with ./start.sh --download-only, then run ./start.sh. The first start takes about 15 minutes (weight loading, kernel compilation, CUDA graphs), so follow the logs. The API comes up on port 8888 with the model id poolside/Laguna-S-2.1-NVFP4. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark. Install and serve Unsloth Qwen3.6 35B-A3B (NVFP4) using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Unsloth-Qwen3.6-35b-NVFP4-DGX-Spark
This recipe needs one DGX Spark. You're running directly on that Spark.
Work through these steps in order:
1. Check my setup before changing anything. Report the GPU, driver and CUDA version; Docker and the NVIDIA container runtime; free disk space (about 50 GB for the weights and caches); free memory; and anything already using GPU memory or port 8888. If something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the paths and options to match what you found in step 1. There's no .env to create: start.sh has sensible defaults. If I have a Hugging Face token, export it as HF_TOKEN before starting. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh: it pulls the ghcr.io/miaai-lab/mia-vllm-gb10-linear-b12x image, downloads the model if needed and serves on port 8888 with the model id unsloth/Qwen3.6-35B-A3B-NVFP4. The first load takes several minutes. It listens on every interface, so keep it on my local network. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my PC with an NVIDIA GPU. Install and serve Qwen3.8 27B (NVFP4) on SGLang with DFlash 2 using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-27B-RTX-6000-PRO-SGLang-DSpark
This recipe needs an NVIDIA RTX PRO 6000 with 96 GB (Blackwell, SM120); the recipe needs the full 96 GB. You're running directly on that machine.
Work through these steps in order:
1. Check my setup before changing anything. Report my operating system; the GPU model, VRAM and driver version (nvidia-smi); Docker and the NVIDIA Container Toolkit if the recipe uses Docker; the Python version if it builds its own Python environment; free disk space (about 28 GB: the ~24 GB NVFP4 checkpoint plus the ~3.8 GB DFlash 2 drafter, pulled into ./.cache/huggingface on first start); free RAM; and anything already using the GPU or port 8888. If my GPU doesn't match what the recipe needs, or something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Follow the README's setup for my operating system and GPU. It runs in Docker with the NVIDIA Container Toolkit (docker run --gpus all must work). Keep the defaults: full 256K context, 8 concurrent requests and FP8 KV. If I have a Hugging Face token, put it in HF_TOKEN. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh: it pulls the weights on first run and waits until the API is ready on port 8888. Check it with curl http://127.0.0.1:8888/v1/models. Thinking mode is on by default. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve MiMo V2.6 Flash using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/MiMo-V2.6-Flash-2x-DGX-Sparks
This recipe needs exactly 2 Sparks connected over ConnectX-7 (it cannot run on one). You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (about 171 GiB free on the head node; the worker reads the checkpoint over NFS); and anything already using GPU memory or port 8888. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. The README's quick start contains the author's own paths and 10.0.0.x addresses; replace them with mine. Use the SGLang base image the README pins, or newer: older engines expand the MXFP4 experts and can take a node down. Keep EAGLE MTP as the drafter unless I say the server is mainly for code, where the README recommends DFlash. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh doctor and fix anything it flags, then download, build, share and serve as the README describes. The API should come up on port 8888 with the model id MiMo-v2.6-Flash. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve Qwen3.8 Flash Next (NVIDIA's NVFP4 checkpoint by default) on TensorFold's Zig engine, across two DGX Sparks using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Qwen3.8-Flash-Dual-DGX-Sparks-TensorFold
This recipe needs two DGX Sparks (or two GB10 systems with 128 GB) linked by a QSFP cable between their ConnectX-7 ports, each with an IPv4 address in one private subnet, and key-based ssh from the first Spark (the head) to the second (the worker). Each needs about 102 GiB free memory at start for the default 1,048,576-token window. You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (about 150 GB on each Spark: the checkpoint (about 133 GB) under ~/.cache/huggingface and the Docker image (scripts/prepare.sh checks both); with WORKER_WEIGHTS=nfs the worker needs only the image); and anything already using GPU memory or port 8888. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. It needs Docker with the NVIDIA container runtime, my user in the docker group, and rsync, on both Sparks. Copy scripts/local.sh.example to scripts/local.sh and set WORKER to the worker's ssh target (and FABRIC_PEER to its ConnectX-7 address if WORKER is on another network). The example uses WORKER_WEIGHTS=nfs, which needs the head to export its Hugging Face cache to the worker once over NFS (this needs sudo, so ask me first); WORKER_WEIGHTS=copy copies the checkpoint to the worker instead. Keep the defaults (QUANT=nvfp4, 16 requests at once, FP8 KV cache, 1,048,576-token window); QUANT=int4ar is the faster checkpoint, with lower general capability per its model card. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh on the head. The first run sets up both Sparks (the prebuilt image on each, the checkpoint downloaded on the head); later starts take about two minutes. It runs a smoke test through both ranks and prints "Qwen3.8-Flash-Next is now LIVE! on port 8888", with the model id Qwen3.8-Flash-Next. The model thinks before it answers, so give test requests enough max_tokens. ./stop.sh stops both ranks. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark cluster. Install and serve GLM-5.3, Z.ai's full model (EXL3 2.75-bit, Mia-AiLab/GLM-5.3-EXL3-2.75bpw-TensorFold), on TensorFold across three DGX Sparks using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/GLM-5.3-EXL3-3x-DGX-Sparks-TensorFold
This recipe needs three DGX Sparks (GB10, 128 GB each) joined by a triangle of direct ConnectX-7 cables (one QSFP cable per pair of Sparks, each cable its own subnet, RoCE v2), one network all three share for ssh, key-based ssh from the head (rank 0) to both workers, Docker with the NVIDIA container runtime on each, and nothing else large on their GPUs (each rank holds about 86 GiB of weights). You're running on the head Spark and can reach the other Spark(s) over SSH.
Work through these steps in order:
1. Check my setup before changing anything. Report how many Sparks I have and their hostnames and IPs; the ConnectX-7 (RoCE) interfaces and IB HCA names on each node, and whether the nodes can reach each other over them; passwordless SSH from the head to every worker; Docker and the NVIDIA container runtime on every node; free disk space on each node (on the head, about 273 GiB for the checkpoint and 2.4 GiB for the DSpark drafter in its Hugging Face cache, with 100 GB left free; the workers keep no copy of the weights (they read them from the head over a read-only NFS export), only the image (about 25 GB)); and anything already using GPU memory or port 8888. If I don't have enough Sparks, or something required is missing, stop and tell me what's needed.
2. Clone the repo on the head node and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the head and worker IPs, SSH users, network interfaces, IB HCAs and paths to match what you found in step 1. On the head, export ~/.cache/huggingface read-only over NFS to both workers' cable subnets as the README's "Weights over NFS" section shows (this needs sudo on the head, so ask me first). Copy scripts/local.sh.example to scripts/local.sh and set WORKER=user@<rank 1> and WORKER2=user@<rank 2> in it (plus FABRIC_PEER / FABRIC_PEER2 if a worker is reached over another network than its cable). Keep the defaults in scripts/config.sh (CP=1, the 499,712-token window, fp4 KV cache, the DSpark drafter). Run DRY_RUN=1 ./start.sh first: it prints the memory plan and every rank's command without changing anything. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh on the head. The first run sets up all three Sparks (image, checkpoint, NFS mounts, kernels); later starts take 3 to 5 minutes. It runs a smoke test through all three ranks and prints "GLM-5.3-EXL3 is now LIVE! on port 8888", with the model id GLM-5.3-EXL3. The model thinks before it answers, so give test requests enough max_tokens. ./stop.sh stops all three ranks. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my machine. Install and serve NVIDIA Nemotron 3.5 Lightning 30B-A3B (NVFP4) with DSpark using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Nemotron3.5-Lightning-DGX-Spark-RTX-5090-6000-PRO
This recipe needs one DGX Spark, an RTX 5090 (32 GB) or an RTX 6000-class card. You're running directly on the machine that will serve it; the recipe supports 1 Spark, RTX 5090 or RTX PRO 6000.
Work through these steps in order:
1. Check my setup before changing anything. Report which of the supported machines this is; the GPU, VRAM (or unified memory) and driver/CUDA version; Docker and the NVIDIA container runtime or the Python version, whichever the recipe uses; free disk space (at least 30 GiB for the Docker image and both weight sets); free RAM; and anything already using the GPU or port 8888. If my hardware isn't one the recipe supports, or something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Follow the README's setup, including its per-platform settings for my hardware. Settings are environment variables set before ./start.sh. The default MEM_FRACTION_STATIC=0.78 is for a DGX Spark only; on other cards use the README's GPU Memory Quick Reference (0.30–0.45 on an RTX 5090, 0.60–0.70 on an RTX PRO 6000 Blackwell) and set CUDA_GRAPH_MAX_BS_DECODE=2 on the 5090. The script requires at least 80 GiB of host RAM; if I have less, tell me before changing anything. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh: it checks the prerequisites, downloads the model and DSpark draft, pulls lmsysorg/sglang:dev-nemotron3-5-lightning and serves on port 8888. It binds to all interfaces by default, so keep it on my local network. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark. Install and serve Ling 3.0 Flash (INT4) with the DSpark draft using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Ling-3.0-Flash-SGLang-DSpark-DGX-Spark
This recipe needs one DGX Spark. You're running directly on that Spark.
Work through these steps in order:
1. Check my setup before changing anything. Report the GPU, driver and CUDA version; Docker and the NVIDIA container runtime; free disk space (25–30 GB for the weights and the runtime image (the script wants 40 GiB or more free)); free memory; and anything already using GPU memory or port 8888. If something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the paths and options to match what you found in step 1. Settings are environment variables with safe defaults (SPEC_ALGO=dspark, 256k context, 6 concurrent requests), so there's no .env to edit. Keep the defaults unless I ask; the README's MEM_FRACTION_STATIC table shows the memory trade-off. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Download with ./start.sh --download-only, then run ./start.sh. It pulls the public lmsysorg/sglang:dev-Ling-3.0-flash image; don't try to clone inclusionAI's private sglang repo. The API comes up on port 8888 (the model name can be "auto"), and thinking is on by default. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my machine. Install and serve Meta Muse Glimmer 30B (NVFP4) with the DFlash drafter using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Muse-Glimmer-30B-DGX-Spark-RTX-5090-6000-PRO
This recipe needs one DGX Spark, an RTX 5090 (32 GB) or an RTX PRO 6000 (96 GB). You're running directly on the machine that will serve it; the recipe supports 1 Spark, RTX 5090 or RTX PRO 6000.
Work through these steps in order:
1. Check my setup before changing anything. Report which of the supported machines this is; the GPU, VRAM (or unified memory) and driver/CUDA version; Docker and the NVIDIA container runtime or the Python version, whichever the recipe uses; free disk space (about 19 GB of weights plus 5.1 GB for the DFlash drafter and about 10 GB for the Python environment); free RAM; and anything already using the GPU or port 8888. If my hardware isn't one the recipe supports, or something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Follow the README's setup, including its per-platform settings for my hardware. It needs Python 3.12 (not 3.13 or newer) and CUDA 13 (driver 570+). Set GPU_MEM_UTIL for my hardware from the README's platform table: 0.42 on a DGX Spark (the default), about 0.46 on an RTX PRO 6000, and about 0.93–0.95 on a dedicated RTX 5090. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh: it builds the venv, installs the pinned vLLM fork with precompiled kernels, downloads the weights and drafter, and serves on port 8888 with the model id muse-glimmer-30b. Stop it with ./stop.sh, because orphaned engine processes keep memory pinned. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my Linux server with four RTX PRO 6000 GPUs. Install and serve GLM 5.3 Flash (EXL3 4-bit, Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold) on TensorFold v0.6.5, across four RTX PRO 6000 GPUs in one Linux host using the Aevonix Research recipe, made with Mia's AI Lab:
https://github.com/Aevonix/GLM-5.3-Flash-EXL3-4x-RTX-PRO-6000-TensorFold
This recipe needs one Linux x86_64 host with four 96 GB RTX PRO 6000 Blackwell GPUs and CUDA peer access between every pair, Docker with the NVIDIA Container Toolkit (my user in the docker group), Bash, Python 3, Git, GNU patch, curl, gzip and flock, and at least 90,000 MiB free on each GPU at launch. You're running directly on that machine.
Work through these steps in order:
1. Check my setup before changing anything. Report my operating system; the GPU model, VRAM and driver version (nvidia-smi); Docker and the NVIDIA Container Toolkit if the recipe uses Docker; the Python version if it builds its own Python environment; free disk space (about 240 GiB: 180 GiB for the checkpoint, 10 GiB for the DFlash2 drafter, 40 GiB for the image and build, and 10 GiB for kernels); free RAM; and anything already using the GPU or port 8020. If my GPU doesn't match what the recipe needs, or something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Follow the README's setup for my operating system and GPU. Settings come from the environment, then scripts/local.sh, then .env, then the defaults in scripts/config.sh. Keep the defaults: 40 concurrent requests, the 1,048,576-token window, FP8 dense layers, FP8 KV cache and the DFlash2 drafter. The published numbers were measured with each GPU at 250 W; the script warns when the power limit differs and never changes it, so don't change it either without asking me. The DFlash2 drafter is non-commercial only: DRAFTER=mtp ./start.sh on the first run avoids it (one request at a time). DRY_RUN=1 ./start.sh previews every step without Docker or GPUs. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh with the GPUs idle. The first run checks the host, builds TensorFold with all 86 patches, downloads the checkpoint and drafter, compiles kernels and autotunes the launch table; later runs reuse what is ready. It sends a smoke request and prints "GLM-5.3-Flash-EXL3 is now LIVE! on port 8020": the API is http://127.0.0.1:8020/v1 with the model id glm-5.3-flash. Thinking is off by default. ./stop.sh stops it and archives the logs; ./start.sh restart restarts it. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my Linux server with two RTX PRO 6000 GPUs. Install and serve GLM 5.3 Flash (EXL3 4-bit, Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold) on TensorFold v0.6.5, across two RTX PRO 6000 GPUs in one Linux host using the Aevonix Research recipe, made with Mia's AI Lab:
https://github.com/Aevonix/GLM-5.3-Flash-EXL3-2x-RTX-PRO-6000-TensorFold
This recipe needs one Linux x86_64 host with two 96 GB RTX PRO 6000 Blackwell GPUs and CUDA peer access between them, Docker with the NVIDIA Container Toolkit (my user in the docker group), Bash, Python 3, Git, GNU patch, curl, gzip, GNU coreutils and flock, and at least 90,000 MiB free on each GPU at launch. You're running directly on that machine.
Work through these steps in order:
1. Check my setup before changing anything. Report my operating system; the GPU model, VRAM and driver version (nvidia-smi); Docker and the NVIDIA Container Toolkit if the recipe uses Docker; the Python version if it builds its own Python environment; free disk space (about 240 GiB: 180 GiB for the checkpoint, 10 GiB for the DFlash2 drafter, 40 GiB for the image and build, and 10 GiB for kernels); free RAM; and anything already using the GPU or port 8022. If my GPU doesn't match what the recipe needs, or something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Follow the README's setup for my operating system and GPU. Settings come from the environment, then scripts/local.sh, then .env, then the defaults in scripts/config.sh. Keep the defaults: GPUS=0,1, 4 concurrent requests, the 524,288-token window, 4-bit dense layers, FP8 KV cache and the DFlash2 drafter. The published numbers were measured with each GPU at 300 W; the script warns when the power limit differs and never changes it, so don't change it either without asking me. The DFlash2 drafter is non-commercial only: DRAFTER=mtp ./start.sh on the first run avoids it (one request at a time). DRY_RUN=1 ./start.sh previews every step without Docker or GPUs. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh with the GPUs idle. The first run checks the host, builds TensorFold with all 86 patches, downloads the checkpoint and drafter, compiles kernels and runs the one-time two-GPU launch-table autotune; later runs reuse what is ready. It sends a smoke request and prints "GLM-5.3-Flash-EXL3 is now LIVE! on port 8022": the API is http://127.0.0.1:8022/v1 with the model id glm-5.3-flash. Thinking is off by default. ./stop.sh stops it and archives the logs; ./start.sh restart restarts it. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
You are setting up a local AI model server on my NVIDIA DGX Spark. Install and serve NVIDIA Nemotron 3.5 Bolt 74B-A9B (NVIDIA's 24GB checkpoint, with the LiLiCorr drafter) on TensorFold's Zig engine, on a single Spark using Mia's AI Lab recipe:
https://github.com/MiaAI-Lab/Nemotron-3.5-Bolt-TensorFold-DGX-Spark
This recipe needs one DGX Spark (or another GB10 system with 128 GB): the default 16 streams need about 55 GiB free (MemAvailable) when the server starts, and loading takes about 51 GB, so the rest of the Spark stays free for other work. You're running directly on that Spark.
Work through these steps in order:
1. Check my setup before changing anything. Report the GPU, driver and CUDA version; Docker and the NVIDIA container runtime; free disk space (about 23 GiB for the checkpoint under ~/.cache/huggingface plus the Docker image (scripts/prepare.sh checks both)); free memory; and anything already using GPU memory or port 8888. If something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the .env example files, the CHANGELOG, and every warning, "Do not" and "Gotchas" section. Treat those as hard rules. Use the README's own install steps and scripts; don't improvise a different setup.
3. Configure it for my hardware. Create .env from the example file the README names and set the paths and options to match what you found in step 1. It needs Docker with the NVIDIA container runtime and my user in the docker group. There's no .env to create: every setting lives in scripts/config.sh and can be overridden from the environment or a .env file. Keep the defaults (up to 16 streams at once, a 262,144-token window each, bf16 KV cache, image and video input on, the LiLiCorr drafter on). Keep at least 10 GiB of memory free under load: running out of unified memory freezes the Spark. The API listens on every interface (HOST 0.0.0.0), so keep it on my local network. Keep every other default unless my hardware needs a change, and tell me each change you make and why.
4. Download the weights and bring the server up with the recipe's scripts. Run ./start.sh. No prebuilt image is published yet, so the first run builds TensorFold with the recipe's patches and engine, then downloads the checkpoint (23 GiB) and the drafter (1.3 GB); later starts reuse them and load in about 45 seconds. It runs a smoke test and prints LIVE, with the model id Nemotron-3.5-Bolt on port 8888. The model thinks before it answers, so give test requests enough max_tokens (a request without one gets 32,768). ./stop.sh stops it. The first boot is slow (loading weights, warm-up), so follow the logs until the server reports healthy instead of assuming it failed.
5. Verify it works. List the models on the OpenAI-compatible endpoint (/v1/models), send a short chat completion and a streaming request, and run the recipe's smoke test or benchmark if it has one. Compare the decode speed with the README's numbers, if it lists any, and flag any big gap.
6. Finish with a summary: the endpoint URL and model id, every file and setting you changed, and the commands to check status, read the logs, stop and restart the server.
Ground rules: ask me before installing system packages, using sudo, rebooting, or stopping containers that aren't part of this recipe. Never delete model weights or other data. Keep the API on my local network; don't expose it to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
No recipe for that setup yet. More are on the way.
Decode speeds are single-stream figures published in each recipe's README (prose prompts wherever the README separates them); many were measured with sparkDash, my open-source benchmark dashboard. The meters run from 0 to 60 tok/s. GitHub numbers update every 15 minutes.
Lab tools
The dashboard behind every number.
For DGX Sparks & NVIDIA GPUs
sparkDash
A real-time dashboard for one or many DGX Sparks and NVIDIA GPU machines, and the benchmark behind the speeds on this page.
Monitor one or many DGX Sparks and NVIDIA GPU machines in one window
Live GPU, unified memory, storage and network metrics
Detects your local LLM server (vLLM, SGLang, llama.cpp and more) and shows live tok/s
Built-in decode and prefill benchmarks, the ones behind the numbers above
Prompt Showcase: stream up to 32 prompts side by side
You are installing sparkDash, the open-source monitoring and benchmark dashboard from Mia's AI Lab, for NVIDIA DGX Sparks and NVIDIA GPU machines:
https://github.com/MiaAI-Lab/sparkDash
Work through these steps in order:
1. Check my setup before changing anything. Report which machine this is (a DGX Spark or a Linux PC with an NVIDIA GPU); Docker with Compose; the NVIDIA driver (nvidia-smi); free disk space; and whether port 5555 is free. Ask me which other Sparks or GPU machines I want to monitor and their IPs, and check that this machine can SSH into each of them with a key. If something required is missing, stop and tell me what's needed.
2. Clone the repo and read the README in full before running anything, including the Quick start, Configuration and Security sections. Treat the security notes as hard rules.
3. Start it with Docker as the README describes (docker compose up --build -d). It listens on 127.0.0.1:5555 by default; keep it that way. If I want it from another computer, set up the SSH tunnel the README shows, or ask me before binding it to the network, which also requires setting SPARKDASH_TOKEN.
4. Mount my SSH key into the container as the README explains, so it can reach the other machines. Prefer keys over passwords, and never delete config/.secrets-key.
5. Open the dashboard, add each machine with the right role (head, worker, standalone, or a dedicated GPU host), and confirm that GPU, memory and network metrics are streaming. If an LLM server is running, check that the LLM card detects it and shows tok/s.
6. Finish with a summary: the dashboard URL, how to open it from my laptop, the machines you added, and the commands to update, stop and restart it.
Ground rules: ask me before installing system packages, using sudo, or changing SSH settings on any machine. Don't expose the dashboard to the internet. If a step fails, read the logs and the README's troubleshooting notes before trying fixes, and tell me what you changed.
Tier list
Model tier list
Every model I've tried, cloud or local, ranked by how well it worked for me.
