Serve DeepSeek V4-Flash with vLLM on a GH200 server, from the shared content, and query it through an OpenAI-compatible API.
A GH200 has 96 GB of GPU memory and 573 GB of system memory that the GPU reaches over NVLink-C2C. That combination runs language models larger than the GPU memory alone: vLLM keeps the dense layers on the GPU and moves the mixture-of-experts weights to Grace memory.
On COSMOS the model weights (157 GB) and the vLLM container are shared content, published read-only as the llm project. The GPU-compute base copies them to the local data disk at boot and checks them, because vLLM loads its weights five times faster from local disk than from the network share. You enable the project, start the server with one script, and send requests to it.
After completing this tutorial you will be able to:
| Difficulty | Intermediate |
|---|---|
| Estimated time | 35 min on a fresh server (12 min copy + 9 min start), 10 min afterwards |
| Domain / sandbox | rrail (srv8.rrail or srv9.rrail) |
| Image | gpu-compute-gh200.ndz |
| Transmits? | no |
| Last verified | 2026-11-14, gpu-compute-gh200-20261114 and content llm 20261114 on srv8.rrail |
Background knowledge
Account & access
Devices / nodes
| Resource | Role | Qty | Notes |
|---|---|---|---|
srv8.rrail or srv9.rrail |
inference server | 1 | NVIDIA GH200 480GB; needs the 7.8 TB data disk for the local copy |
Disk images
| Image | Load onto | Provides |
|---|---|---|
gpu-compute-gh200.ndz |
the GH200 server | driver, Docker with GPU support, the shared-content tools |
Software components
| Component | Version | Source |
|---|---|---|
| Model | deepseek-ai/DeepSeek-V4-Flash-0731 |
shared content llm 20261114, hf/ |
| vLLM | vllm/vllm-openai:v0.33.0 (arm64) |
shared content, containers/ (loaded into Docker on first use) |
| Start script | llm-serve.sh |
shared content, bin/ |
One server. The weights are copied from the read-only share to /mnt/datadisk/cache/llm and checked. The server listens on the node's control-network address, port 8000, so other COSMOS machines can reach it; every request needs the API key.
cosmos-content enable llm
cosmos-content sync
enabled llm
llm cache 20261114 (verified)
sionna-bench nfs (nothing marked copy)
Run the start script as a service, so it survives your SSH session:
systemd-run --unit llm-serve --collect bash /opt/content/llm/bin/llm-serve.sh v4flash
journalctl -u llm-serve -f
On the first start the script loads the vLLM container from the share and creates an API key in /root/llm/api-key (mode 600, kept on this server only). The log then shows the weights loading and, minutes later, the server becoming ready:
... [default_loader.py:506] Loading weights took 69.40 seconds
... [model_runner.py:428] Model loading took 45.04 GiB memory and 102.607078 seconds
INFO: Application startup complete.
From a fresh copy it takes about 9 minutes until the API answers; vLLM compiles its GPU kernels on the first start and reuses them from your work area (/mnt/datadisk/work/llm) afterwards. Stop following the log with Ctrl-C; the server keeps running.
KEY=$(cat /root/llm/api-key)
BIND=$(ip -4 -o addr show CTRL | awk '{print $4}' | cut -d/ -f1)
curl -s -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' http://$BIND:8000/v1/chat/completions \
-d '{"model":"deepseek-v4-flash","max_tokens":200,"messages":[{"role":"user","content":"In two sentences, what is a GH200?"}]}' \
| python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["choices"][0]["message"]["content"]); print(d["usage"])'
The GH200 is NVIDIA's Grace Hopper superchip, which tightly integrates a Grace CPU and Hopper GPU using a high-speed NVLink-C2C interconnect for unified memory access. It is designed for large-scale AI training, inference, and high-performance computing workloads that require massive memory bandwidth and energy efficiency.
{'prompt_tokens': 93, 'total_tokens': 206, 'completion_tokens': 113, 'prompt_tokens_details': None, 'completion_tokens_details': {'reasoning_tokens': 49}}
The wording of the answer differs from run to run. This one took 1.9 s. Any OpenAI-compatible client works: point it at http://<server CTRL address>:8000/v1 with the key.
llm-serve.sh runs vLLM in Docker with the GPU. It mounts the weights read-only from /opt/content/llm/hf and gives vLLM a writable cache in your work area.v4flash profile keeps 65,536 tokens of context, stores the attention cache in FP8, and offloads 100 GB of expert weights to Grace memory (--cpu-offload-gb 100 --cpu-offload-params experts). PORT, MAXLEN, OFFLOAD_GB, GPU_UTIL and EXTRA override the defaults.HF_HUB_OFFLINE=1: nothing is downloaded at run time; the weights are the published release.cosmos-content sync reports llm cache 20261114 (verified), the log ends with Application startup complete., and the request returns an answer with a usage block.
curl -s -H "Authorization: Bearer $(cat /root/llm/api-key)" http://$BIND:8000/v1/models | python3 -c 'import json,sys; print([m["id"] for m in json.load(sys.stdin)["data"]])'
['deepseek-v4-flash']
systemctl stop llm-serve
docker rm -f llm-serve
Before saving an image, remove the vLLM container image (docker rmi vllm/vllm-openai:v0.33.0); it is reloaded from the share on the next start. Delete /root/llm/api-key if the next user should not inherit it.
| Symptom | Cause | Fix |
|---|---|---|
no /opt/content/llm/hf |
the project is not enabled | Setup step 2 |
cosmos-content says nfs (no data disk) |
no data disk: the weights would load from the share | it still works, about 5 minutes slower per start |
the server stops during start, CUDA out of memory |
another process holds GPU memory | nvidia-smi; stop it, or lower GPU_UTIL |
401 Unauthorized |
missing or wrong key | send Authorization: Bearer $(cat /root/llm/api-key) |
Group: GPU computing · Image: gpu-compute-gh200.ndz · Content: llm 20261114 · Last verified 2026-11-14 on srv8.rrail