Load gpu-compute-gh200.ndz on a GH200 server, check the GPU, run a GPU container, and use the shared content and the local data disk.
gpu-compute-gh200.ndz is the starting image for GPU work on srv8.rrail and srv9.rrail. It holds Ubuntu 24.04 for arm64 with NVIDIA's 64k-page kernel, the NVIDIA driver, Docker with GPU support, and the tools that connect the server to the shared content. It has no real-time kernel settings, so all 72 Grace cores and the whole GPU are yours.
Large software (Python environments, model weights, SDKs) is not in the image. It lives on a read-only network share and is used from there, or from a checked local copy where that is faster. GPU computing on the GH200 servers explains how that works, and what to watch for when you save an image of your own.
After completing this tutorial you will be able to:
/opt/content;| Difficulty | Beginner |
|---|---|
| Estimated time | 15 min |
| Domain / sandbox | rrail (srv8.rrail or srv9.rrail) |
| Image | gpu-compute-gh200.ndz |
| Transmits? | no |
| Last verified | 2026-11-14, gpu-compute-gh200-20261114 on srv8.rrail |
Background knowledge
Account & access
Devices / nodes
| Resource | Role | Qty | Notes |
|---|---|---|---|
srv8.rrail or srv9.rrail |
GPU computer | 1 | NVIDIA GH200 480GB: 72-core Arm Neoverse V2, Hopper GPU with 96 GB HBM3, 573 GB system memory, 7.8 TB data disk. arm64 only |
Disk images
| Image | Load onto | Provides |
|---|---|---|
gpu-compute-gh200.ndz |
the GH200 server | Ubuntu 24.04 arm64, kernel 6.17.0-1018-nvidia-64k, NVIDIA driver 610.57.04, Docker 29 with the NVIDIA container toolkit, cosmos-content, cosmos-datadisk, cosmos-shared-content |
Software components
| Component | Version | Source |
|---|---|---|
| NVIDIA driver | 610.57.04 (open kernel modules) | in the image |
| Docker + NVIDIA container toolkit | 29.x + 1.20 | in the image |
| Shared content | per project | read-only share at /opt/aerial-data, used through /opt/content/<project> |
One server, no radio. The server mounts the shared content read-only from the COSMOS network storage. Its local data disk holds copies of the content that are faster to use locally, and your work area.
rrail domain:ssh <username>@console.rrail.cosmos-lab.org
omf load -t srv8.rrail.cosmos-lab.org -i gpu-compute-gh200.ndz -r 0 -o 2400 -b 600
Read the last line: 1 node successfully imaged means it worked.omf tell -a on -t srv8.rrail.cosmos-lab.org
ssh root@srv8.rrail.cosmos-lab.org
nvidia-smi --query-gpu=name,driver_version,memory.total,persistence_mode,power.limit --format=csv
name, driver_version, memory.total [MiB], persistence_mode, power.limit [W]
NVIDIA GH200 480GB, 610.57.04, 97871 MiB, Enabled, 900.00 W
Persistence mode is on, so the GPU stays initialised between jobs and the first CUDA call of a job does not pay a start-up delay.
docker run --rm --gpus all ubuntu:24.04 nvidia-smi -L
Unable to find image 'ubuntu:24.04' locally
24.04: Pulling from library/ubuntu
...
Status: Downloaded newer image for ubuntu:24.04
GPU 0: NVIDIA GH200 480GB (UUID: GPU-5c1bb047-743a-985b-b831-684e240b0cbf)
--gpus all hands the GPU to the container. Containers must be built for arm64; most images on Docker Hub and NVIDIA NGC publish one.
cosmos-content list
gpu-base 20261114
sionna-bench 20261114
Each project is used through /opt/content/<project>. Enable the ones you want, then point the links:
cosmos-content enable sionna-bench
cosmos-content sync
cosmos-content status
sionna-bench nfs (nothing marked copy)
sionna-bench -> /opt/aerial-data/sionna-bench/20261114
nfs (nothing marked copy) means this project runs straight from the share: it was measured to be just as fast from there. A project whose items are marked for copying shows cache <version> (verified) instead, and its link points into /mnt/datadisk/cache. Enabled projects are synced again at every boot.
cat /run/cosmos/datadisk
df -h /mnt/datadisk
mounted (existing VG datadisk)
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/datadisk-data 7.8T 55G 7.4T 1% /mnt/datadisk
Work in /mnt/datadisk/work/<your username>. It is fast and large, but it is not storage: whoever uses the server next may load another image that wipes it. Copy anything you want to keep off the server before your reservation ends.
cosmos-shared-content check
KEEP docker ubuntu:24.04 - not an archive on the share (built or changed on this node)
WARNING: 1 item(s) above are KEPT in the image: they were changed or created on this node, or did not come from the share. Remove what you do not need before saving: every GB makes the image slower to load.
This lists what prepare.sh would remove (unmodified copies of shared content) and what stays in a saved image. Here the ubuntu:24.04 image pulled in Step 2 would stay; remove it with docker rmi ubuntu:24.04 if you do not need it.
cosmos-datadisk.service mounts the data disk at boot. It creates it only when its three drives are completely empty, and never wipes a disk that holds anything; then the server runs without a data disk and all content comes from the share.cosmos-content.service syncs the enabled projects at boot. Items a project marks as copy go to /mnt/datadisk/cache/<project>/<version>, and every file is checked against the share's SHA256SUMS before the link is pointed at it./opt/aerial-data is mounted read-only and on demand (an automount), so a server without the share still boots./root/prepare.sh runs the hooks in /etc/cosmos/prepare.d before it unmounts the share. The GH200 hook is cosmos-shared-content clean.nvidia-smi names the GH200 with driver 610.57.04, a container sees the same GPU, cosmos-content list names the published projects, and /mnt/datadisk is mounted.
cat /.omf-image-name
systemctl --failed --no-legend
gpu-compute-gh200.ndz
The second command prints nothing: no failed services.
Remove what you created that you do not want in a saved image (containers, pulled images, files outside your work area). Copy your results off the server. Then turn it off from the console:
omf tell -a offh -t srv8.rrail.cosmos-lab.org
| Symptom | Cause | Fix |
|---|---|---|
exec format error when starting a container |
the image was built for x86 only | use an arm64 (linux/arm64) image |
cosmos-content says nfs (no data disk) |
the data disk was not mounted: its drives hold something from another image | cat /run/cosmos/datadisk gives the reason; the content works from the share |
ls /opt/aerial-data hangs |
the share is not reachable | check the network; the server works without it, but shared projects are not available |
omf load refused |
no reservation for this server | book it; see Prerequisites |
Group: GPU computing · Image: gpu-compute-gh200.ndz · Last verified 2026-11-14 on srv8.rrail