COSMOS has two NVIDIA GH200 Grace Hopper servers, srv8.rrail and srv9.rrail. This section covers using them as GPU computers: machine learning, simulation with Sionna, and serving large language models. Two x86 servers in sb1 also carry a Tesla V100: see GPU compute on the sb1 V100 servers. For the 5G physical layer on the GPU (NVIDIA Aerial with O-RAN radios), see NVIDIA Aerial on COSMOS.
This page explains how the GH200 images are built and how they run. Read it before you save an image of your own.
| Item | Each of srv8.rrail and srv9.rrail |
|---|---|
| Server | Supermicro ARS-111GL-NHR |
| Superchip | NVIDIA GH200 480GB: a 72-core Arm Neoverse V2 CPU (Grace) and a Hopper GPU on one module, joined by NVLink-C2C |
| GPU memory | 96 GB HBM3 (nvidia-smi: 97,871 MiB), power limit 900 W |
| System memory | 573 GB visible to Linux. The GPU can also use it over NVLink-C2C, which is how models larger than 96 GB run |
| Local disks | The system disk (880 GB) holds the image. A separate 7.8 TB data disk (three NVMe drives) is used as a cache and work area |
| Architecture | arm64: the servers boot arm64 images only, and x86 binaries or containers do not run |
| Image | Use it for |
|---|---|
gpu-compute-gh200.ndz |
Start here. The GPU-compute base: NVIDIA driver 610.57.04, Docker with GPU support, and the shared-content tools described below. No real-time kernel settings, so all 72 cores are available |
aerial-26-2-gh200.ndz |
NVIDIA Aerial (5G on the GPU). See NVIDIA Aerial on COSMOS |
ubuntu2404-arm64-baseline.ndz |
Plain Ubuntu 24.04 arm64 without a GPU driver, for building something from scratch |
Load an image the usual way, with the large-image options. The image is small, so a load takes a few minutes:
omf load -t srv8.rrail.cosmos-lab.org -i gpu-compute-gh200.ndz -r 0 -o 2400 -b 600
omf tell -a on -t srv8.rrail.cosmos-lab.org
Large software is not inside the images: Python environments with PyTorch, model weights, container archives and SDKs add up to hundreds of GB. Loading an image runs at about 22 MB/s, so every GB inside an image costs about 47 s of every load. Instead:
/opt/aerial-data. COSMOS staff publish it there, one directory per project and version. Each version has a MANIFEST.json and a SHA256SUMS checksum file, and current points to the version in use. Nodes can read the share but cannot change it, so nothing a user does on one machine can break the content every image depends on.cosmos-content service copies each enabled project's "copy" items into /mnt/datadisk/cache, then checks every file against the published checksums. A copy that fails the check is not used./opt/content/<project>. It points to the checked local copy when there is one, and straight to the network share otherwise (nothing to copy, no data disk, or a failed check). Your commands do not change either way.Whether an item is copied is decided per item, by measurement: copy it only when running from the network is slower. Content that is only read once, or that runs just as fast from the share, stays there and costs nothing at boot.
| Content | Running from the network share vs. a local copy | Decision |
|---|---|---|
| Aerial SDK (16 GB, 72,000 files) | the GPU slot timing is about 2× slower from the share; reading the tree takes 166 s from the share vs. 30 s locally | copied at boot (Aerial image) |
| Container images (Aerial cuBB, layer 2) | Docker runs images only from its local store | loaded at boot into Docker |
| Sionna Bench (5 GB Python environment) | the same speed: cold start 6.4 s vs. 7.0 s, fast test suite (267 tests) 112 s vs. 111 s (measured 2026-11-14 on gpu-compute-gh200) |
runs from the share |
| Language model weights (DeepSeek V4-Flash, 157 GB, plus the vLLM container) | vLLM loads the weights in 371 s from the share vs. 69 s locally; the server is ready in 18 min vs. 8.7 min (measured 2026-11-14) | copied at boot |
The first copy of a project includes checking every file: Sionna Bench (5 GB) took 59 s, the language model (167 GB) 12 minutes. After that, the boot-time check only confirms that nothing changed (6 to 7 s).
Commands:
cosmos-content list # projects on the share and their current version
cosmos-content enable sionna-bench # copy this project at every boot
cosmos-content sync # copy + check now, without a reboot
cosmos-content status # where each /opt/content/<project> points
| Location | What it is | Lifetime |
|---|---|---|
/opt/aerial-data, /opt/content/* |
The shared content | Read-only |
/mnt/datadisk/cache |
Local copies of the shared content | Rebuilt by cosmos-content whenever needed |
/mnt/datadisk/work |
Your work area: training outputs, checkpoints, builds | Not safe, see the warning below |
| The image (system disk) | What omf save captures |
Until you delete your image |
The data disk is not storage. Whoever reserves the server after you can load any image, and that image may reformat or fill the data disk. Never keep the only copy of anything on it. Copy results you want to keep off the server before your reservation ends.
The image only sets the data disk up when its three drives are completely empty. It never wipes a disk that holds anything, and in that case it runs from the network share.
You can install your own software and save an image with omf save, as on any COSMOS node. Two things on a GH200 are different.
Copies of the shared content would end up in your image. Aerial's container images, for example, are loaded from the share into Docker's store on the system disk at every boot, so they are part of anything you save. That makes your image tens of GB larger, and every load of it slower, for content the next boot copies again anyway.
prepare.sh removes those copies for you, but only when it can prove they are unmodified. Before saving, run /root/prepare.sh as on every COSMOS image. On the GH200 images (gpu-compute-gh200.ndz and aerial-26-2-gh200.ndz) it calls cosmos-shared-content, which works through these rules:
| What | Removed only if | Otherwise |
|---|---|---|
| A Docker image | its content-addressed ID matches an archive on the share that is reloaded automatically (the Aerial containers at every boot, a project's containers on first use) and no container (running or stopped) was created from it | kept: an image you built or changed has a different ID, a container may hold your changes, and images nothing reloads (such as the Aerial image's 5G core) belong to the image |
| A copied content tree | every file matches the share's checksums, nothing is missing, links are identical, and the only extra files are ones the software writes when it runs (__pycache__, *.pyc) |
kept, with the differences listed |
| Anything else, or the share is not reachable | never removed | kept |
Anything it keeps is listed with the reason. Read the output before you save:
cosmos-shared-content check # what would be removed, what is kept and why; changes nothing
To keep everything regardless, run touch /etc/cosmos/keep-shared-content before prepare.sh.
Keep your own images small. Start from gpu-compute-gh200.ndz. Install only what must be in the image, and put large data and outputs in the work area instead. Every GB in your image is about 47 s added to each load.
Your saved images count against a 120 GB allowance per user, and GH200 images reach it quickly. The allowance covers all your saved images together, on every COSMOS domain. An image saved from
aerial-26-2-gh200.ndzis about 14 GB even afterprepare.shhas removed the shared copies. Add a Python environment, a container you built, or a dataset left on the system disk, and a single image can pass 50 GB, so two or three saves fill the allowance. When you are over it,omf savewill refuse to save a new image. The allowance is being introduced now: it starts with warnings only.
- Check what an image would carry with
cosmos-shared-content checkbefore you save it.- Delete images you no longer need from a console:
rm /mnt/images/<your image>.ndz(you can only delete your own).- A project that needs more can ask for a larger allowance.
The GH200 servers are in the rrail project domain. Each server is booked as its own resource, and omf load, omf save and omf tell refuse a server you have no reservation for.
Status, November 2026. The servers are being opened to all COSMOS users in phases. Read-only shared content, per-user image allowances, a shared scratch area and per-reservation login are being put in place first. Until then, access is through the
rrailproject.
| Tutorial | What you learn |
|---|---|
| The GPU-compute base | load the image, check the GPU, run a GPU container, use the shared content and the data disk |
| Sionna Bench | run NVIDIA's Sionna Bench from the shared content, and where its published results are |
| Serving a large language model | serve DeepSeek V4-Flash (157 GB) with vLLM, larger than the GPU memory, and query it through an OpenAI-compatible API |
| The AI Telco Engineer | run NVIDIA's agentic PHY-design framework against the language model on the same GH200 (guide, not packaged) |
| ISAC sensing: replaying NVIDIA's Aerial ISAC dataset | replay NVIDIA's public SRS I/Q recording through the E3 Agent into a COSMOS sensing dApp and score it against camera labels (replay only; live sensing on a COSMOS cell is not done yet) |
| GPU compute on the sb1 V100 servers | the Tesla V100 servers in sb1: driver and PyTorch rules for Volta, Sionna Bench, AITE workers against the GH200's model (guide, not packaged) |
| NVIDIA Aerial on COSMOS | the 5G physical layer on the GPU, with O-RAN radios (in the 4G/5G section) |