Part of: NVIDIA Aerial on COSMOS · 4G / 5G & O-RAN · Step 1 of the Aerial learning path
Run NVIDIA Aerial's GPU physical layer (cuPHY) on a GH200 server, srv8.rrail or srv9.rrail. Decode an uplink (PUSCH) slot, build a downlink (PDSCH) slot and a synchronisation block (SSB), check each against NVIDIA's reference output, read the GPU time per slot, then find how many cells one GPU can carry.
There is no radio, no fronthaul and no layer 2 here: each pipeline reads a test vector, an HDF5 file holding the channel's configuration, its input and the output NVIDIA's MATLAB model produced for it. So a mismatch can only come from the GPU pipeline itself. This is the reference run for every later Aerial tutorial.
Concepts
- PUSCH / PDSCH — the uplink and downlink data channels. SSB — the synchronisation and broadcast block a phone finds first.
- cuPHY pipeline — the chain of CUDA kernels for one channel (for PUSCH: channel estimation, equalisation, demodulation, LDPC decoding, CRC).
- Test vector (TV) — an HDF5 file with the inputs and reference outputs for one channel and slot.
- Graphs mode — the pipeline's kernels are captured once as a CUDA graph and replayed each slot, which removes launch overhead.
- Cell capacity — the largest number of cells for which every channel finishes inside its latency budget in every slot.
aerial-run.| Difficulty | Intermediate |
|---|---|
| Estimated time | 35–45 min (image load about 5, first boot about 5, one-time SDK copy about 4; the full capacity sweep adds about 70) |
| Domain | rrail (reserved project domain) |
| Node | srv8.rrail or srv9.rrail (NVIDIA GH200, arm64; this page uses srv9) |
| Image | aerial-26-2-gh200.ndz |
| Transmits? | No |
| Last verified | 2026-11-05 · aerial-26-2-gh200.ndz (14.2 GB, SDK and containers on NFS) · Aerial 26-2 · srv9.rrail |
rrail project group, and SSH access to console.rrail.cosmos-lab.org — see Getting started and SSH access.This tutorial is run by hand. It is not yet a cosmos-tutorial package, so there is no Ansible or Jupyter mode and no parity check yet.
Devices / nodes
| Resource | Role | Qty | Notes |
|---|---|---|---|
srv8.rrail or srv9.rrail |
runs cuPHY on its GPU | 1 | NVIDIA GH200 480GB, 72-core Arm Neoverse V2; boots arm64 images only. Each has a 7.8 TB local scratch volume at /mnt/datadisk |
Disk images
| Image | Load onto | Provides |
|---|---|---|
aerial-26-2-gh200.ndz |
srv9 (or srv8) |
Ubuntu 24.04 arm64, NVIDIA 6.17-nvidia-64k kernel, GPU driver 610.57.04, DOCA 3.3.0, aerial-run, and the mounts below. The Aerial SDK and the cuBB container are not in the image: they come from the COSMOS Aerial share at boot |
Software components
| Component | Version | Where |
|---|---|---|
| Aerial CUDA-Accelerated RAN | 26-2 | source, build.aarch64/ binaries, pyAerial: /opt/aerial/26-2/sdk on the NFS share (read-only), also reachable as /opt/aerial-cuda-accelerated-ran |
| cuBB container | nvcr.io/nvidia/aerial/aerial-cuda-accelerated-ran:26-2-cubb (CUDA 13.3) |
loaded at boot from /opt/aerial/26-2/containers by aerial-images.service; started by aerial-run |
| Test vectors | generated with Aerial 26-2's 5GModel | testVectors/ in the SDK |
One server. Everything runs on its GPU inside the cuBB container:
| Path on the server | Inside the container | What it is |
|---|---|---|
/opt/aerial (NFS, read-only) |
— | the COSMOS Aerial share: 26-2/sdk/ (the SDK) and 26-2/containers/ (the docker images) |
/mnt/datadisk/aerial-sdk (your local copy, made in Setup) |
/opt/nvidia/cuBB (the working directory) |
Aerial source tree, its build.aarch64/ binaries, testVectors/ with the tutorial/*.yaml cell configurations |
the directory you run aerial-run from |
/work |
your results |
aerial-run mounts the SDK named by AERIAL_SDK (default: the NFS copy at /opt/aerial-cuda-accelerated-ran). /mnt/datadisk is a 7.8 TB local cache and scratch area. Any user of the machine (any image) may wipe it, so keep nothing there you cannot recreate, and copy results you want to keep off the node before you release it. Nothing on it is backed up. See GPU computing on COSMOS for the GH200 images, what the data disk is for, and saving your own image.
At boot, the image's nvidia.service locks the GPU clock at 1980 MHz and turns MIG off, so timings are repeatable.
Log into the console:
ssh <username>@console.rrail.cosmos-lab.org
Load the image, keeping the full disk with -r 0. Use the full node name: srv8/srv9 exist in other domains too.
omf load -i aerial-26-2-gh200.ndz -t srv9.rrail.cosmos-lab.org -r 0 -o 3600 -b 900
Loading disk image onto 1 node
Domain: rrail.cosmos-lab.org
Image: aerial-26-2-gh200.ndz
Disk: pci:0004:01:00.0
Boot reset timeout: 900s
Resize: full disk
Disk loading timeout: 3600s
Powering nodes off, then on into PXE...
...
Booted 1/1 nodes
Loading image 'aerial-26-2-gh200.ndz' via 224.0.0.5:7170...
-----------------------------
Imaging Process Done
1 node successfully imaged - Topology saved in '/tmp/omf-load-1793883647-topo-success.rb'
-----------------------------
The load takes about 4 minutes after the node has booted into the imaging system (about 2 more). Read the summary line: 1 node successfully imaged is success. A TIMEOUT line is a failure.
The load leaves the node powered off. Turn it on:
omf tell -a on -t srv9.rrail.cosmos-lab.org
It boots in about three minutes. During boot, aerial-images.service loads the Aerial docker images from the share, which takes about two more minutes (the 40 GB cuBB image is most of it).
Log into the node. Check that the share and the local volume are mounted, that the images are loaded and that the GPU clock is locked:
ssh root@srv9
findmnt -n -o SOURCE,OPTIONS /opt/aerial | cut -c1-40
findmnt -n -o SOURCE,SIZE /mnt/datadisk
systemctl is-active nvidia.service aerial-images.service
docker images --format '{{.Repository}}:{{.Tag}} {{.Size}}' | grep cubb
nvidia-smi --query-gpu=name,clocks.gr,mig.mode.current --format=csv,noheader
10.0.0.41:/opt/aerial ro,relatime,vers=3
/dev/mapper/datadisk-data 7.8T
active
active
nvcr.io/nvidia/aerial/aerial-cuda-accelerated-ran:26-2-cubb 40.2GB
NVIDIA GH200 480GB, 1980 MHz, Disabled
If aerial-images.service is still activating, wait: aerial-run waits for it too, and prints loading Aerial images from /opt/aerial while it does.
Copy the SDK to the local volume and use that copy for everything on this page:
rsync -a --delete /opt/aerial/26-2/sdk/ /mnt/datadisk/aerial-sdk/
export AERIAL_SDK=/mnt/datadisk/aerial-sdk
The first copy takes about 4 minutes (16 GB); later runs of the same rsync only check for changes (about 15 s). The SDK copy is re-synced from /opt/aerial/26-2/sdk whenever it is missing or stale, so run this step at the start of every session. Add the export to ~/.bashrc if you log in again.
Why a local copy. The GPU times of the multi-cell PDSCH example (Step 4) roughly double, and jump around, when the SDK is read straight from the NFS share: 820 µs per slot instead of 380 µs, measured on the same node. The single-cell steps are unaffected. Timing measurements should always run from the local copy.
Run every command below as root on the server, from /root, with AERIAL_SDK set as in Setup step 5. aerial-run <command> runs one command in the cuBB container, in the SDK directory. Each run first prints the container's CUDA and licence banner (omitted below).
cd /root
aerial-run ./build.aarch64/cuPHY/examples/pusch_rx_multi_pipe/cuphy_ex_pusch_rx_multi_pipe \
-i testVectors/TVnr_7201_PUSCH_gNB_CUPHY_s0p0.h5 -m 1
-m 1 runs the pipeline in graphs mode. It runs the slot 1000 times and prints two timing tables, then the result check:
PuschRx Pipeline[00]: Metric - GPU Time usec (using CUDA events, over 1000 runs)
PuschRx Pipeline[00]: Phase Mean Min P90 P99 Max
PuschRx Pipeline[00]: Run-P1 1.323 1.280 1.344 1.440 1.536
PuschRx Pipeline[00]: Run-P2 186.464 181.056 188.544 189.536 204.800
PuschRx Pipeline[00]: Run-P3 7.781 3.008 10.720 10.944 11.008
PuschRx Pipeline[00]: Setup-P1 4.795 4.704 4.928 4.928 6.272
PuschRx Pipeline[00]: Setup-P2 15.834 10.784 18.595 19.264 29.056
PuschRx Pipeline[00]: Total 216.197 205.120 221.024 222.208 250.432
PuschRx Pipeline[00]: Metric - CPU Time usec (using wall clock w/ 1 ms delay kernel, over 1000 runs)
PuschRx Pipeline[00]: Phase Mean Min P90 P99 Max
...
PuschRx Pipeline[00]: Total 30.871 26.496 33.056 37.763 185.090
...
[CUPHY.PUSCH_RX] Cell # 0 : TbIdx: 0 Metric - Block Error Rate : 0.0000 (Error CBs 0, Mismatched CBs 0, MismatchedCRC CBs 0, Total CBs 5)
[CUPHY.PUSCH_RX] Cell # 0 : Metric - TB CRC Error :(MismatchedCRC TBs 0, Total TBs 1)
Setup is per-slot configuration, Run-P2 the main processing (channel estimation through LDPC decoding).Try TVnr_7031_PUSCH_gNB_CUPHY_s0p0.h5, a one-code-block slot: it should also show 0 errors.
aerial-run ./build.aarch64/cuPHY/examples/pdsch_tx/cuphy_ex_pdsch_tx \
testVectors/TVnr_3001_PDSCH_gNB_CUPHY_s0p0.h5 20 0 1
The arguments are the test vector, 20 timing iterations, AAS mode off, graphs mode on. The example checks every stage against the reference before timing:
[CUPHY.PDSCH_TX] CRC Error Count: 0; GPU output compared w/ reference dataset(s) <tb*_cbs> from <testVectors/TVnr_3001_PDSCH_gNB_CUPHY_s0p0.h5>
[CUPHY.PDSCH_TX] LDPC Error Count: 0; GPU output compared w/ reference dataset <tb*_codedcbs> from <testVectors/TVnr_3001_PDSCH_gNB_CUPHY_s0p0.h5>
[CUPHY.PDSCH_TX] Fused Rate Matching and Modulation Mapper: Found 0 mismatched QAM symbols out of 16380
[CUPHY.PDSCH_TX] PDSCH: Found 0 mismatched symbols out of 183456 in {3276, 14, 4} output tensor.
[CUPHY.PDSCH_TX] DL pipeline: 36.14 us (avg. over 20 iterations)
The stages are CRC attachment, LDPC encoding, rate matching with modulation, and the final resource grid: 3276 subcarriers × 14 OFDM symbols × 4 antenna ports. The example prints a deprecation notice: NVIDIA is replacing it with pdsch_tx_multi_cell, used in Step 4.
The SSB example reads a small YAML file that lists the test vector for each cell and slot. The image has them in testVectors/tutorial/:
cat testVectors/tutorial/ssb_1cell.yaml
aerial-run ./build.aarch64/cuPHY/examples/ssb_tx_multi_cell/cuphy_ex_ssb_tx_multi_cell \
-i testVectors/tutorial/ssb_1cell.yaml -r 100 -k -m 1
cells: 1
slots:
- SSB:
- testVectors/TVnr_1902_SSB_gNB_CUPHY_s0p0.h5
...
[CUPHY.SSB_TX] ====> TV testVectors/TVnr_1902_SSB_gNB_CUPHY_s0p0.h5: Test PASS
[CUPHY.SSB_TX] Slot # 0, SSB pipeline(s) GPU-run only in Graphs mode: 13.16 us (avg. over 100 iterations)
[CUPHY.SSB_TX] --> Cell # 0 : 10.42 us (avg over 100 iterations)
-k turns on the reference check and -r 100 times 100 iterations.
The same examples take several cells per slot. Each cell has its own pipeline, and the pipelines share the GPU:
aerial-run ./build.aarch64/cuPHY/examples/ssb_tx_multi_cell/cuphy_ex_ssb_tx_multi_cell \
-i testVectors/tutorial/ssb_4cell.yaml -r 100 -k -m 1
aerial-run ./build.aarch64/cuPHY/examples/pdsch_tx_multi_cell/cuphy_ex_pdsch_tx_multi_cell \
-i testVectors/tutorial/pdsch_4cell.yaml -r 100 -k -m 1
aerial-run ./build.aarch64/cuPHY/examples/pusch_rx_multi_pipe/cuphy_ex_pusch_rx_multi_pipe \
-i testVectors/tutorial/pusch_4cell.yaml -m 1
What to look for:
Test PASS for every cell; about 12 µs per cell, 18.8 µs for the whole slot.AERIAL_SDK.Block Error Rate 0.0000 and MismatchedCRC TBs 0 for cells 0–3.The slot time grows by much less than 4× because the four cells' kernels overlap on the GPU.
NVIDIA's cubb_gpu_test_bench runs every channel of a realistic TDD pattern for N cells at once, using CUDA MPS (one process per channel group), and measures each channel's latency against its budget. Its Python driver sweeps N. Copy the driver to a scratch directory on the local volume, because it writes its results (and, if a run aborts, multi-GB core dumps) next to itself:
mkdir -p /mnt/datadisk/bench && cp -r $AERIAL_SDK/testBenches/perf/. /mnt/datadisk/bench/
cd /mnt/datadisk/bench
aerial-run bash -lc "cd /work && python3 measure.py --yaml cubb_gpu_test_config.yaml --start 1 --cap 4 --slots 120 --device_max_connections 16"
Auto detect GPU enabled: GH200 detected, running on NVIDIA GH200 480GB
Number of active cells: 1+0(8,12,8,90,132,12)
Number of active cells: 2+0(8,12,8,90,132,12)
Number of active cells: 3+0(8,12,8,90,132,12)
Number of active cells: 4+0(8,12,8,90,132,12)
Warning: max cell count 04+00 (DL 4C / UL 4C) passed based on 120 slots run, unknown cell capacity (100% on time for all channels), please try larger cell counts
This quick run takes about 90 s. The numbers in brackets are the GPU SMs given to each channel group. For a capacity figure, use 400 slots and a wider range (about 70 minutes):
aerial-run bash -lc "cd /work && python3 measure.py --yaml cubb_gpu_test_config.yaml --start 10 --cap 24 --slots 400 --device_max_connections 16"
Cell capacity is 21+00 (DL 21C / UL 21C) based on 400 slots run (100% on time for all channels)
Why
--device_max_connections 16. NVIDIA's sample configuration sets 20 connections per CUDA context. The bench opens seven or more contexts, and the GH200's MPS server allows 120 connections in all, so with 20 every run aborts withCUDA_ERROR_MPS_MAX_CONNECTIONS_REACHED. Sixteen fits; NVIDIA's own perf scripts use 8–16.
Per-channel latency distributions for each cell count are in 008_012_008_090_132_012_sweep_graphs_avg_F08.json. To plot one:
aerial-run bash -lc "cd /work && python3 compare.py --filename 008_012_008_090_132_012_sweep_graphs_avg_F08.json --cells 21+0"
pyAerial is cuPHY with a Python interface. The SDK has it built, with its own Python environment (including Jupyter) at /opt/nvidia/cuBB/pyaerial/.venv inside the container. NVIDIA's example notebooks are in /opt/nvidia/cuBB/pyaerial/notebooks. Two run without any extra data: example_ldpc_coding (LDPC encoding and decoding on the GPU) and example_pusch_simulation (a complete PUSCH link simulation over a channel model).
Run both and keep the executed copies, with their plots, in a directory of your own. The SDK on the share is read-only, so the notebooks must write somewhere else: --output-dir /work sends them to the directory you start from:
mkdir -p /mnt/datadisk/pyaerial-run && cd /mnt/datadisk/pyaerial-run
aerial-run bash -lc "cd /work && /opt/nvidia/cuBB/pyaerial/.venv/bin/jupyter nbconvert --to notebook --execute --ExecutePreprocessor.timeout=1800 --output-dir /work /opt/nvidia/cuBB/pyaerial/notebooks/example_ldpc_coding.ipynb /opt/nvidia/cuBB/pyaerial/notebooks/example_pusch_simulation.ipynb"
[NbConvertApp] Converting notebook /opt/nvidia/cuBB/pyaerial/notebooks/example_ldpc_coding.ipynb to notebook
[NbConvertApp] Writing 2501773 bytes to /work/example_ldpc_coding.ipynb
[NbConvertApp] Converting notebook /opt/nvidia/cuBB/pyaerial/notebooks/example_pusch_simulation.ipynb to notebook
[NbConvertApp] Writing 3575821 bytes to /work/example_pusch_simulation.ipynb
Both take about 4 minutes together. Copy the two .ipynb files to your own machine to read them; any notebook viewer shows the cells and plots. The kernel prints a warning that it is "running over TCP without encryption": the notebook kernel and Jupyter talk over localhost inside the container, so this is harmless here.
You have a working setup when:
Test PASS for SSB.nvidia-smi shows less than 1980 MHz), or the SDK is being read from the NFS share instead of the local copy (echo $AERIAL_SDK).GPU times per slot, mean:
| GPU / CPU platform | PUSCH TV 7201 (1 cell) | PDSCH TV 3001 (1 cell) | PDSCH (4 cells, slot) | SSB (1 cell) | SSB (4 cells, slot) | Cell capacity (F08, 400 slots) | Measured |
|---|---|---|---|---|---|---|---|
| NVIDIA GH200 480GB at 1980 MHz + NVIDIA Grace, 72 Arm Neoverse V2 cores / 72 threads | 216 µs | 37 µs | 380 µs | 13.3 µs | 19.2 µs | 21 (10–21 all 100 % on time; 22+ cannot be set up with the stock configuration) | 2026-11-05 · aerial-26-2-gh200 (NFS SDK, local copy) · capacity 2026-11-01 |
Nothing runs in the background: every aerial-run container exits when its command ends. /mnt/datadisk is a local cache and scratch area that any user of the machine (any image) may wipe. Delete /mnt/datadisk/bench and /mnt/datadisk/pyaerial-run when you no longer need them; the SDK copy (/mnt/datadisk/aerial-sdk) may stay as a cache, and Setup re-syncs it if it is gone. Power the node off when you are done:
omf tell -a offh -t srv9.rrail.cosmos-lab.org
| Symptom | Likely cause | Fix |
|---|---|---|
CUDA_ERROR_MPS_MAX_CONNECTIONS_REACHED at every cell count in Step 5 |
the sample configuration's 20 connections per context exceed the GH200's MPS limit of 120 | add --device_max_connections 16 |
cuphy::cuphy_exception ... An invalid argument was provided at 22 cells and above |
the stock configuration cannot set up more than 21 cells | expected; the capacity line still reports 21 |
cuphyHDF5_exception ... No such scalar or structure field |
the test vector was generated by a different Aerial release | use the vectors in the image, or regenerate them with the 26-2 5GModel |
| a core dump of several GB appears in the bench directory | an aborted bench run (both cases above) | delete it |
| times 2–3× higher than the table | GPU clock not locked | systemctl restart nvidia.service, then check nvidia-smi |
| 4-cell PDSCH about twice the table, or jumping between runs | the SDK is read from the NFS share | export AERIAL_SDK=/mnt/datadisk/aerial-sdk (Setup step 5) |
aerial-run prints loading Aerial images from /opt/aerial and waits |
the node has just booted and aerial-images.service is still loading the cuBB image |
wait about two minutes |
aerial-run: image load failed |
/opt/aerial is not mounted (the share was unreachable at boot) |
mount /opt/aerial, then run aerial-run again; it retries the load |
Read-only file system |
something tried to write into the SDK on the share | work in the local copy (/mnt/datadisk/aerial-sdk) or write to /work |
omf load reports failure |
-r 0 missing, or timeouts too short |
use exactly the command in Setup |
testBenches/README.md in the source tree: the multi-cell bench and how its test vectors are made.Author(s): COSMOS team · Last verified: 2026-11-05 on srv9.rrail · Tested image/release: aerial-26-2-gh200.ndz (SDK and containers on the /opt/aerial share) / Aerial 26-2 · Tags: aerial, cuphy, gpu, gh200, 5g