Set up an NVIDIA Tesla V100 server in sb1 for GPU computing, run the Sionna Bench channel-estimation benchmarks on it, and run AI Telco Engineer workers on it against the language model served on a GH200.
Two x86 servers in the sb1 sandbox, srv1-lg1.sb1 and srv2-lg1.sb1, each carry an NVIDIA Tesla V100-PCIE-16GB. The V100 (Volta, compute capability sm_70) is older than the GPUs current software is built for, and that is the main thing this page is about:
pip install torch (and any pip install sionna that pulls it in) installs a PyTorch with no V100 kernels. Every Python environment and every container image must take PyTorch from the cu126 index instead.With those two rules the V100 runs Sionna Bench with the same numerical results as the GH200, and runs AI Telco Engineer (AITE) workers. The language model AITE needs does not fit a 16 GB GPU; it stays on a GH200 in rrail, which the sb1 server reaches over the control network.
Image:
gpu-compute-x86.ndz(2026-11-17) carries the NVIDIA driver, Docker and the container toolkit, so Setup steps 2–3 are needed only on another image. Its load-back test onsrv2-lg1.sb1on 2026-11-17 checked the driver, a GPU container and a PyTorch matrix multiply on the V100. The Sionna Bench and AITE figures on this page were measured earlier the same day on a hand install (steps 2–3 onduranta.ndz), with the same driver and PyTorch versions.
This page is a guide: it is not packaged as a cosmos-tutorial, and you run every step by hand.
After completing this tutorial you will be able to:
sm_70 kernels;| Difficulty | Intermediate |
|---|---|
| Estimated time | install: not timed; Sionna Bench fast tests about 2 min, the builtin stage about 37 min; AITE worker runs: hours |
| Domain / sandbox | sb1 (srv1-lg1.sb1, srv2-lg1.sb1); the language model on a GH200 in rrail |
| Image | gpu-compute-x86.ndz (NVIDIA 580.178.04, Container Toolkit 1.20.2; PyTorch installed at first use) |
| Transmits? | no |
| Last verified | 2026-11-17 on srv2-lg1.sb1: gpu-compute-x86.ndz load-back (driver, GPU container, PyTorch matmul); Sionna Bench and AITE on a hand install on duranta.ndz, driver 580.178.04. Unpackaged guide: no cosmos-tutorial run modes |
Background knowledge
Account & access
sb1 — see Make a reservation. For the AITE part, also a running language-model endpoint on a GH200 and a copy of its API key.Devices / nodes
| Resource | Role | Qty | Notes |
|---|---|---|---|
srv1-lg1.sb1 or srv2-lg1.sb1 |
runs Sionna Bench and the AITE workers | 1 | 2× Intel Xeon Gold 6226 (24 cores, hyperthreading off), 187 GB RAM, Tesla V100-PCIE-16GB on NUMA node 0 (the even-numbered cores); measured on srv2-lg1.sb1 only |
srv8.rrail or srv9.rrail |
language model for AITE | 1 | only for the AITE part; see Serving a large language model |
Disk images
| Image | Load onto | Provides |
|---|---|---|
gpu-compute-x86.ndz |
the V100 server | the NVIDIA driver, Docker and the container toolkit; cosmos-gpu-check and cosmos-gpu-torch-venv |
duranta.ndz |
the V100 server, until the GPU image exists | the base the recipe was verified on: Ubuntu 24.04.4, kernel 6.8.1-1015-realtime (PREEMPT_RT), BIOS boot, no Secure Boot |
Software components
| Component | Version | Source |
|---|---|---|
| NVIDIA driver | 580.178.04 (nvidia-headless-580-server, proprietary DKMS module); reports CUDA 13.0 |
Ubuntu archive |
| NVIDIA Container Toolkit | 1.20.2 | NVIDIA's apt repository |
| PyTorch | 2.11.0+cu126 (CUDA 12.6, architectures sm_50 to sm_90) |
the PyTorch cu126 index |
| Sionna Bench | commit e1d2fdb, with the ext/delfeld2025 submodule |
GitHub, NVlabs/sionna-bench |
| Sionna / Sionna RT | 2.1.0 / 2.2.0 (Sionna Bench); 2.0.1 (AITE host), 2.2.0 (AITE worker image) | PyPI |
| AITE | commit be94f40f |
GitHub, NVlabs/the-ai-telco-engineer |
One sb1 server for the GPU work. Sionna Bench needs nothing else. For AITE, the orchestrator, its workers and their containers run on the V100 server, the embed/rerank server runs on its CPU, and every language-model request goes over the control network to the vLLM server on the GH200 (port 8000, with its API key).
Reserve the server, log into the sb1 console and load the GPU image, then skip to step 4. The image is large, so keep the partition size (-r 0) and give the load time; on 2026-11-17 loads without -v failed now and then on sb1 while loads with -v succeeded, so keep -v and retry once if a load reports a failed node.
ssh <username>@console.sb1.cosmos-lab.org
omf load -v -t srv2-lg1.sb1.cosmos-lab.org -i gpu-compute-x86.ndz -r 0 -o 2400 -b 600 --boot
Steps 2–3 are the hand install, for an image without the driver (they were done on duranta.ndz, whose kernel is PREEMPT_RT).
Driver. Install the 580-server branch with its proprietary DKMS module. The module build refuses a real-time kernel unless IGNORE_PREEMPT_RT_PRESENCE=1 is set; on duranta.ndz's PREEMPT_RT kernel it then built and loaded without problems:
apt-get update
IGNORE_PREEMPT_RT_PRESENCE=1 DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
nvidia-headless-580-server nvidia-utils-580-server nvidia-dkms-580-server
DKMS builds the module only for the running kernel: for any other kernel the image boots, run IGNORE_PREEMPT_RT_PRESENCE=1 dkms autoinstall -k <kernel version>.
Docker GPU access. Install the NVIDIA Container Toolkit, register it with Docker, then reboot once for both steps:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
> /etc/apt/sources.list.d/nvidia-container-toolkit.list
apt-get update && apt-get install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=docker
reboot
The reboot is needed because nouveau is loaded: the driver package blacklists it and rebuilds the initramfs. Docker picks up the new runtime at the same reboot. On duranta.ndz, a reboot or Docker restart leaves the image's OAI CN5G containers stopped (their restart policy is no); that matters only if you also use the 5G core on this server.
Check the driver. On gpu-compute-x86.ndz, cosmos-gpu-check runs this step and the next in one go, and cosmos-gpu-torch-venv /opt/gpu-venv builds a PyTorch 2.11.0 (cu126) virtual environment and checks a matrix multiply on the GPU (about 100 s). By hand:
nvidia-smi
Driver Version: 580.178.04 CUDA Version: 13.0
with one Tesla V100-PCIE-16GB, 16384 MiB, about 35 W idle against a 250 W cap. The kernel log line module verification failed: signature and/or required key missing - tainting kernel is harmless here: these servers boot without Secure Boot.
Check a GPU container.
docker run --rm --gpus all ubuntu:24.04 nvidia-smi -L
GPU 0: Tesla V100-PCIE-16GB
Remove the test image afterwards (docker rmi ubuntu:24.04) if you will save an image.
This guide is not packaged, so there is no cosmos-tutorial run for it. Each part below is independent.
Build the environment on local disk. It uses the same source and package versions as the published GH200 reproduction, except PyTorch, which comes from the cu126 index, and the CUDA 13 packages, which are left out (PyTorch brings its own CUDA 12 libraries):
apt-get install -y python3.12-venv git
mkdir -p /opt/sionna-bench && cd /opt/sionna-bench
git clone https://github.com/NVlabs/sionna-bench.git src && git -C src checkout e1d2fdbeed3186201de1929e88169446583b54a4
git -C src submodule update --init --recursive
python3 -m venv venv && venv/bin/pip install -U pip
venv/bin/pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu126
grep -v -E '^(nvidia-|cuda-toolkit|cuda-bindings|cuda-pathfinder|-e |torch==|triton==)' pip-freeze.txt > requirements-v100.txt
venv/bin/pip install -r requirements-v100.txt && venv/bin/pip install --no-deps -e src
pip-freeze.txt is the package list of the GH200 environment; place it in /opt/sionna-bench before the grep. The submodule step is required: without ext/delfeld2025, pytest collects 408 tests instead of 426. The finished environment is 6.9 GB.
Check that this PyTorch can use the V100:
/opt/sionna-bench/venv/bin/python -c 'import torch; print(torch.__version__, torch.cuda.get_device_name(0), "sm_70" in torch.cuda.get_arch_list())'
The 2026-11-17 install reported torch 2.11.0+cu126, CUDA 12.6 and an architecture list from sm_50 to sm_90, which includes sm_70.
Then run the same stages as on the GH200, from /opt/sionna-bench/src. Pin the jobs to the GPU's NUMA node (the even-numbered cores on these servers), for example with systemd-run -p CPUAffinity="0 2 4 6 8 10 12 14 16 18 20 22":
cd /opt/sionna-bench/src
../venv/bin/pytest -q -p no:cacheprovider -m "not slow and not training and not system_variants"
The fast tests end with 267 passed, 159 deselected. They run on the CPU only.
../venv/bin/python scripts/mimo_ofdm_chest/reproduce_results.py --paper builtin --prepare-assets \
--asset-root /opt/sionna-bench/work/assets --output-dir /opt/sionna-bench/work/out
The builtin stage runs the classical LS and LMMSE estimators. On 2026-11-17 its LMMSE result at 15 dB was BER 2.54e-4, BLER 2.976e-3, NMSE 5.324e-4.
The AITE host part is the same as on the GH200 (its Setup steps 3–6), with three changes for x86 and the V100.
The host environment keeps a Volta-capable PyTorch (install it before the requirements):
mkdir -p /opt/aite/runs && cd /opt/aite
git clone https://github.com/NVlabs/the-ai-telco-engineer.git && git -C the-ai-telco-engineer checkout be94f40f4bdd255b3e4dfa8bf32798821813311b
python3 -m venv venv && venv/bin/pip install -U pip
grep -v faiss the-ai-telco-engineer/requirements.txt > req-host.txt
venv/bin/pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu126
venv/bin/pip install -r req-host.txt faiss-cpu sionna==2.0.1 bs4 sentence-transformers einops fastapi uvicorn
Then create tei_shim.py and aite-run.sh in /opt/aite exactly as on the GH200 page, and start the shim on the CPU (TEI's GPU images do not support Volta, and the CPU is enough):
systemd-run --unit=aite-tei-shim -p CPUAffinity="1 3 5 7 9 11 13 15" --property=WorkingDirectory=/opt/aite \
-E HF_HOME=/opt/aite/hf -E OMP_NUM_THREADS=8 -E EMBED_MODEL=BAAI/bge-base-en-v1.5 \
/opt/aite/venv/bin/uvicorn tei_shim:app --host 127.0.0.1 --port 8001
The documentation index does not depend on the CPU architecture. Copying /opt/aite/runs/api_doc_cache (13 MB) from a GH200 run saves rebuilding it, as long as the embedder is the same (BAAI/bge-base-en-v1.5).
The API key is copied from the GH200 into a 0600 file, never printed. From a machine with root SSH to both servers:
ssh root@srv8.rrail.cosmos-lab.org 'cat /root/llm/api-key' | ssh root@srv2-lg1.sb1.cosmos-lab.org 'mkdir -p /root/llm && umask 077 && cat > /root/llm/api-key'
The worker image must be changed for Volta. Create the run folder first, as in Setup step 6 of the GH200 page (the 2026-11-17 run used R=/opt/aite/runs/otfs_detector-v100-20261117). The task's Dockerfile (FROM nvcr.io/nvidia/pytorch:24.12-py3 plus pip install sionna) now pulls PyTorch 2.15 for CUDA 13.2: no sm_70 kernels, and it needs a newer driver than 580. In the run folder, replace docker/dockerfile_agent_container with:
FROM nvcr.io/nvidia/pytorch:24.12-py3
RUN pip install --no-cache-dir torch==2.11.0 torchvision==0.26.0 --index-url https://download.pytorch.org/whl/cu126 \
&& pip install --no-cache-dir sionna==2.2.0 \
&& python -c "import torch, sionna; print(torch.__version__, sionna.__version__)"
and build it yourself before the run, with host networking (DNS from Docker's bridge network was unreliable on this server):
cd $R/docker && docker build --network host -t agent_otfs_detector -f dockerfile_agent_container .
docker run --rm --gpus all --network none agent_otfs_detector python -c 'import torch, sionna; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0), sionna.__version__)'
2.11.0+cu126 True Tesla V100-PCIE-16GB 2.2.0
The image is 44 GB on disk.
Run configuration. As on the GH200 page, with base_url pointing at the GH200's control-network address, and fewer workers for the 16 GB GPU: num_workers 2, num_gpus 1, population_size 4, num_ideas 2, timeout 3600, hyperparameter_tuner {n_trials 10, timeout 600}. Launch it with aite-run.sh as a systemd-run unit pinned to the GPU's NUMA node, exactly as on the GH200 page.
cu126 builds include sm_70; the CUDA 13 builds (PyPI's default, cu130 and later) do not, and they also need a newer driver. A Python package that depends on torch silently replaces a pinned build with PyPI's unless the pinned one is installed first, which is why every recipe here installs PyTorch before anything else.nvidia-smi shows the V100 with driver 580.178.04, the container check prints Tesla V100-PCIE-16GB, the Sionna fast tests end with 267 passed, and the worker image check prints 2.11.0+cu126 True Tesla V100-PCIE-16GB 2.2.0.
Sionna Bench, same commands on both platforms. Wall time, with average (maximum) GPU utilization and GPU memory:
| Platform | Fast tests (267 pass) | examples.run_custom_benchmark |
papers.delfeld2025.run_benchmark |
reproduce_results.py --paper builtin --prepare-assets |
Measured |
|---|---|---|---|---|---|
| 2× Intel Xeon Gold 6226, 24 cores / 24 threads + NVIDIA Tesla V100-PCIE-16GB | 125 s (CPU only) | 71 s, 63 % (74), 1.0 GB | 147 s, 83 % (97), 1.5 GB | 2218 s (LMMSE 2039 s), 69 % (100), 11.8 GB, 113 W | 2026-11-17, duranta.ndz + this recipe, driver 580.178.04, torch 2.11.0+cu126 |
| NVIDIA GH200 480GB: 72 Arm Neoverse V2 cores + Hopper GPU, 96 GB HBM3 | 124 s (2026-11-10), 103–112 s (2026-11-14) | 40 s, 40 % (46), 1.4 GB | 61 s, 59 % (71), 2.0 GB | 948 s (2026-11-14) | gpu-compute-gh200, torch 2.11.0+cu130 |
The V100 is about 2.4 times slower on the GPU-bound evaluation stages and the same on the CPU-only tests. The numbers agree: builtin LMMSE at 15 dB gives BER 2.54e-4 / BLER 2.976e-3 / NMSE 5.324e-4 on the V100 and 2.93e-4 / 2.976e-3 / 5.325e-4 on the GH200; delfeld2025 at 15 dB gives BLER 1.984e-3 on both and NMSE 2.373e-4 vs. 2.369e-4.
AITE otfs_detector, generation 0 only. The V100 run shared the GH200's language model with a run on the GH200 itself, and was stopped after one generation when that model was shut down. It is a check that the pipeline works on a V100, not a benchmark:
| Platform | Workers | Result | GPU use | Measured |
|---|---|---|---|---|
| 2× Intel Xeon Gold 6226, 24 cores / 24 threads + NVIDIA Tesla V100-PCIE-16GB; model on a GH200 | 2 at a time, population 4, timeout 3600 s | 1 candidate succeeded, with 3 Pareto points: NVE 2.216 at 60.0 ms per batch, 9.14 at 30.9 ms, 69.7 at 9.7 ms (1.0 = the EP baseline; lower is better). One candidate timed out; two were stopped before their first evaluation | average utilization 8 %, peak memory 9.5 GB of 16 GB | 2026-11-17, hand install on duranta.ndz |
Idea generation took 19 minutes, against 6 minutes when the GH200 run had the model to itself. Token use was not measured: AITE does not count tokens, and the model's counters were shared with the other run. Complexity (ms per batch) depends on the GPU, so compare it only with runs on the same GPU.
systemctl stop aite-otfs aite-tei-shim
rm -f /root/llm/api-key
Copy anything you want to keep from /opt/sionna-bench and /opt/aite/runs off the server. Before saving an image, remove the API key copy, the test image (docker rmi ubuntu:24.04) and the AITE worker image (docker rmi agent_otfs_detector, 44 GB).
| Symptom | Likely cause | Fix |
|---|---|---|
| the driver install fails to build the module on a real-time kernel | the NVIDIA module build refuses PREEMPT_RT | set IGNORE_PREEMPT_RT_PRESENCE=1 (Setup step 2) |
nvidia-smi finds no device after installing an -open driver |
the open kernel modules do not support Volta | install the proprietary nvidia-dkms-580-server |
no kernel image is available for execution on the device, or torch.cuda.is_available() is False |
a CUDA 13 PyTorch from PyPI (no sm_70, needs a newer driver) |
reinstall torch==2.11.0 from the cu126 index, before any package that depends on it |
pip inside docker build fails with Name or service not known |
unreliable DNS from Docker's bridge network on this server | docker build --network host |
| pytest collects 408 tests, not 426 | the ext/delfeld2025 submodule is missing |
git -C src submodule update --init --recursive |
| AITE retries a request forever | the language model on the GH200 was stopped | keep the endpoint running until every AITE run that uses it has finished; agree on it with whoever else uses it |
CUDA out of memory in a worker's evaluation |
candidates can allocate tens of GB; the V100 has 16 GB | 2 workers per GPU, or PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True |
| candidates using bfloat16 or TF32 are slow or fail | the V100 has neither | expect the agents to fall back; nothing to fix on the server |
cu126 index) and NVIDIA Container ToolkitAuthor(s): COSMOS team · Last verified: 2026-11-17 on srv2-lg1.sb1 (gpu-compute-x86.ndz load-back; Sionna Bench and AITE on a hand install on duranta.ndz; unpackaged guide) · Tested image/release: gpu-compute-x86-20261117.ndz; duranta.ndz + NVIDIA 580.178.04, Container Toolkit 1.20.2, torch 2.11.0+cu126, Sionna Bench e1d2fdb, AITE be94f40f · Tags: gpu, v100, cuda, sionna, aite