Run NVIDIA's AI Telco Engineer (AITE), where language-model agents write, test and tune PHY-layer code, on one GH200 server that also serves the language model, so no prompt or code leaves COSMOS.
The AI Telco Engineer (AITE) is an open-source agentic framework from NVIDIA Research. You give it a task, such as "design an OTFS detector that beats this baseline", and an evaluation harness. A manager agent proposes ideas, worker agents turn each idea into PyTorch/Sionna code inside their own Docker container, run the harness, tune hyperparameters with Optuna, and report back. Each generation is scored on a leaderboard, and the manager refines the workers' prompt for the next one. The repository ships two tasks, otfs_detector and pilotless.
AITE needs an OpenAI-compatible language-model endpoint. On COSMOS that endpoint is our own: DeepSeek V4-Flash served by vLLM on the same GH200, as in Serving a large language model. One GH200 does both jobs: vLLM takes 70 % of the GPU memory and the AITE evaluations share the rest.
This page is a guide: it is not packaged as a cosmos-tutorial, and you run every step by hand. It records the run of 2026-11-17, a much smaller configuration than the paper's (4 workers on 1 GPU and 3 generations, against 16 workers on 16 GPUs and up to 300 generations with GPT-5.5). Read the results honestly: the pipeline works end to end with our model, but in this budget its best detector only matched the task's baseline, at about 70 times the run time of the solution the paper reports.
After completing this tutorial you will be able to:
| Difficulty | Advanced |
|---|---|
| Estimated time | install: not timed; language model start: 12 min; first-run index and worker image: about 13 min; the run shown here: 3 h 21 min |
| Domain / sandbox | rrail (srv8.rrail or srv9.rrail) |
| Image | gpu-compute-gh200.ndz |
| Transmits? | no |
| Last verified | 2026-11-17, gpu-compute-gh200-20261114b on srv8.rrail, AITE be94f40f, vLLM v0.33.0. Unpackaged guide: no cosmos-tutorial run modes |
Background knowledge
Account & access
Devices / nodes
| Resource | Role | Qty | Notes |
|---|---|---|---|
srv8.rrail or srv9.rrail |
language model, AITE orchestrator, workers and their containers | 1 | NVIDIA GH200 480GB: 72 Arm Neoverse V2 cores, 96 GB HBM3 GPU memory, 480 GB LPDDR5X; needs the 7.8 TB data disk |
Disk images
| Image | Load onto | Provides |
|---|---|---|
gpu-compute-gh200.ndz |
the GH200 server | NVIDIA driver 610.57.04, Docker with the NVIDIA Container Toolkit, the shared-content tools |
Software components
| Component | Version | Source |
|---|---|---|
| AITE | commit be94f40f4bdd255b3e4dfa8bf32798821813311b (2026-07-21) |
GitHub, NVlabs/the-ai-telco-engineer |
| Language model | deepseek-ai/DeepSeek-V4-Flash-0731, revision 7872f01b, served as deepseek-v4-flash |
shared content llm, see the LLM page |
| vLLM | vllm/vllm-openai:v0.33.0 (arm64) |
shared content llm |
| AITE host environment | Python 3.12 venv: torch 2.15.0+cu132, sionna 2.0.1, faiss-cpu 1.16.0, langchain 1.3.2, optuna 4.9.0, sentence-transformers 6.3.0 | PyPI, installed in Setup |
| Embedder / reranker | BAAI/bge-base-en-v1.5 / cross-encoder/ms-marco-MiniLM-L6-v2, on the CPU |
Hugging Face, through the small server in Setup step 4 |
| Worker image | nvcr.io/nvidia/pytorch:24.12-py3 + sionna 2.2.0 (which brings torch 2.15.0+cu132) |
built by AITE from the task's Dockerfile |
One server, no radio. Everything below runs on the GH200:
tei_shim.py) runs on 8 CPU cores, bound to 127.0.0.1:8001. AITE's Sionna-documentation search tool uses it./opt/aite/venv. Each worker drives its own Docker container, which runs the candidate code on the same GPU, in the roughly 28 GB vLLM leaves free.Load and start the GPU-compute base as in its tutorial, Setup steps 1–3. Then enable and copy the llm project as in the LLM page, Setup step 2.
Start the language model with 70 % of the GPU memory, so that the AITE evaluations fit beside it, and with a 262,144-token context. The start script reads both from the environment:
systemd-run --unit llm-serve --collect -E GPU_UTIL=0.70 -E MAXLEN=262144 bash /opt/content/llm/bin/llm-serve.sh v4flash
journalctl -u llm-serve -f
Wait for Application startup complete. (about 12 minutes on the first start). The 2026-11-17 run did not use this exact command: it started the same vLLM version and model revision with a local copy of the start script, GPU_UTIL=0.70 and MAXLEN=131072. With 131,072 tokens one worker's conversation outgrew the context (see Troubleshooting), so use 262,144.
Install AITE and its host environment. faiss-gpu-cu12 has no arm64 wheel, so it is replaced by faiss-cpu (the index is small):
apt-get install -y python3.12-venv
mkdir -p /opt/aite/runs && cd /opt/aite
git clone https://github.com/NVlabs/the-ai-telco-engineer.git
git -C the-ai-telco-engineer checkout be94f40f4bdd255b3e4dfa8bf32798821813311b
python3 -m venv /opt/aite/venv && /opt/aite/venv/bin/pip install -U pip
grep -v faiss the-ai-telco-engineer/requirements.txt > /tmp/aite-req.txt
/opt/aite/venv/bin/pip install -r /tmp/aite-req.txt faiss-cpu sionna==2.0.1 bs4
/opt/aite/venv/bin/pip install sentence-transformers einops fastapi uvicorn
The host only uses Sionna to extract its API documentation for the search index, so the CPU is enough here.
Start the embed/rerank server. AITE's documentation tool speaks the native TEI API (POST /embed, POST /rerank), and the task refuses to start without an embedder. TEI publishes no arm64 image, so this small server provides the same two endpoints with sentence-transformers. Save it as /opt/aite/tei_shim.py:
#!/usr/bin/env python3
"""Minimal TEI-compatible embedding + rerank server for AITE's SionnaDoc tool.
AITE's SionnaDoc client speaks the native TEI API, not the OpenAI one:
POST /embed {"inputs": [str, ...]} -> [[float, ...], ...]
POST /rerank {"query": str, "texts": [str,...]} -> [{"index": i, "score": s}, ...]
TEI has no arm64 image, so this serves the same two models with sentence-transformers.
Env: EMBED_MODEL (default nomic-ai/nomic-embed-text-v1.5), RERANK_MODEL
(default cross-encoder/ms-marco-MiniLM-L6-v2), DEVICE (cpu|cuda, default cpu).
Run: uvicorn tei_shim:app --host 127.0.0.1 --port 8001
"""
import os
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from sentence_transformers import CrossEncoder, SentenceTransformer
EMBED_MODEL = os.environ.get("EMBED_MODEL", "nomic-ai/nomic-embed-text-v1.5")
RERANK_MODEL = os.environ.get("RERANK_MODEL", "cross-encoder/ms-marco-MiniLM-L6-v2")
DEVICE = os.environ.get("DEVICE", "cpu")
MAX_INPUTS = 256
embedder = SentenceTransformer(EMBED_MODEL, device=DEVICE, trust_remote_code=True)
reranker = CrossEncoder(RERANK_MODEL, device=DEVICE)
app = FastAPI()
class EmbedRequest(BaseModel):
inputs: list[str] | str
class RerankRequest(BaseModel):
query: str
texts: list[str]
def _check_size(items: list[str]) -> None:
if not items or len(items) > MAX_INPUTS:
raise HTTPException(status_code=413, detail=f"need 1..{MAX_INPUTS} inputs")
@app.post("/embed")
def embed(req: EmbedRequest) -> list[list[float]]:
items = [req.inputs] if isinstance(req.inputs, str) else req.inputs
_check_size(items)
return embedder.encode(items, normalize_embeddings=True).tolist()
@app.post("/rerank")
def rerank(req: RerankRequest) -> list[dict]:
_check_size(req.texts)
scores = reranker.predict([(req.query, t) for t in req.texts])
ranked = sorted(enumerate(scores), key=lambda p: -float(p[1]))
return [{"index": i, "score": float(s)} for i, s in ranked]
@app.get("/health")
def health() -> dict:
return {"embed": EMBED_MODEL, "rerank": RERANK_MODEL, "device": DEVICE}
Then run it as a service with the embedder AITE was verified with:
systemd-run --unit=aite-tei-shim --property=WorkingDirectory=/opt/aite -E HF_HOME=/opt/aite/hf \
-E OMP_NUM_THREADS=8 -E EMBED_MODEL=BAAI/bge-base-en-v1.5 \
/opt/aite/venv/bin/uvicorn tei_shim:app --host 127.0.0.1 --port 8001
The paper's embedder, nomic-embed-text-v1.5, does not load under transformers 5.23 ('NomicBertModel' object has no attribute 'get_extended_attention_mask'), so this guide uses BAAI/bge-base-en-v1.5. The reranker is the paper's.
Save the launcher as /opt/aite/aite-run.sh (chmod 755). It passes the API key to AITE from the key file, so the key never appears on a command line:
#!/bin/bash
# Run one AITE task folder against the local vLLM endpoint. Usage: aite-run.sh <task-run-folder>
# The key is read from the 0600 file and exported only into this process.
set -euo pipefail
TASK="${1:?usage: aite-run.sh <task-run-folder>}"
AITE="${AITE:-/opt/aite/the-ai-telco-engineer}"
KEY_FILE="${KEY_FILE:-/root/llm/api-key}"
[ -r "$KEY_FILE" ] || { echo "missing $KEY_FILE" >&2; exit 1; }
[ -f "$TASK/config.json" ] || { echo "no config.json in $TASK" >&2; exit 1; }
export MODEL_API_KEY="$(cat "$KEY_FILE")"
cd "$TASK"
exec /opt/aite/venv/bin/python -u "$AITE/launch.py" "$TASK"
Copy the task into a run folder, so that every output of the run stays under /opt/aite/runs:
R=/opt/aite/runs/otfs_detector-$(date -u +%Y%m%d)
cp -a /opt/aite/the-ai-telco-engineer/tasks/otfs_detector $R
ip -4 -o addr show CTRL | awk '{print $4}' | cut -d/ -f1 # the address vLLM listens on
Edit $R/config.json with any editor. These are the values the 2026-11-17 run ended with; put the address printed above into both base_url lines:
"num_workers": 4, "num_gpus": 1, "population_size": 4, "num_ideas": 2,
"timeout": 3600,
"hyperparameter_tuner": {"n_trials": 10, "timeout": 600},
"tools_config": {
"eval_timeout": 300,
"sionna_doc_config": {
"cache_dir_path": "/opt/aite/runs/api_doc_cache",
"embedding_model": "BAAI/bge-base-en-v1.5",
"embedding_base_url": "http://127.0.0.1:8001",
"reranker_model": "cross-encoder/ms-marco-MiniLM-L6-v2",
"reranker_base_url": "http://127.0.0.1:8001",
"retrieve_k": 12, "rerank_top_n": 4,
"summarize_llm": {}
}
},
"agent_llm": {"model": "deepseek-v4-flash", "base_url": "http://<CTRL address>:8000/v1/", "temperature": 1.0, "top_p": 1.0, "model_kwargs": {}},
"manager_llm": {"model": "deepseek-v4-flash", "base_url": "http://<CTRL address>:8000/v1/", "temperature": 1.0, "top_p": 1.0, "model_kwargs": {}}
summarize_llm {} turns off the summaries of the 34 Sionna tutorials, which would otherwise be 34 sequential calls to a reasoning model; the tutorials are indexed as plain markdown instead. Set num_generations to the number of generations you want.
This guide is not packaged, so there is no cosmos-tutorial run for it. Start AITE and a GPU logger as services, so they survive your SSH session:
systemd-run --unit=aite-otfs --property=StandardOutput=append:$R/launch.log --property=StandardError=append:$R/launch.log \
--property=KillSignal=SIGINT --property=TimeoutStopSec=120 /opt/aite/aite-run.sh $R
systemd-run --unit=aite-gpumon nvidia-smi --query-gpu=timestamp,memory.used,power.draw,utilization.gpu \
--format=csv,noheader -l 10 -f $R/gpu.csv
tail -f $R/launch.log
On the first run AITE builds the documentation index (about 6 minutes on 8 Grace cores; it is cached in /opt/aite/runs/api_doc_cache for later runs) and the worker image from the task's docker/dockerfile_agent_container (6–7 minutes). The NGC PyTorch image is multi-arch, so the Dockerfile builds on arm64 unchanged.
To watch the leaderboard in a browser, run AITE's viewer from the clone and open port 8002 through an SSH tunnel:
cd /opt/aite/the-ai-telco-engineer
/opt/aite/venv/bin/python scripts/leaderboard/serve.py --workspace $R/workspaces --port 8002
To stop the run, systemctl stop aite-otfs. The unit sends SIGINT, which AITE handles as a graceful Ctrl-C.
evaluate tool, which runs the harness in its container. Optuna then tunes the candidate's hyperparameters. The manager reads the results and refines the workers' prompt for the next generation.otfs_detector the score is the NVE: the candidate's block error rate divided by that of the task's expectation-propagation (EP) baseline detector. 1.0 matches the baseline, lower is better. The complexity is the time in milliseconds for one llr batch on the GPU the evaluation ran on. Each candidate gives a Pareto front of (NVE, complexity) points, one per hyperparameter setting.The 2026-11-17 run (otfs_detector, 4 workers, launched at 03:49Z, finished at 07:10Z, 3 h 21 min):
| Phase | Time (UTC) | Outcome |
|---|---|---|
| Documentation index | 03:49–03:55 | 1310 API and 1928 tutorial chunks, 3035 kept, embedded on the CPU |
| Generation 0, worker timeout 1800 s | 03:55–04:34 | 0 of 4 succeeded. Each worker spent about 25 minutes reading and self-testing in long reasoning turns, managed one evaluation, got a block error rate of 1.0 everywhere (its LLRs and labels did not match) and timed out |
| Generation 1, resumed with timeout 3600 s | 04:34–05:56 | 4 of 4 succeeded, with the prompt refined from generation 0; 20 evaluations in total |
| Generation 2 | 05:56–07:10 | 3 of 4 succeeded. One worker failed with HTTP 400: its conversation had passed the 131,072-token context |
At the end the leaderboard ($R/workspaces/leaderboard.json) lists 12 candidates, 7 successful, in 6 idea clusters.
Print the leaderboard as a table. Save this as /opt/aite/leaderboard_summary.py:
#!/usr/bin/env python3
"""Print an AITE leaderboard.json as a table: one row per (candidate, HP config). Usage: leaderboard_summary.py <leaderboard.json>"""
import json
import sys
def main(path: str) -> None:
with open(path) as f:
board = json.load(f)
print(f"candidates={board.get('total_candidates')} successful={board.get('successful_candidates')} clusters={board.get('num_clusters')}")
for cid, desc in (board.get("cluster_descriptions") or {}).items():
print(f"cluster {cid}: {str(desc)[:150]}")
rows = []
for cands in board["clusters"].values():
for c in cands:
if not c.get("success"):
rows.append((c["workspace_id"], c["cluster"], "FAIL", "", (c.get("error") or "")[:70]))
continue
for hp in c.get("hp_results") or []:
rows.append((c["workspace_id"], c["cluster"], hp.get("metric"), hp.get("complexity"), ""))
for r in sorted(rows, key=lambda r: (r[2] == "FAIL", r[2] if r[2] != "FAIL" else 0)):
print(*r, sep="\t")
if __name__ == "__main__":
main(sys.argv[1])
/opt/aite/venv/bin/python /opt/aite/leaderboard_summary.py $R/workspaces/leaderboard.json | head -8
candidates=12 successful=7 clusters=6
cluster 0: Approach 1: Sparse iterative soft-interference-cancellation MMSE with QAM posterior moment matching. Set Equalizer subclass channel_form='sparse', top
...
gen01-0007 3 0.9639 200.5
gen01-0004 2 1.63 116.4
gen01-0006 2 1.7957 666.4
The columns are candidate, idea cluster, NVE and ms per batch. Your candidates will differ: the model samples at temperature 1.0.
Results of the 2026-11-17 run. Complexity is measured on the GPU that ran the evaluation, so it compares only within one platform. The paper's own solution was re-scored with the same harness on the same GPU as the reference:
| Platform | Candidate | Idea | NVE (lower is better; 1.0 = EP baseline) | ms per batch | Measured |
|---|---|---|---|---|---|
| NVIDIA GH200 480GB (72 Arm Neoverse V2 cores + Hopper GPU, 96 GB HBM3) | gen01-0007 | sparse soft interference cancellation / Gaussian message passing | 0.9639 (equals the baseline) | 200.5 | 2026-11-17, gpu-compute-gh200-20261114b |
| same | gen01-0004 | sparse CG-LMMSE | 1.63 | 116.4 | same |
| same | gen02-0010 | sparse CG-LMMSE (fresh idea) | 1.88 / 1.94 / 2.28 | 153.4 / 129.0 / 87.7 | same |
| same | gen02-0009 | non-Gaussian row-message detector | 14.3 | 47.3 | same |
| same | reference: the paper's selected solution, re-scored | 0.6386 | 2.84 | same |
So in 3 generations with 4 workers, our model reached the baseline but not the paper's solution, which is both more accurate and about 70 times faster.
The pilotless task is not verified. Its harness (eval.py) scores the paper's own solution at NVE 109.53 (complexity 6.0e-03), with Sionna 2.0.1 and 2.2.0 alike, while the paper evaluates that solution with a goodput sweep instead. Until that mismatch is understood, a pilotless leaderboard cannot be read against the paper. This is an open item.
Language-model throughput on the same server (vLLM's own counters and a benchmark before the run):
| Platform | Single-stream decode | Prefill, 21k-token prompt | 4 concurrent streams | Tool call | Measured |
|---|---|---|---|---|---|
| NVIDIA GH200 480GB (72 Arm Neoverse V2 cores + Hopper GPU, 96 GB HBM3), vLLM at 70 % of GPU memory | 68.1 tok/s (first token 1.55 s) | 7,768 tok/s (first token 2.72 s) | 120.1 tok/s in total | OK, 1.9 s | 2026-11-17, gpu-compute-gh200-20261114b, vLLM v0.33.0 |
During the AITE run the model sustained 100–112 tok/s in total with 3–4 streams and 67 tok/s with one. The whole run used 28.2 M prompt tokens (96.3 % served from vLLM's prefix cache), 1.07 M generated tokens and about 775 requests. The GPU stayed at about 99 % utilization (the model never stops decoding), averaged 266 W (peak 497 W), and its memory peaked at 97.3 GB, which is full: three worker scratch runs hit CUDA out-of-memory errors, and the agents rewrote their code.
A standalone check of the evaluation harness, outside AITE: the paper's OTFS solution returns SUCCESS, 0.6386, 2.843 in 18 s with a peak of 2.5 GB of GPU memory.
systemctl stop aite-otfs aite-gpumon aite-tei-shim
systemctl stop llm-serve
docker rm -f llm-serve
docker ps -a # no worker containers should be left
Copy what you want to keep from /opt/aite/runs off the server: the system disk is wiped by the next image load, and the data disk is not storage (see GPU computing). Before saving an image, delete /root/llm/api-key and remove the worker image (docker rmi agent_otfs_detector, about 37 GB).
| Symptom | Likely cause | Fix |
|---|---|---|
| every worker of the first generation times out | a reasoning model at about 30 tok/s per stream needs more than 1800 s per worker | set "timeout": 3600 or more |
| a worker fails with HTTP 400 | its conversation passed the model's context length | start vLLM with MAXLEN=262144 (Setup step 2) |
CUDA out of memory in a worker's evaluation |
candidates allocated up to 24–26 GB each while 4 shared about 28 GB | fewer workers, or PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and a memory-fraction rule in the task prompt |
| the orchestrator retries forever | the language-model endpoint was stopped during the run | keep llm-serve up until AITE has finished, including its manager summary; stop AITE first |
pip finds no faiss-gpu-cu12 |
no arm64 wheel | install faiss-cpu (Setup step 3) |
| the task refuses to start, empty embedding configuration | no embedder configured or the shim is not running | Setup step 4, then systemctl status aite-tei-shim |
'NomicBertModel' object has no attribute 'get_extended_attention_mask' |
the paper's embedder does not load under transformers 5.23 | EMBED_MODEL=BAAI/bge-base-en-v1.5; the index cache is specific to the embedder, so delete api_doc_cache when you change it |
| the worker image has a different Sionna from the host | the Dockerfile installs an unpinned sionna (it resolved 2.2.0; the host has 2.0.1) |
pin sionna==2.0.1 in the run folder's docker/dockerfile_agent_container for reproducible runs |
401 Unauthorized from the model |
missing key | aite-run.sh reads /root/llm/api-key; check that it exists and is readable by root |
Author(s): COSMOS team · Last verified: 2026-11-17 on srv8.rrail (unpackaged guide) · Tested image/release: gpu-compute-gh200-20261114b, AITE be94f40f, vLLM v0.33.0, DeepSeek-V4-Flash-0731 7872f01b · Tags: gpu, gh200, llm, aite, sionna, agents