Bring up a full 5G Standalone network across two server-class COSMOS nodes — an Aether OCUDU (srsRAN-Project–based) gNB on one host driving a USRP, and an Aether SD-Core 5G core on the other — with all inter-machine O-RAN traffic (N2/N3) and the network-SDR fronthaul carried on the domain's 10 GbE data plane, and a real Quectel UE attaching over the air. Runs on SB1 or the grid.
For what SD-Core is made of, and the operational rules that govern it, see the Aether SD-Core framework page. For the RAN side see OCUDU (srsRAN).
This tutorial deploys a production-shaped, split 5G SA system from two prebuilt baremetal images: a gNB host (OCUDU container + SDR) and a core host (RKE2 + SD-Core). It is the two-machine counterpart to the single-node tri-stack tutorial — here the RAN and the 5G core live on separate servers and talk to each other over a dedicated high-speed data plane, which is how a real gNB↔core (N2/N3) deployment is wired.
The defining design choice is that every inter-machine flow runs on the data plane, not the control/management network: the AMF (N2/NGAP) and UPF (N3/GTP-U) endpoints, and — when you use a network USRP (N310/X310) — the radio fronthaul too, all sit on the 10 GbE plane with jumbo frames (MTU 9000). The control network is used only for SSH/management.
You can drive the gNB with either a USB B210 (local) or a network N310/X310 (reachable over the data plane). The core, the UE, and the carrier configuration are identical either way, so you switch radios with a single script.
Because the images hardcode one domain's IPs, this tutorial uses symbolic addresses (CORE_DATA_IP, GNB_DATA_IP, …) that you fill in from your sandbox's domain page. An appendix lists every per-domain value to set.
After completing this tutorial you will be able to:
| Difficulty | Advanced |
| Estimated time | 2–3 hours |
| Domain / sandbox | SB1 (sb1.cosmos-lab.org) or the grid — two server nodes |
| Topic group | 4G/5G & O-RAN |
| Last verified | 2026-06-21 (OCUDU rel-0.7.0 gNB + SD-Core 4.0.0, network N310, band n78) |
Background knowledge
kubectl get pods), Helm, and your sandbox's IP-plane layout — see your domain page.Account & access
Devices / nodes (names are illustrative — pick two servers + one UE node from your reservation)
| Resource | Role | Qty | Notes |
|---|---|---|---|
Core host (e.g. srv1) |
SD-Core 5GC | 1 | RKE2 single-node k8s; data-plane NIC on the 10 GbE plane |
gNB host (e.g. srv2) |
OCUDU gNB | 1 | runs the ocudu-gnb container (host networking); data-plane NIC on 10 GbE |
UE node (e.g. node1) |
Quectel UE | 1 | Quectel RM520N-GL modem (wwan0), AT on /dev/ttyUSB2 |
| USRP | radio | 1 | B210 (USB3 on the gNB host) or N310/X310 (network, reachable over the data plane) |
Disk images (shared on /mnt/images, load with omf load)
| Image | Load onto | Provides |
|---|---|---|
aether-ocudu-0.7.0-data2-20260621.ndz |
gNB host | OCUDU gNB container + configs + start/radio scripts + data-plane networkd drop-ins |
aether-sdcore-4.0.0-data2-20260621.ndz |
core host | RKE2 + SD-Core 4.0.0 (Helm) + macvlan/iptables services + core scripts |
Software components
| Component | Version (in image) | Source |
|---|---|---|
| OCUDU gNB (srsRAN-Project–based) | aetherproject/ocudu:rel-0.7.0 → gnb 26.04.0 |
Aether docs · srsRAN |
| UHD (in the OCUDU image) | 4.7.0.0 | Ettus UHD |
| Aether SD-Core | Helm chart 4.0.0 (oci://ghcr.io/omec-project/sd-core); AMF rel-3.0.0, SMF rel-4.0.0, UPF rel-2.4.2 |
omec-project · SD-Core docs |
| Aether OnRamp (deployer) | Ansible + RKE2 + Helm (/home/native/onramp) |
aether-onramp |
| USRP B210 / N310 / X310 | UHD type=b200 / n3xx / x300 |
Ettus KB |
| Quectel RM520N-GL UE | quectel-CM |
Quectel forums |
Spectrum / RF — band n78 TDD, dl_arfcn 627340 = 3410.1 MHz, 20 MHz, SCS 30 kHz, PCI 1, PLMN 00101, TAC 1, SSB ARFCN 626976, SIM IMSI 001010000000033. Use only your reserved spectrum; for OTA operation follow the sandbox's RF/antenna rules.
Two servers plus a UE node; all O-RAN traffic on the data plane (jumbo, MTU 9000):
data plane (10 GbE, MTU 9000)
[ core host ]───────────────────────────[ gNB host ]───USRP (B210 USB | N310/X310 net)
SD-Core 5GC N2 (NGAP) / N3 (GTP-U) OCUDU gnb │ OTA
AMF=CORE_DATA_IP binds GNB_DATA_IP ▼
UPF macvlans [ Quectel UE on UE node ]
│ management/control network (SSH only) │
Fill in for your domain (from the domain page):
| Symbol | Meaning | SB1 example (data2) |
|---|---|---|
CORE_DATA_IP |
core-host data-plane IP (AMF/N2, UPF/N3) | 10.39.1.1 |
GNB_DATA_IP |
gNB-host data-plane IP (gNB bind) | 10.39.1.5 |
DATA_GW |
data-plane gateway | 10.39.0.1 |
RADIO_DATA_IP |
network USRP data IP (N310/X310 only) | per your SDR reservation |
CORE_CTRL_IP |
core-host control IP (RKE2 node-ip) |
10.37.1.1 |
Private subnets are fixed and do not change between domains: UPF access
192.168.252.0/24, UPF core192.168.250.0/24, UE pool192.168.100.0/24.
ssh <username>@console.<sandbox>.cosmos-lab.org
omf tell -a offh -t <core-host>,<gnb-host>,<ue-node>
omf load -i aether-sdcore-4.0.0-data2-20260621.ndz -t <core-host>
omf load -i aether-ocudu-0.7.0-data2-20260621.ndz -t <gnb-host>
omf tell -a on -t <core-host>,<gnb-host>,<ue-node>
omf stat -t <core-host>,<gnb-host>,<ue-node>
CORE_DATA_IP, GNB_DATA_IP, DATA_GW, CORE_CTRL_IP and (for a network USRP) RADIO_DATA_IP. See Appendix — Porting to another domain; if you load these on the same domain they were saved on, skip this step.The core autostarts on boot (RKE2 + Helm). Verify or (re)start it:
ssh root@<core-host>.<sandbox>.cosmos-lab.org /usr/local/sbin/core-start.sh
# healthy = all NFs Running; amf service externalIP = CORE_DATA_IP
Stop/scale it down with /usr/local/sbin/core-stop.sh (scales NF deployments to 0; keeps mongodb/kafka). Confirm AMF reachability: kubectl get svc amf -n aether-5gc shows externalIP CORE_DATA_IP.
ssh root@<gnb-host>.<sandbox>.cosmos-lab.org
/usr/local/sbin/ocudu-use-n310.sh # network N310/X310 over the data plane (default)
# or
/usr/local/sbin/ocudu-use-b210.sh # local USB B210
# generic start/stop of whatever radio is configured:
/usr/local/sbin/ocudu-start.sh / /usr/local/sbin/ocudu-stop.sh
Each radio script writes the correct ru_sdr block, asserts the data-plane networking (MTU 9000, N3/SDR routes, socket buffers, NUMA pinning), starts the gNB, and runs the NG-Setup guard. Verify:
docker logs --tail 30 ocudu-gnb 2>&1 | grep -iE 'Cell|N2:|NG Setup'
docker logs --since 10s ocudu-gnb 2>&1 | grep -c 'PRACH request late' # network USRP: expect 0
expert_phy.max_proc_delay: 8 absorbs that latency (raise to 10–12 if PRACH request late reappears). For an X310 set ru_sdr.device_args to type=x300,… and a valid srate (e.g. 30.72)./usr/local/sbin/b210-reset.sh (a hard FX3 hang needs a physical replug).ssh root@<ue-node>.<sandbox>.cosmos-lab.org
/root/modem-pdu-on-reg.sh # NV reset + SA-only + PDU on the INITIAL reg connection + keepalive
ip -4 addr show wwan0 | grep inet ; ping -I wwan0 -c4 8.8.8.8
The ocudu-data-keep watchdog then keeps the session up. If attach is flaky (registers then drops / ends in limited-service), clear the AMF UE context and re-run — see Troubleshooting.
You have a working 5G SA system when:
kubectl get pods -n aether-5gc shows all NFs Running (UPF upf-0 5/5); amf externalIP = CORE_DATA_IP.docker logs ocudu-gnb shows the cell up and NG Setup successful (NGAP established to the AMF).wwan0 has an IP from 192.168.100.0/24 and ping -I wwan0 8.8.8.8 is 0% loss.ping -M do -s 8972 RADIO_DATA_IP succeeds and PRACH request late count stays 0.Free the nodes when done. If you modified an image and want to keep it, re-save it — and you must run /root/prepare.sh on the node first (it resets netplan/hostname and wipes /tmp+logs; it does not touch /root, /etc/systemd/network/, or /etc/sysctl.d/, which must persist):
ssh root@<node> "nohup bash /root/prepare.sh >/root/prepare.log 2>&1 &" # detached: netplan reset can drop SSH
# wait for "ready to save" in /root/prepare.log, then on the console:
omf tell -a off -t <node>
omf save -n <node> # prints "Image saved successfully: <auto-name>.ndz"
omf tell -a on -t <node>
omf tell -a offh -t <core-host>,<gnb-host>,<ue-node>
Rename the saved .ndz to something meaningful and verify the new image boots and works before deleting any prior image.
| Symptom | Likely cause | Fix |
|---|---|---|
gNB stuck at unknown-PLMN after an AMF restart |
NG-Setup race — AMF needs ~25 s for simapp to re-push the PLMN | ocudu-ngap-guard recovers it automatically; a guard restart re-pins the gNB |
UE registers then drops (UE Context Release / limited-service) |
stale AMF UE context | clear the AMF context + restart amf, restart the NGAP guard, re-run modem-pdu-on-reg.sh (see command below) |
| Modem stuck in SEARCH | left in CFUN=0 (RF off) after watchdog churn |
check AT+CFUN? = 1 |
| UE node unreachable over mgmt | wwan0 default route black-holes mgmt |
reach it via its control IP (ssh root@<ctrl-ip>) |
| k8s API down after a domain move | RKE2 node-ip not set to the control IP |
set node-ip: CORE_CTRL_IP in /etc/rancher/rke2/config.yaml (etcd identity) |
PRACH request late on a network USRP |
fronthaul latency over the routed data plane | raise expert_phy.max_proc_delay to 10–12; confirm jumbo MTU end-to-end |
Helm upgrade: buildCustomCert: unable to decode base64 certificate |
SD-Core CA-reuse bug on 4.x upgrades | kubectl delete secret 5g-control-plane-ca 5g-control-plane-ca-private -n aether-5gc, retry |
Added 2026-09-20 after this cost a day. One SMF pod restart breaks the PDU path two ways at
once, and the table above predates the finding:
Error during Process: datapath down, indefinitely,bessd underneath is perfectly healthy;transport 5GSM Message to SMF and| symptom | restart |
|---|---|
AMF logs transport 5GSM Message to SMF then goes silent |
AMF |
UPF logs datapath down; SMF logs PfcpSessCreateFailure |
upf-0 |
| anything | never the SMF |
Check the association directly — it is the gate that predicts whether a PDU session can be made:
kubectl -n aether-5gc logs upf-0 -c pfcp-agent --since=3m | grep -i association
# healthy: association setup done between nodes local: <upf> remote: <smf-pod-ip>
And datapath down does not mean BESS is down — verify before chasing it:
kubectl -n aether-5gc exec upf-0 -c bessd -- bessctl show worker # expect a RUNNING worker
kubectl -n aether-5gc exec upf-0 -c bessd -- bessctl show port # expect Link UP + counters
CHECKSUM_PARTIAL and is forwarded into GTP without the checksum completed;Clear-context recovery (run from the console host):
ssh root@<core-host> "KUBECONFIG=/etc/rancher/rke2/rke2.yaml kubectl exec mongodb-0 -c mongodb -n aether-5gc -- \
mongosh --quiet --eval 'db.getSiblingDB(\"sdcore_amf\").getCollection(\"amf.data.amfState\").deleteMany({}); \
db.getSiblingDB(\"aether\").getCollection(\"subscriptionData.contextData.amf3gppAccess\").deleteMany({})'; \
KUBECONFIG=/etc/rancher/rke2/rke2.yaml kubectl delete pod -n aether-5gc -l app=amf"
ssh root@<gnb-host> "systemctl restart ocudu-ngap-guard"
ssh root@<ue-node> "/root/modem-pdu-on-reg.sh"
Platform B = 2× Intel Xeon Gold 6126 @ 2.60 GHz (24 cores / 48 threads), matching the
platform letters on the OCUDU page.
| RAN configuration | UE | Platform | DL, Mbit/s | UL, Mbit/s | Date |
|---|---|---|---|---|---|
| OCUDU over the air, USRP B210 | Quectel RM520N-GL | B | 23.6–24.6 | 10.3–11.7 | 2026-09-18 |
Ping 0 % loss, ~25 ms. The same SD-Core has since been driven from an OCUDU O-RAN 7.2 O-DU
against commercial O-RUs — LiteOn FlexFi at DL 141–149 / UL 37.6–58.7 (2026-09-20) — see the
Aether SD-Core page for that table.
COSMOS domains use different /16 planes, so update every per-domain address before first bring-up. Private subnets (192.168.x) never change.
Core host — /home/native/onramp/vars/main.yml: core.ip + amf.ip → CORE_DATA_IP, gnb_ip → GNB_DATA_IP (core.data_iface stays the data-plane NIC name if hardware is identical). /etc/rancher/rke2/config.yaml: node-ip → CORE_CTRL_IP. Redeploy: cd /home/native/onramp && make 5gc-router-install && make 5gc-core-install, then manually helm upgrade sd-core oci://ghcr.io/omec-project/sd-core --version 4.0.0 -n aether-5gc -f /tmp/sdcore-5g-values.yaml (delete the CA secrets first if buildCustomCert errors).
gNB host — /root/gnb_config.yaml: cu_cp.amf.addr → CORE_DATA_IP, cu_cp.amf.bind_addr → GNB_DATA_IP. /etc/systemd/network/10-*-data.{network,link}: ⚠️ these match by MAC — set [Match] MACAddress= to the new node's data-plane NIC MAC (cat /sys/class/net/<nic>/address) or switch to [Match] Name=<nic>; update the [Route] lines (N3 → CORE_DATA_IP, SDR → RADIO_GW), keep MTUBytes=9000, then networkctl reload && networkctl reconfigure <nic>. Update the variables at the top of /usr/local/sbin/ocudu-*.sh (DATA_NIC, radio IP/gateway, N3-route gateway, NUMA cores).
Radio — network N310/X310: set its data IP to the new scheme and update ru_sdr.device_args addr=
b210-reset.sh).UE node — the wwan0 IP is from the (private) UE pool — unchanged; update host-target IPs in the node scripts (the data-keep watchdog's gNB-host target, any iperf targets). RF/PLMN config is unchanged.
Order: core → confirm AMF externalIP + NFs Running → gNB networkd/config/start → confirm NGAP → UE. Verify end-to-end before re-saving for the new domain.
Portability tip: to make a domain move a one-file edit, switch the networkd
[Match]to interface name and factor the per-host IPs into a single sourced/usr/local/etc/ocudu-net.envread by the scripts.
A worked port of these images to the grid domain — core on srv17.grid, gNB on srv19.grid, a grid X310 — beyond the generic domain-move steps above, the grid needs these specific changes.
Network model. The grid uses three /16 planes: control 10.10, server-data 10.11, and a separate SDR plane 10.20 where the X310s live (addressed 10.20.<row>.<col> — e.g. node23-5 → 10.20.23.5). Grid servers have two Mellanox mlx5 NICs (ens10f1np1 = control, ens10f0np0 = data) and NUMA node0 = cores 0–9 (vs. the SB1 servers' Intel ixgbe/enp* NICs and higher core IDs). The gNB's data NIC is multi-homed: a 10.11 address for core↔gNB (N2/N3) plus a 10.20 address for the X310 fronthaul — set both statically (the grid data planes have no DHCP for these images).
Imaging — add the grid.cosmos-lab.org frisbee domain. omf load fails with Frisbee: No domain manager found for domain 'grid.cosmos-lab.org' because the imaging aggregate manager knows the grid only as grid.orbit-lab.org. On the frisbee AM (am-rt1), add a grid.cosmos-lab.org block to /etc/omf-services/frisbee.yml mirroring grid.orbit-lab.org (same multicastIF: 10.10.0.42, a unique startPort), then systemctl restart omf-frisbee. Grid servers are slow to PXE (~110 s to come online); a node that never comes online is dead — pick another.
Core host — relocate k8s to the new IP. RKE2 is pinned to the source domain's node-ip, which does not exist on the grid, so it hangs in "activating." Set node-ip in /etc/rancher/rke2/config.yaml to the grid control IP and run rke2 server --cluster-reset (re-initialises the single-member etcd at the new IP — the pinned node-name: node1 keeps the workloads schedulable and all Kubernetes data is preserved), then systemctl start rke2-server and force-delete the stale Unknown pods. Also: correct the OnRamp hosts.ini ansible_host IPs to the grid, and remove the stale source-domain NIC names in /etc/sysctl.conf (e.g. net.ipv4.conf.enp59s0f0np0.forwarding) that otherwise make make 5gc-router-install fail on its sysctl reload. Then the usual re-IP: set OnRamp data_iface/core.ip/amf.ip/gnb_ip to the grid data plane and run make 5gc-router-install + make 5gc-core-install + a manual helm upgrade (delete the CA secrets first if it errors with buildCustomCert).
gNB host — X310 over the SDR plane. Use a type=x300 radio config:
ru_sdr:
device_driver: uhd
device_args: type=x300,addr=10.20.23.5,master_clock_rate=184.32e6,recv_frame_size=6000,send_frame_size=6000
srate: 23.04 # 184.32e6 / 8
otw_format: sc16
tx_gain: 31 # X310/UBX ranges (tx ~0-31.5 dB, rx ~0-37.5 dB)
rx_gain: 30
clock: internal
sync: internal
⚠️ Frame size is the key X310 tuning. The grid SDR plane passes only ~7000-byte frames (not the full 9000 jumbo). recv/send_frame_size=1472 produces a flood of ~70 underflows/s (UHD warns "1472 ... limits sample rate"); =6000 gives 0 underflows — size the frame to the path MTU (find it with ping -M do -s <n> <X310-IP>). The 32 MB socket buffers (net.core.rmem_max/wmem_max) are also required. Pin the gNB to grid node0 cores (e.g. docker update --cpuset-cpus 2-9 ocudu-gnb), and set the gNB cu_cp.amf.addr/bind_addr and the N3 route to the grid 10.11 data IPs.
⚠️ RF reach / UE. The grid X310 radiates in the grid radio room, which is a separate RF space — a UE located elsewhere in the testbed will not hear the grid cell (a Quectel at SB1/OSC stayed in SEARCH with the grid cell confirmed on-air and clean). Validate UE attach with a UE in the grid's own RF environment (a grid COTS UE, or a soft srsue on another grid X310); otherwise the gNB-side validation is cell-on-air + NGAP-to-core. If you share one UE across domains on the same carrier, keep only one cell on-air at a time (power off the other gNB).
Author(s): COSMOS team · Last verified: 2026-09-18 (OCUDU + SD-Core 4.0.0, USB B210, n78 — end to end with DL 23.6–24.6 / UL 10.3–11.7 Mbit/s; the two-server network-USRP path was last verified 2026-06-21 with a network N310) · Tested image/release: aether-ocudu-0.7.0-data2-20260621.ndz + aether-sdcore-4.0.0-data2-20260621.ndz · Tags: 5g, oran, srsran, ocudu, sd-core, aether, ue