Part of: OCUDU on COSMOS · 4G / 5G & O-RAN · Step 5 of the OCUDU learning path · Previous: Deploy Aether SD-Core · Next: Near-RT RIC + E2/KPM
Split a 5G Standalone network across two servers of your choice: the Aether SD-Core cloud-native core
on one, the OCUDU base station and a software UE on the other, talking N2 and N3 over the domain's data
network. You name the two servers when you run it; every address the experiment needs is read from those
two hosts at run time, so the same tutorial runs on any pair of servers in osc or sb1.
In the earlier steps the core ran on the same machine as the gNB, so the N2 and N3 interfaces never left the
loopback. Here they cross a real network, which is how a gNB and a core are wired in a real deployment. The
second new element is the core itself. SD-Core is a set of containers in a single-node Kubernetes (RKE2)
cluster, deployed with Helm. Operating it means reading pod states and pod logs with kubectl instead of
systemd units, and it has rules of its own, the most important being never restart the SMF (see
Troubleshooting). The Aether SD-Core page explains its
components.
Concepts
- N2 / N3 — control signalling (NGAP over SCTP, port 38412) from the gNB to the AMF, and user data (GTP-U) from the gNB to the UPF. Here both run between the two servers on the
DATA2data network.- Aether SD-Core — the open-source 5G core from the Aether project: AMF, SMF, UPF (BESS), NRF, AUSF, UDM, UDR, PCF and the rest, each a pod in namespace
aether-5gc.- Host-portability delta — the SD-Core image was built on one server and remembers it in ten places (Kubernetes node IP, etcd membership, NIC names, the AMF's external IP, the UPF's network attachments and routes, IP-address bookkeeping, an MTU rule, per-NIC kernel settings). The tutorial rewrites all of them for the server you picked before the core starts.
- COSMOS SIM deb —
cosmos-sim-db, the testbed's one list of SIM cards. The core server installs or updates it from the COSMOS package repository on every run, and the run adds every usable COSMOS SIM to the core. The tutorial itself carries no SIM keys.- Radio profile —
zmq(the default) needs no radio: the software UEsrsUEruns on the gNB server and the two exchange samples over ZMQ sockets.b210andn310put a real cell on the air for a commercial modem on a third node.
| Link | Between | Carried over |
|---|---|---|
| N2 — control | gNB (gNB server) ↔ AMF (core server) | NGAP over SCTP 38412, gNB server DATA2 → core server DATA2 |
| N3 — user data | gNB ↔ UPF | GTP-U, gNB server → the UPF's access network 192.168.252.0/24 (a macvlan on the core server's DATA2) |
| N6 — data network | UPF → internet | the UPF's core network 192.168.250.0/24 and NAT on the core server |
Radio (zmq) |
gNB ↔ srsUE | ZMQ sockets on the gNB server (band 3, 10 MHz, 15 kHz SCS) |
| UE data | srsUE ↔ applications | the tun_srsue interface in network namespace ue1 on the gNB server; UE pool 192.168.100.0/24 |
The gNB server reaches the UPF's access network, and the UE pool, only through the core server, so the run
adds those two routes on the gNB server. The iperf3 server runs on the gNB server, never on the core server
(see Troubleshooting).
| Domains | osc, sb1 |
| Nodes | 2, named in OMF_NODES |
| On the console | python3, ansible-playbook, ssh, scp |
cosmos-tutorial check ocudu-sdcore verifies, on each node:
Not in the package - you provide:
You will also need:
osc or sb1, and SSH access to that domain's console — see Getting started and SSH access;Both servers must have an address on the DATA2 data network; every server in osc and sb1 does.
On the console, name the core server first and the gNB server second, and ask whether the tutorial can run.
The check is read-only.
export OMF_NODES="<core-server>,<gnb-server>"
cosmos-tutorial check ocudu-sdcore
If a server lacks its image, load it first. The SD-Core image is large; keep -r 0 and the long timeouts:
omf load -i aether-sdcore.ndz -t <core-server> -r 0 -o 2400 -b 600
omf load -i ocudu.ndz -t <gnb-server> -r 0 -o 2400 -b 600
omf tell -a on -t <core-server>,<gnb-server>
Read the last line of each load: 1 node successfully imaged.
export OMF_NODES=<node>,<node>
cosmos-tutorial check ocudu-sdcore
cosmos-tutorial run ocudu-sdcore # ansible
cosmos-tutorial run ocudu-sdcore --mode manual # manual
cosmos-tutorial jupyter ocudu-sdcore # jupyter
cosmos-tutorial run ocudu-sdcore --mode parity # parity
cosmos-tutorial run ocudu-sdcore --mode teardown # teardown
All three modes read one parameter file, use one log parser, and write the same JSON result. OMF_NODES is
required and ordered: the core server, then the gNB server, then (only for the b210 and n310 profiles) the
UE node. The first run on a freshly loaded core server takes about five minutes longer than later ones,
because Kubernetes has to re-form its single-node cluster around the new host.
The playbook checks OMF_NODES against the radio profile, then brings the core up on the first server:
the host-portability delta, Kubernetes, the COSMOS SIM deb and its subscribers, the core workloads, the PFCP
gate and an AMF restart. It then reads both servers' DATA2 addresses, renders the gNB configuration, adds the
N3 and UE-pool routes on the gNB server, starts the gNB and an iperf3 server as named units
(ocudu-sdcore-gnb, iperf3-ue), starts srsUE (ocudu-sdcore-ue), measures and writes the result.
The manual mode runs these commands for you from the console. To follow them yourself:
D=$(cosmos-tutorial which ocudu-sdcore)
CORE=<core-server>; GNB=<gnb-server>
Step 1 — Bring up the core (core server). Copy the three node-side helpers and run the bring-up. It is
idempotent; a second run reports every value as already correct.
scp $D/ocudu-sdcore_core.sh $D/ocudu-sdcore_subs.py root@$CORE:/root/
scp $D/ocudu-sdcore_host.py root@$CORE:/root/sdcore-host.py
ssh root@$CORE "bash /root/ocudu-sdcore_core.sh DATA2 001010123456780 CTRL"
[core] deriving from this host: CTRL=<core CTRL address> DATA2=<core DATA2 address>
[core] * amf externalIP: <build server's DATA2> -> <core DATA2 address>
[core] * NAD access-net: enp216s0f1np1 -> DATA2
[core] etcd membership is stale (baked on another host) - running cluster-reset
[core] k8s ready: node1 Ready
[core] SIM inventory: cosmos-sim-db 1.4.0-1
[core] pods ready: 16/16
[core] sd-core: inventory 10 SIMs on 00101; subscribers missing 9, not in user-group1 9
[core] after: missing 0
[core] UPF <-> SMF PFCP association: stable
[core] READY
What each part does: the host-portability delta rewrites the Kubernetes node IP, the UPF's network
attachments and the AMF's external IP for this server (sdcore-host.py); an etcd cluster reset re-forms
the single-node cluster around this server, keeping every workload; GRO is switched off on DATA2 (with it on,
every TCP downlink through the UPF measures zero); the COSMOS SIM deb is updated and every usable COSMOS SIM
added to the core through its webconsole API; the run then waits until the UPF holds a stable PFCP association
with the SMF and restarts the AMF.
Step 2 — Addresses (console). Nothing is stored: both addresses are read from the servers.
AMF=$(ssh root@$CORE "ip -4 -br addr show DATA2 | grep -oE '[0-9]+(\.[0-9]+){3}' | head -1")
GNBIP=$(ssh root@$GNB "ip -4 -br addr show DATA2 | grep -oE '[0-9]+(\.[0-9]+){3}' | head -1")
Step 3 — Start the gNB (gNB server). Render the configuration for this core, copy it over, route N3 and
the UE pool through the core server, and start the gNB. Its standard input is kept open because the gNB exits
when it closes.
python3 $D/ocudu-sdcore_lib.py render-gnb $D/ocudu-sdcore_params.yml $D/ocudu-sdcore_gnb.yml.tmpl /tmp/gnb.yml \
--amf-ip $AMF --gnb-ip $GNBIP --radio zmq
scp /tmp/gnb.yml root@$GNB:/root/gnb-sdcore.yml
ssh root@$GNB "ip route replace 192.168.252.0/24 via $AMF dev DATA2; ip route replace 192.168.100.0/24 via $AMF dev DATA2"
ssh root@$GNB "systemd-run --unit=ocudu-sdcore-gnb --collect --working-directory=/opt/ocudu/build \
bash -c 'sleep infinity | /opt/ocudu/build/apps/gnb/gnb -c /root/gnb-sdcore.yml'"
ssh root@$GNB "ss -A sctp -an | grep 38412"
ESTAB 0 0 <gnb DATA2>:38592 <core DATA2>:38412
An ESTAB association to the core server's port 38412 is the N2 link.
Step 4 — Start the iperf server and the UE (gNB server).
ssh root@$GNB "systemd-run --unit=iperf3-ue --collect /usr/bin/iperf3 -s -B $GNBIP -p 5201"
ssh root@$GNB "ip netns add ue1; systemd-run --unit=ocudu-sdcore-ue --collect \
--working-directory=/opt/srsran4g/build/srsue/src bash -c 'sleep infinity | ./srsue /root/ue-zmq.conf'"
ssh root@$GNB "grep -aE 'Registration Accept|PDU Session' /tmp/ue.log"
ssh root@$GNB "ip netns exec ue1 ip route replace default dev tun_srsue; ip netns exec ue1 ip link set tun_srsue mtu 1400"
[NAS5G ] [I] Handling Registration Accept
[NAS5G ] [I] PDU Session Establishment successful. IP: 192.168.100.2
Step 5 — Measure (gNB server, inside the UE's namespace). Inside ue1 the traffic has to cross the cell,
the GTP tunnel and the core.
ssh root@$GNB "ip netns exec ue1 ping -c5 8.8.8.8"
ssh root@$GNB "ip netns exec ue1 iperf3 -c $GNBIP -p 5201 -R -t 15" # downlink
ssh root@$GNB "ip netns exec ue1 iperf3 -c $GNBIP -p 5201 -t 15" # uplink
The notebook runs the same steps with the same helpers, one cell per step, from the console's Jupyter
(cosmos-tutorial jupyter ocudu-sdcore). Set omf_nodes in its first cell, or export OMF_NODES.
cosmos-tutorial run ocudu-sdcore --mode parity runs the three modes one after the other on the same pair,
compares their results and releases the rig. It is the tutorial's acceptance test.
The result ends "registered": true with a UE address from 192.168.100.0/24, 0 % ping loss and throughput
close to the table below. The zmq cell is a software radio limited to about 25 Mbit/s downlink; the point is
the path, not the speed.
| Pair (core + gNB server) | gNB server CPU | Radio | Ping | DL, Mbit/s | UL, Mbit/s | Date |
|---|---|---|---|---|---|---|
osc |
2× Intel Xeon Gold 6126, 24C/48T | zmq |
0 % loss, 28–29 ms | 25.9–26.4 | 4.95–5.02 | 2026-11-06, parity green |
sb1 |
2× Intel Xeon Gold 6226, 24C/24T | zmq |
0 % loss, 29–32 ms | 26.1–26.5 | 4.90–4.97 | 2026-11-06, parity green |
sb1 |
2× Intel Xeon Gold 6226, 24C/24T | n310 + Quectel RM520N-GL (n41, 40 MHz), RSRP −74 dBm, SINR 28 dB |
0 % loss | 114–116 | 35.3–35.4 | 2026-11-06, manual mode |
Over the air, the same core served an OCUDU B210 cell at DL 23.6–24.6 / UL 10.3–11.7 Mbit/s (2026-09-18) and
OCUDU O-RAN 7.2 cells with Benetel and LiteOn radios (2026-09-22); see the
Aether SD-Core page.
Change any of these with cosmos-tutorial run ocudu-sdcore --params my.yml. An unknown key or a wrong type is refused. cosmos-tutorial params ocudu-sdcore prints the file.
| Key | Default | Meaning |
|---|---|---|
sdcore_ns |
aether-5gc |
|
sdcore_kubeconfig |
/etc/rancher/rke2/rke2.yaml |
|
sdcore_data_role |
DATA2 |
Role interface that carries N2 (NGAP) and N3 (GTP-U). |
sdcore_host_tool |
/root/sdcore-host.py |
|
sdcore_amf_image |
ghcr.io/omec-project/5gc-amf:rel-3.1.0 |
The AMF of SD-Core chart 4.0.0 (rel-3.0.0) lists one subscribed S-NSSAI TWICE in the Allowed NSSAI when a UE requests two S-NSSAIs that map to it (a modem that has registered on SD-Core once requests "01.010203:01"); a Quectel rejected such an Accept on 2026-09-20. |
ocudu_gnb_bin |
/opt/ocudu/build/apps/gnb/gnb |
|
ocudu_gnb_cfg |
/root/gnb-sdcore.yml |
RENDERED from ocudu-sdcore_gnb.yml.tmpl |
ocudu_gnb_log |
/tmp/gnb-sdcore.log |
|
ocudu_log_level |
warning |
⚠️ all_level: info logs EVERY SLOT — 611 MB in 1.5 h, and it depresses throughput enough to invalidate a measurement. |
ocudu_radio |
zmq |
Which radio this run uses. |
radios |
3 keys | |
ocudu_pci |
1 |
|
ocudu_plmn |
00101 |
|
ocudu_tac |
1 |
must be in the AMF's tai list (SD-Core serves tac 1 and 2) |
ocudu_sst |
1 |
|
ocudu_imsi |
001010000000033 |
|
ocudu_dnn |
internet |
|
ue_at_dev |
/dev/quectel-at |
udev symlink; ttyUSB numbers shift after CFUN=1,1 |
ue_netns |
uens |
The Quectel's QMI device and netdev are found on the UE by ocudu-sdcore_ue.sh (modem_resolve): a UE node can carry two modems, so cdc-wdm0/wwan0 is never assumed. |
ue_mtu |
1400 |
⭐ THE number that decides whether this experiment works at all. |
iperf_server_host |
gnb |
⭐ The iperf server may NOT run on the CORE host. |
iperf_port |
5201 |
|
iperf_secs |
15 |
To run over the air, set ocudu_radio: n310 (or b210) in your own parameter file, give it with --params,
and add the UE node as the third entry in OMF_NODES. The N310 profile looks the radio's address up in the
COSMOS inventory; the B210 must be on the gNB server's USB3.
registered and ue_ipv4 come from the UE's own log: the AMF accepted the registration and the SMF gave it
an IPv4 session. ping and iperf3 run inside the UE's namespace, so every packet crosses the cell, the GTP
tunnel between the two servers and the UPF. tcp_csum_errors counts TCP segments the UE threw away because
their checksum was wrong; anything above zero points at the iperf server's location or at an offload setting
on the core server.
| Symptom | Cause | Fix |
|---|---|---|
the core run hangs, RKE2 stays activating |
the image's etcd membership names the server it was built on |
the run detects not a member of the etcd cluster and runs rke2 server --cluster-reset once; if it does not, run the reset by hand: systemctl stop rke2-server; rke2-killall.sh; rke2 server --cluster-reset; systemctl start rke2-server |
UPF pod stays Init, multus logs Link not found |
a network attachment still names the build server's NIC | the delta sets DATA2; delete the upf-0 pod so it re-attaches |
new pods die with no IP addresses available in range set |
stale IP-address files from the build server | the run's ipam-gc step drops them |
UE registers, PDU session rejected (Invalid Choice, or Request rejected, unspecified) and the UPF logs no association found |
a subscriber change made the SMF re-associate with the UPF, and the request arrived inside that window | the run waits for a stable association setup done before it reports READY; if you change subscribers by hand, wait for it too |
the gNB logs N2: Failed to connect to AMF ... Connection timed out; ARP for the core server's data address stays INCOMPLETE |
the core server has a second port in the same data subnet (sb1: DATA2 and DATA2a), and the core's ARP policy (arp_filter, arp_ignore) then answers on neither |
the run gives the other port's subnet route a higher metric, so DATA2 answers; check with ip route get <gnb DATA2 address> on the core server - it must say dev DATA2 |
UE registers and gets an address, but no traffic passes; on the core bessctl show port shows the access port receiving and transmitting nothing |
the UPF still routes the RAN subnet of the server the image was built on | the run rewrites the UPF's routes to this server's data subnet and restarts upf-0; check with kubectl -n aether-5gc get sts upf -o yaml \| grep 'ip route replace' |
| ping and UDP work, but every TCP download measures 0 | GRO on the core server's DATA2 merges TCP segments that the UPF cannot forward |
ethtool -K DATA2 gro off on the core server (the run does this) |
iperf3 with the server on the core server never works |
packets from the core server's own stack reach the UPF as CHECKSUM_PARTIAL and arrive with a wrong TCP checksum |
run the iperf server on the gNB server |
| downlink of exactly 0.00 over the air while uplink works | the core gives the UE MTU 1460, the GTP path carries 1440 | set the UE interface to MTU 1400 (the run does) |
UE registers then drops; AMF logs build InitialContextSetupRequest failed: sequence of size is larger than upperbound |
stale state inside a long-running AMF process | restart the AMF (kubectl -n aether-5gc rollout restart deployment/amf); the run does this every time |
AMF logs transport 5GSM Message to SMF and nothing follows |
the AMF's NRF cache went stale, usually after an SMF restart | restart the AMF, never the SMF |
UPF logs datapath down, SMF PfcpSessCreateFailure |
the UPF holds its association to an old SMF pod | delete upf-0, never the SMF |
gNB stuck at unknown-PLMN / NG Setup refused |
the gNB advertises a TAC the AMF does not serve | SD-Core serves TAC 1 and 2; the parameters use 1 |
a Quectel rejects the Registration Accept; the AMF lists one S-NSSAI twice in the Allowed NSSAI (kubectl -n aether-5gc exec mongodb-0 -c mongodb -- mongosh sdcore_amf shows two identical entries for the UE) |
the AMF was set back to chart 4.0.0's own rel-3.0.0, which maps a UE's two requested S-NSSAIs (sst 1 / sd 010203 and sst 1) onto the one subscribed slice without removing the duplicate |
keep sdcore_amf_image at rel-3.1.0 (omec-project/amf #703); the rest of chart 4.0.0, including its UPF, stays as shipped |
Danger: Never restart the SMF. One SMF restart wedges the UPF (it keeps the association to the old SMF pod and refuses every new one with
datapath down) and re-stales the AMF's NRF cache. Restart the AMF orupf-0instead.
The PFCP association is the gate that predicts whether a PDU session can succeed:
ssh root@$CORE "kubectl -n aether-5gc logs upf-0 -c pfcp-agent --since=3m | grep -i association"
# healthy: association setup done between nodes local: <upf> remote: <smf-pod-ip>
cosmos-tutorial run ocudu-sdcore --mode teardown stops the gNB, srsUE and the iperf server on the gNB server
and removes the ue1 namespace. It leaves the core running: SD-Core is slow to bring up, and the next run on the
same pair starts from it. Release the servers with omf tell -a offh -t <core-server>,<gnb-server>.
Package cosmos-experiments-ocudu 0.1.0-31 - on every console it is released to (cosmos-tutorial list shows the installed version). The code is installed on the console in /usr/share/cosmos-experiments/ocudu-sdcore/. To change anything, cosmos-tutorial copy ocudu-sdcore puts an editable copy in the current directory, and cosmos-tutorial run ocudu-sdcore then runs that copy instead of the installed one (cosmos-tutorial which ocudu-sdcore shows which will run). Repository (GitLab account needed, not needed to run or change it): packages/cosmos-experiments-ocudu/usr/share/cosmos-experiments/ocudu-sdcore.
Verified 2026-11-06: three-mode parity on an osc pair and on an sb1 pair, each from a freshly loaded core server, with the refreshed
aether-sdcore image (aether-sdcore-4.0.0-20261106: Ubuntu 24.04 updates, kernel 6.8.0-149, cosmos-ifname
1.24 in place of the legacy interface renamer, the COSMOS package repository and cosmos-sim-db) and
ocudu.ndz.