Part of: Aether on COSMOS · 4G / 5G & O-RAN tutorials
Load the Aether SD-Core image onto a COSMOS server, adapt it to that host, bring every 5G core network function up on its single-node Kubernetes cluster, provision subscribers, and verify the core — including the user-plane association that decides whether a PDU session can ever be created. The end state is a core ready for a gNB on another server.
In every tutorial before this step the 5G core was a dependency: Open5GS as native processes or OAI CN5G as Docker containers, on the same machine as the gNB, started by a script and rarely looked at. This tutorial makes the core the object of study. Aether SD-Core runs each network function (NF) — the Access and Mobility Management Function (AMF), the Session Management Function (SMF), the User Plane Function (UPF), the NF Repository Function (NRF) and the rest — as a pod in a Kubernetes cluster, configured through a Helm chart, on a server of its own. The Aether SD-Core page describes each component; this tutorial stands them up and teaches you to read their health.
People run SD-Core for two reasons: to study a core the way a production operator runs one (declarative configuration, pods that are rescheduled, service discovery), and to put the core on its own host so that N2 and N3 cross a real network. Aether's SD-RAN part is not supported on COSMOS. The next tutorial, OCUDU + SD-Core on two servers, connects a gNB and a commercial UE to the core you build here.
How the pieces talk: the core host's cluster runs everything in namespace aether-5gc. The AMF is published on the host's data-plane address through a Kubernetes Service with an externalIP, so a gNB elsewhere reaches it for N2 (NGAP over SCTP, port 38412). The UPF reaches the data plane through two macvlan interfaces on fixed private subnets — access 192.168.252.0/24 (N3, GTP-U from the gNB) and core 192.168.250.0/24 (N6, toward the internet). simapp pushes subscribers, slices and the served PLMN/TAC into the core a few seconds after the NFs start. The Topology figure shows the core host on the right.
What is new compared with the previous steps: the core is Kubernetes workloads, so you work with kubectl and helm instead of systemctl and docker; configuration is a Helm values file, not per-NF files; and the image carries state tied to the host it was built on, which you must undo before the core will start anywhere else.
helm upgrade re-renders all of them, so a ConfigMap edited by hand is overwritten at the next upgrade. Persistent changes — subscribers included — go into the values file. The Aether SD-Core page explains the RKE2 / Helm / OnRamp layering.After completing this tutorial you will be able to:
Running;kubectl cannot show: the UPF's PFCP association with the SMF, and the BESS dataplane's health;upf-0, never the SMF;| Learning path | Step 5 of 7 — Cloud-native core. Previous: OCUDU with a commercial O-RU (O-RAN 7.2). Next: OCUDU + SD-Core on two servers |
| Difficulty | Advanced (Kubernetes and Helm) |
| Estimated time | not timed end to end; the bring-up script waits up to 5 min for the API and 5 min for pod readiness |
| Domain / sandbox | osc or sb1 — one server |
| Core | Aether SD-Core, Helm chart 4.0.0, on single-node RKE2 |
| Topic group | Cellular 4G/5G & O-RAN |
| Last verified | 2026-09-19 — image adapted and brought up on an sb1 server by the bring-up script: 16/16 pods Running, NG Setup from an OCUDU gNB. The same bring-up script, from aether-sdcore-4.0.0-20261106, ran in the two-server tutorial's parity runs on an osc and an sb1 pair on 2026-11-06. The OnRamp/Helm subscriber path (Step 4) was not re-run. |
Background knowledge
Account & access
Devices / nodes
| Resource | Role | Notes |
|---|---|---|
| One server (the core host) | RKE2 + SD-Core | needs a data-plane interface (N2/N3) and a control interface (Kubernetes node IP); see your domain page |
Disk image (load with omf load)
| Image | Provides |
|---|---|
aether-sdcore.ndz (currently aether-sdcore-4.0.0-20261106) |
RKE2, SD-Core Helm release 4.0.0 in namespace aether-5gc with every workload scaled to 0, Aether OnRamp in /home/native/onramp, the UPF macvlan and iptables services, /usr/local/sbin/core-start.sh and core-stop.sh |
Software components
| Component | Version | Source |
|---|---|---|
| Aether SD-Core | Helm chart 4.0.0 (oci://ghcr.io/omec-project/sd-core): AMF rel-3.0.0, SMF rel-4.0.0, UPF upf-bess / pfcp-agent rel-2.4.2 |
omec-project |
| Aether OnRamp | Ansible + Helm deployer in /home/native/onramp |
aether-onramp |
| RKE2 | per OnRamp vars/main.yml |
RKE2 docs |
ocudu-sdcore_host.py (installed as sdcore-host.py), ocudu-sdcore_core.sh, ocudu-sdcore_subs.py |
host-portability tool, core bring-up script, SIM provisioning through the core's REST API | the ocudu-sdcore tutorial in cosmos-experiments-ocudu on every console (cosmos-tutorial which ocudu-sdcore) |
cosmos-sim-db |
the COSMOS SIM inventory | installed or updated on the core host by the bring-up script, from the COSMOS package repository |
Spectrum / RF — none. The core has no radio.
Figure — The two-server layout this tutorial prepares. You build the right-hand server: RKE2 running the AMF, SMF, UPF (BESS), NRF and the other NFs. The gNB server on the left and the UE are added in the next tutorial; N2 and N3 will arrive over the 10 GbE data plane.
Figure — SD-Core runs in namespace aether-5gc on an RKE2 cluster; the gNB reaches the core over N2/N3.
core host
control iface (CTRL) ── RKE2 node-ip ── Kubernetes API, etcd
data iface (DATA2) ──┬── Service amf externalIP = data IP ← N2 (NGAP/SCTP 38412) from a gNB
├── macvlan access 192.168.252.0/24 ← N3 (GTP-U) from a gNB
└── macvlan core 192.168.250.0/24 → N6 toward the internet
namespace aether-5gc: amf smf upf-0 (bessd, pfcp-agent, routectl, web, arping)
nrf ausf udm udr pcf nssf webui simapp mongodb kafka ...
pods: 10.42.0.0/16 services: 10.43.0.0/16 (RKE2 defaults carried by this image)
UE pool: 192.168.100.0/24
Walk through it by plane. The control plane from a future gNB arrives on the data interface at the AMF's externalIP (N2); inside the cluster the AMF finds the SMF, the authentication functions and the policy functions through the NRF. The SMF programs the UPF over N4 (PFCP), pod to pod on the same node. The user plane enters the UPF on the access macvlan (N3) and leaves on the core macvlan (N6). The control interface carries only the Kubernetes node address and SSH. Interface names are COSMOS role names (CTRL, DATA2), which the image's cosmos-ifname assigns on osc and sb1 servers; on a host without role names, pass the real NIC names instead (for example eno2np1 and enp94s0f0np0).
Imaging overwrites the node's disk, so the node is powered off first.
ssh <username>@console.<sandbox>.cosmos-lab.org
omf tell -a offh -t <core-host>
omf load -i aether-sdcore.ndz -t <core-host> -r 0 -o 2400 -b 600
omf tell -a on -t <core-host>
omf stat -t <core-host>
/root on the core host:D=$(cosmos-tutorial which ocudu-sdcore)
scp $D/ocudu-sdcore_core.sh $D/ocudu-sdcore_subs.py root@<core-host>.<sandbox>.cosmos-lab.org:/root/
scp $D/ocudu-sdcore_host.py root@<core-host>.<sandbox>.cosmos-lab.org:/root/sdcore-host.py
Everything below runs as root on the core host. kubectl and helm need the RKE2 kubeconfig:
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
sdcore-host.py show derives, from this host's own interfaces, every value the image hard-codes for its build host, and changes nothing. Pass the control and data interfaces: role names where they exist, real NIC names otherwise.
python3 /root/sdcore-host.py show
# or, on a host without role names:
python3 /root/sdcore-host.py --ctrl-role <ctrl-nic> --data-role <data-nic> show
What to look for: the control address that will become the RKE2 node-ip; the data interface that will become the macvlan master, OnRamp's data_iface and the network-attachment master; and the data address that will become OnRamp's amf.ip, the AMF Service's externalIP and the address a gNB must target. It also prints the route a gNB host needs to reach the UPF (ip route replace 192.168.252.0/24 via <data-ip> dev <data-iface>). If it stops with no IPv4 on …, the interface names are wrong for this host; the message lists the interfaces it found.
The image records its build host in several places, and a core loaded elsewhere does not start until they are rewritten: RKE2 hangs in activating, and the UPF alone stays in Init while the other pods look healthy. The main ones:
| # | What the image carries | What the script sets it to |
|---|---|---|
| 1 | node-ip: in /etc/rancher/rke2/config.yaml |
this host's control address |
| 2 | the raw NIC name in the macvlan unit aether-5gc-macvlan.service |
the data interface |
| 3 | data_iface: in /home/native/onramp/vars/main.yml |
the data interface |
| 4 | amf.ip: in the same file |
this host's data address |
| 5 | the etcd membership, pinned to the build host | reset with rke2 server --cluster-reset |
| 6 | externalIPs on the Service aether-5gc/amf |
this host's data address |
| 7 | master: on the multus network attachments access-net and core-net |
the data interface |
The script also rewrites the UPF's routes to the RAN subnet, deletes stale pod-address files under /var/lib/cni/networks/k8s-pod-network/ (without that, new pods fail with no IP addresses available in range set), and scales every Deployment and StatefulSet back up: the image is saved scaled to 0, so the script reads the intended replica counts from the Helm release.
ocudu-sdcore_core.sh does all of this idempotently, printing a [core] line per stage: it applies the host delta with sdcore-host.py; starts RKE2 and, if the journal says the node is not a member of the etcd cluster, runs the cluster reset; installs or updates cosmos-sim-db and adds every missing COSMOS SIM through the core's REST API; scales the workloads; waits for every pod to be ready and for a stable UPF↔SMF PFCP association; and restarts the AMF deployment, clearing the stored AMF and SMF context of the IMSI given. Its arguments are the data interface, an IMSI and the control interface:
bash /root/ocudu-sdcore_core.sh DATA2 001010123456780 CTRL
# on a host without role names, pass the real NIC names instead of DATA2 / CTRL
[core] etcd membership is stale (baked on another host) - running cluster-reset
[core] k8s ready: node1 Ready
[core] SIM inventory: cosmos-sim-db 1.4.0-1
[core] pods ready: 16/16
[core] after: missing 0
[core] UPF <-> SMF PFCP association: stable
[core] READY
A re-run reports every host value as already correct. The first run on a freshly loaded host takes about five minutes longer, while the cluster re-forms around the new address.
--cluster-reset rewrites the single-member etcd to this node's address while keeping the cluster's data and the sd-core Helm release, which saves a full redeploy. If you need to run it by hand — sdcore-host.py apply prints this sequence when the API is unreachable:
systemctl stop rke2-server && /usr/local/bin/rke2-killall.sh
/usr/local/bin/rke2 server --cluster-reset # then Ctrl-C when it says to
systemctl start rke2-server
Info: On the server the image was built on, none of this is needed: the image's own
/usr/local/sbin/core-start.shverifies or restarts the core, as in the two-server tutorial.
Pod status is necessary but not sufficient. Check it, then check the address a gNB will use, then the deployed chart:
kubectl get pods -n aether-5gc
kubectl get svc amf -n aether-5gc
helm list -n aether-5gc
What to look for: every pod Running, upf-0 at 5/5; the amf Service's external IP equal to this host's data address; the sd-core release at chart 4.0.0. Trust helm list, not OnRamp's vars/main.yml, for the deployed version: the file in the OnRamp checkout can differ from the chart that is installed.
The UPF's dataplane health and its association with the SMF are invisible to kubectl. The association line is the one that predicts whether a PDU session can be created:
kubectl -n aether-5gc logs upf-0 -c pfcp-agent --since=3m | grep -i association
# healthy: association setup done between nodes local: <upf> remote: <smf-pod-ip>
kubectl -n aether-5gc exec upf-0 -c bessd -- bessctl show worker # expect a RUNNING worker
kubectl -n aether-5gc exec upf-0 -c bessd -- bessctl show port # expect Link UP + counters
What to look for: association setup done, a RUNNING worker, and the access and core ports Link UP. attached workers 0 in the bessd log is pipeline-construction noise, present on working runs too.
Subscribers, device groups and slices live in OnRamp's values template, /home/native/onramp/deps/5gc/roles/core/templates/radio-5g-values.yaml, under omec-sub-provision → simapp. The shape of one subscriber, its device group and the slice — an excerpt, with other keys omitted and the secret values redacted (never put a SIM's key or OPc on a wiki page or in a shared file):
subscribers:
- ueId-start: "001010000000033"
ueId-end: "001010000000033"
plmnId: "00101"
opc: <redacted>
op: <redacted>
key: <redacted>
sequenceNumber: <redacted>
device-groups:
- name: "user-group1"
imsis:
- "001010000000033"
ip-domains:
- dnn: internet
mtu: 1460 # Value is sent to UE when PDU Session Established
ue-ip-pool: {{ core.upf.default_upf.ue_ip_pool }} # IP address pool for subscribers
network-slices:
- name: "default"
slice-id:
sd: "010203"
sst: 1
The same file carries the served sites: gnb1 with tac: 1, gnb2 with tac: 2, PLMN 001/01. SD-Core serves only the TACs listed there.
To apply a change, re-render the values and upgrade the release. OnRamp's own Helm step fails on this OCI-only setup with no repositories found, so render with OnRamp and upgrade by hand:
cd /home/native/onramp && make 5gc-core-install
helm upgrade sd-core oci://ghcr.io/omec-project/sd-core --version 4.0.0 -n aether-5gc -f /tmp/sdcore-5g-values.yaml
If the upgrade fails with buildCustomCert: unable to decode base64 certificate, delete the control-plane CA secrets and retry; every NF certificate is re-issued:
kubectl delete secret 5g-control-plane-ca 5g-control-plane-ca-private -n aether-5gc
After the upgrade simapp pushes the subscribers into the core a few seconds after the NFs restart. A helm upgrade re-renders every NF's configuration from the values, so run the bring-up script again afterwards to re-apply this host's values (Step 2).
Info: You do not need a Helm upgrade for COSMOS SIM cards: the bring-up script adds every usable card from
cosmos-sim-dbthrough the core's REST API (ocudu-sdcore_subs.py) on each run, andsimappre-seeds the image's own list at every core start. Never put a SIM's key or OPc in a shared file.
Two rules decide whether the next tutorial's gNB comes up cleanly.
Restart the AMF before a gNB connects. An AMF can intermittently fail to encode the Initial Context Setup Request, which stalls a UE's registration at T3550 in a way that looks like a radio fault. A restart before the gNB connects is a cheap precaution (about 20 s); the bring-up script already did it. Do it again whenever the core has run for a long time before a gNB starts. Restarting the AMF drops any gNB's SCTP association, so do it before the gNB starts, not after.
kubectl -n aether-5gc rollout restart deployment/amf
kubectl -n aether-5gc rollout status deployment/amf --timeout=180s
Never restart the SMF. One SMF restart wedges the UPF's PFCP agent — it keeps the association to the old SMF pod, then refuses every new one with datapath down — and re-stales the AMF's cached SMF entry. The fixes are to restart the AMF or upf-0:
| Symptom | Restart |
|---|---|
AMF logs transport 5GSM Message to SMF then goes silent |
AMF — kubectl -n aether-5gc rollout restart deployment/amf |
UPF logs datapath down; SMF logs PfcpSessCreateFailure |
upf-0 — kubectl -n aether-5gc delete pod upf-0 |
| anything | never the SMF |
Danger: An SMF restart breaks every PDU session on the core, silently, until the UPF pod is restarted. The core may be shared with a RIC or other users' gNBs; restart only what the table names.
End-to-end check. The next tutorial attaches a UE to this core: by default an srsUE over ZMQ on the gNB server, or a commercial modem on a B210 or N310 cell.
Kubernetes gives pods and Services their own address ranges. This image's cluster carries RKE2's defaults in /etc/rancher/rke2/config.yaml:
cni: multus,canal
cluster-cidr: 10.42.0.0/16 # pods (RKE2 default)
service-cidr: 10.43.0.0/16 # services (RKE2 default)
Both lie inside COSMOS NJ's 10.0.0.0/10. They are not routed off the node — from another server the pod addresses fall through to the default route — so nothing collides today. The hazard is a future testbed allocation in either range: every node hosting pods would silently lose reachability to it, while every other host reached it fine. COSMOS therefore reserves 10.128.0.0/14 for node-local virtualised workloads. A newly created SD-Core cluster takes:
cluster-cidr: 10.128.0.0/20 # pods
service-cidr: 10.129.0.0/24 # services
and the next cluster 10.128.16.0/20 and 10.129.1.0/24. The pod range is carved into one /24 per node, so a /20 serves 16 nodes; the service range needs one address per Service, and SD-Core has about thirty. These values take effect only when a cluster is created; they cannot be changed on a running RKE2 cluster, and the current image still carries the defaults. The stale-IP cleaner in Step 2 matches the image's 10.42. pod range.
| Check | Command | Expected |
|---|---|---|
| Kubernetes up | kubectl get nodes |
the node Ready |
| All NFs | kubectl get pods -n aether-5gc |
every pod Running; upf-0 5/5 (16/16 on the verified run) |
| Deployed chart | helm list -n aether-5gc |
sd-core at chart 4.0.0 |
| AMF reachable for N2 | kubectl get svc amf -n aether-5gc |
external IP = this host's data address |
| UPF ↔ SMF | kubectl -n aether-5gc logs upf-0 -c pfcp-agent --since=3m \| grep -i association |
association setup done between nodes … |
| UPF dataplane | kubectl -n aether-5gc exec upf-0 -c bessd -- bessctl show worker / show port |
a RUNNING worker; ports Link UP |
| End to end | the two-server tutorial | NG Setup from the gNB; UE registration, PDU session, ping 0 % loss |
core-stop.sh scales the NF deployments to 0 and keeps MongoDB and Kafka, so the subscriber data survives:
/usr/local/sbin/core-stop.sh
To keep a modified core as a new image, follow the save procedure in the two-server tutorial's Cleanup: run /root/prepare.sh first, scale the workloads to 0 before saving, and verify the new image boots and works before deleting any earlier one. Then power the node off from the console and release the reservation:
omf tell -a offh -t <core-host>
| Symptom | Cause | Fix |
|---|---|---|
rke2-server hangs in activating; journal says not a member of the etcd cluster |
etcd membership still names the build host | rke2 server --cluster-reset (Step 2); do not wipe etcd |
| Kubernetes API down after moving to another host or domain | RKE2 node-ip not set to this host's control address |
sdcore-host.py apply, or set node-ip in /etc/rancher/rke2/config.yaml |
upf-0 alone stuck in Init, multus Link not found; every other pod healthy |
the access-net / core-net network attachments still name the build host's NIC |
sdcore-host.py apply, then delete the upf-0 pod so it re-attaches |
New pods fail with no IP addresses available in range set |
stale per-IP allocation files carried in the image | re-run the bring-up script (Step 2), which deletes them |
Pods Pending for minutes after a hard reboot; node tainted disk-pressure |
kubelet latches a false DiskPressure taint for the eviction transition period | wait it out, or set --eviction-pressure-transition-period=30s under kubelet-arg: in /etc/rancher/rke2/config.yaml; force-delete Unknown pods |
make 5gc-core-install fails at Helm with no repositories found |
OnRamp's Helm module runs helm repo update on an OCI-only setup |
let it render, then run the manual helm upgrade (Step 4) |
make 5gc-router-install fails on a sysctl reload |
/etc/sysctl.conf still names the build host's NICs |
remove those forwarding lines |
helm upgrade fails: buildCustomCert: unable to decode base64 certificate |
CA reuse on 4.x upgrades | delete 5g-control-plane-ca and 5g-control-plane-ca-private, retry |
After a chart upgrade bessd crash-loops; PFCP datapath down |
chart 4.1.0's UPF (upf-bess rel-2.4.3) regression |
stay on chart 4.0.0; helm rollback sd-core 1 |
UPF logs datapath down; SMF PfcpSessCreateFailure; UE gets PDU Session Establishment Reject cause 31 |
the UPF holds its PFCP association to an old SMF pod — usually after an SMF restart | delete upf-0; never restart the SMF |
AMF logs transport 5GSM Message to SMF, then nothing |
the AMF's cached SMF entry is stale | restart the AMF |
gNB NG Setup rejected with unknown-PLMN just after the core starts |
simapp has not yet pushed the PLMN/TAC; an OCUDU gNB does not retry |
restart the gNB until NG Setup succeeds |
| gNB NG Setup fails although PLMN matches | the gNB's TAC is not 1 or 2 | set the TAC in both supported_tracking_areas[].tac and cell_cfg.tac |
Registration stalls at T3550; AMF log shows build InitialContextSetupRequest failed: sequence of size is larger than upperbound |
the AMF fails to encode the request (intermittent); the gNB never gets a UE context | restart the AMF deployment, then the gNB (Step 5) |
A Quectel modem deregisters right after Registration Accept (gNB release cause nas) |
chart 4.0.0's AMF (rel-3.0.0) puts the same S-NSSAI twice in the Allowed NSSAI |
run the AMF at rel-3.1.0, as the two-server tutorial does (sdcore_amf_image) |
| UE has an address but downlink is exactly 0.00 Mbit/s | the core advertises MTU 1460; the GTP path carries 1440 | set the UE interface MTU to 1400 |
iperf3 server on the core host never works |
traffic from the core host reaches the UPF over macvlan with unfinished checksums | run the server on the gNB host or any other host |
On this wiki
External
sd-core Helm chart.Author(s): COSMOS team · Last verified: 2026-09-19 (bring-up script on an sb1 server, 16/16 pods, NG Setup from OCUDU; host delta + UE PDU session and internet data on osc 2026-09-18) · page written 2026-09-24 · Tested image/release: aether-sdcore-4.0.0-data2-20260703.ndz (2026-09-19), aether-sdcore-4.0.0-20261106 (2026-11-06, via the two-server tutorial) · SD-Core chart 4.0.0 · Tags: aether, sd-core, omec, 5gc, core, kubernetes, rke2, helm, onramp, 5g, sa, tutorial