Run the O-RAN Software Community (O-RAN SC) Service Management and Orchestration (SMO) stack,
N release, on one COSMOS server: the non-real-time RIC with its A1 policy service, the SDN-R O1
controller, the VES event collector and the TEIV topology service, on a single-node Kubernetes
cluster. At the end every SMO endpoint answers on the node, and you can open its web interfaces
from your laptop.
The earlier tutorials on the learning path built a 5G network and then, with the
near-RT RIC, looked inside it at one-second
granularity. This tutorial deploys the layer above: the SMO, the O-RAN management system that
configures the network elements, collects their alarms and measurements, keeps an inventory of the
network, and runs the non-real-time RAN Intelligent Controller (non-RT RIC) that sets policy
for the near-RT RIC. Nothing on this page touches a radio, a gNB or a UE; the SMO works on
timescales of seconds to days, through management interfaces.
The O-RAN SC packages the SMO as Helm charts in its it-dep repository. COSMOS builds them into a
disk image, smo.ndz (currently smo-20260925): Ubuntu 24.04 with a single-node Kubernetes cluster, the N-release SMO
installed, and all 72 container images cached on disk. A boot-time service, smo-up, makes the
image portable: on the server it was built on it restarts the cluster; on any other server it
builds a new cluster on that server's control address and reinstalls the SMO from the cached
images. You can therefore load the image on any suitable server and get a running SMO without
pulling anything from the internet except three small items described below.
Two paths are given. Path A loads the prebuilt image, and is the one to use unless you need to
change the build. Path B builds the same system from baseline.ndz with the scripts the image
was made from, for readers who want to modify the cluster, the chart values or the release.
The Topology figure places the SMO in the O-RAN architecture. The last section of Instructions
lists which COSMOS RAN stacks the SMO can manage today and which it cannot; read it before you
point the SMO at a real radio.
After completing this tutorial you will be able to:
/var/log/smo-up.log, and tell acurl and every pod with kubectl;baseline.ndz and say which three upstream container images had to| Learning path | Step 6 of 7 — RAN intelligence (SMO / non-RT). Previous: OCUDU + SD-Core on two servers. Alongside: near-RT RIC E2/KPM. Next: Amarisoft UE sweep |
| Domain / testbed | COSMOS osc sandbox (built on srv13.osc); any COSMOS server that meets the sizing below |
| Duration | Path A: image load, plus 4–6 min of firmware POST on HP Gen9 servers, plus 20–40 min for the first boot. Path B: longer; pulling the 72 container images dominates |
| Difficulty | Intermediate |
| Software | O-RAN SC SMO, N release (it-dep branch n-release, commit 8f797d6) |
| Platform | Ubuntu 24.04, Kubernetes 1.32.8 (kubeadm, containerd, Calico v3.30.1), Helm 3.18.6 |
| New interfaces | A1 (non-RT RIC → near-RT RIC), O1 (SMO → network elements) |
| Topic group | Cellular 4G/5G & O-RAN |
| Last verified | 2026-09-25 |
Background knowledge
kubectl get pods, kubectl logs, NodePort services. SeeAccount & access
omf load, omf tell, omf stat — seeDevices / nodes
| Role | Qty | What it needs |
|---|---|---|
| SMO server | 1 | Upstream minimum for the SMO: 64 GB RAM, 20 vCPU, 100 GB disk. The image was built on an HP DL380 Gen9 (2× Intel Xeon E5-2680 v3, 24 cores / 48 threads, 755 GB RAM). The server needs internet access at boot (see Troubleshooting) |
Disk images
| Image | Load onto | Provides |
|---|---|---|
smo.ndz (→ smo-20260925) |
SMO server (Path A) | Ubuntu 24.04, Kubernetes 1.32.8, Helm 3.18.6, the O-RAN SC SMO N release installed, 72 container images cached, smo-up.service |
baseline.ndz |
SMO server (Path B) | Ubuntu 24.04 base for building from scratch |
Software components
| Component | Version | Source |
|---|---|---|
| O-RAN SC SMO | N release, it-dep branch n-release, commit 8f797d6 |
it-dep/smo-install |
| Installer | smo-install/scripts/layer-2/2-install-oran.sh default release |
it-dep |
| Kubernetes | 1.32.8, kubeadm + containerd | pkgs.k8s.io |
| Calico | v3.30.1 | Project Calico manifest |
| Helm | 3.18.6 (the installer requires ≥ 3.12 and < 4.0) | get.helm.sh |
Spectrum / RF / special — none. The SMO uses no radio.
Figure — The SMO and its non-RT RIC sit above the RAN. They reach the near-RT RIC over A1 and every managed element over O1; the near-RT RIC reaches the O-CU and O-DU over E2. Notice that the SMO is nowhere on the path between the UE and the core.
This tutorial deploys the top box of the figure on one server. Inside that server the SMO is a
Kubernetes cluster of one node with three namespaces:
your laptop
│ ssh -L <port>:<node>:<port>
▼
console.<sandbox>.cosmos-lab.org
│ CTRL network
▼
SMO server (CTRL address, NodePorts 30091 30094 30205 30417 31575 32001, plus Keycloak and MinIO on ports assigned at install)
┌──────────────────────────────────────────────────────────────────────────────┐
│ onap SDN-R + ODLUX (O1/NETCONF), VES collector, Strimzi Kafka, │
│ policy / CLAMP, CPS / NCMP │
│ nonrtric A1 Policy Management Service, Information Coordinator Service, │
│ rApp manager + service manager, control panel, Kong gateway │
│ smo TEIV topology, RAN-PM pipeline, Keycloak, MinIO, InfluxDB, OPA, │
│ Redpanda console │
└──────────────────────────────────────────────────────────────────────────────┘
│ A1 (to a near-RT RIC) │ O1 NETCONF / VES (to O-DU, O-CU)
▼ ▼
not wired on COSMOS today Duranta via the OAI O1 adapter;
(see "Connecting the SMO") OCUDU via its O1 adapter sidecar
Walk through it by function.
onap holds the parts the SMO borrows from the Open Network Automation Platform (ONAP).nonrtric is the non-RT RIC. The A1 Policy Management Service (A1 PMS) keeps the listsmo holds shared services. TEIV (Topology Exposure and Inventory) keeps the networknonrtric-realm, which Kafka and the SMO workloads authenticate against;Every user-facing service is a Kubernetes NodePort: it listens on a fixed port on the server's
own addresses, including its CTRL (control-network) address. That is what the curl checks and the
SSH tunnel use.
Cluster address plan. COSMOS reserves 10.128.0.0/14 for Kubernetes and OpenStack. The SMO
cluster uses pods 10.130.0.0/18 and services 10.131.0.0/20, set in /root/smo-k8s-setup.sh. The
upstream defaults (pods 10.244.0.0/16, services 10.96.0.0/12, Calico pool 192.168.0.0/16) would
collide with COSMOS addresses 10.96–10.111 and with the COSMOS 5G UE pool 192.168.100.0/22.
Log into the console of the domain that holds your server:
ssh <username>@console.<sandbox>.cosmos-lab.org
Both paths start by imaging the server. HP ProLiant Gen9 servers spend 4–6 minutes in firmware
POST after every power change, and one that is still powering down when omf load starts misses
the network-boot check-in. So turn the server off and wait until omf stat reports it off
before loading:
omf tell -a offh -t <node>
omf stat -t <node> # repeat until the node reports off
The image is several gigabytes, and the default omf load timeout is too short for an image of
that size; -o 2400 -b 600 lengthens the timeouts, and -r 0 keeps the image's own partition
layout.
omf load -i smo.ndz -t <node> -r 0 -o 2400 -b 600
omf tell -a on -t <node>
omf stat -t <node>
What to look for: omf load ends with the summary line 1 node(s) successfully imaged. Judge
the load by that line, not by the exit status or the last percentage. If omf stat still shows the
node off after the on, issue omf tell -a on -t <node> again.
When the server has passed POST and booted, log in from the console:
ssh root@<node>
smo-up.service starts at every boot and runs /root/smo-up.sh, which logs to
/var/log/smo-up.log. In order, the script:
omf load does not always do it, and the SMOnonrtric-realm, and restores it if not (Keycloak runs instart-dev mode with no volume, so a restart loses the realm);Follow it:
tail -f /var/log/smo-up.log
systemctl status smo-up
The first boot after omf load takes 20–40 minutes. What to look for, in this order:
| Log line | Meaning |
|---|---|
grew root from …G to …G |
root filesystem enlarged (absent if it was already full size) |
restoring local image tag … |
a substituted image tag was missing and was restored (absent if all three were present) |
cluster belongs to this host (…) or rebuilding the cluster on … (…) |
restart on the build server, or rebuild on any other |
keycloak realm present or keycloak realm missing (HTTP …); re-running keycloak-init |
realm check; after a restore the script also restarts Kafka and the smo workloads |
all N pods ready |
the SMO is up |
O-RAN SC SMO on <node> (<address>): followed by the endpoint list |
the endpoints to use |
When it is done, systemctl status smo-up shows the unit active (exited). If the script ends
with not all pods ready after 20 min: it lists the pods that are not Ready; see Troubleshooting.
The SMO's services are NodePorts on the server's CTRL address. Set a variable with that address
and ask each service for its health or landing page:
CTRL=$(ip -4 -o addr show dev CTRL | awk '{print $4}' | cut -d/ -f1 | head -1)
export KUBECONFIG=/etc/kubernetes/admin.conf
KC=$(kubectl -n smo get svc keycloak -o jsonpath='{.spec.ports[0].nodePort}')
MINIO=$(kubectl -n smo get svc minio-web -o jsonpath='{.spec.ports[0].nodePort}')
for u in 30091/ 30094/status 30094/a1-policy/v2/policy-types 30205/odlux/index.html \
30417/healthcheck 31575/ $MINIO/ $KC/realms/nonrtric-realm; do
printf '%-36s %s\n' "$u" "$(curl -s -o /dev/null -m 10 -w '%{http_code}' http://$CTRL:$u)"
done
What to look for: 200 on every line. The endpoints are:
| Service | Port / path |
|---|---|
| Non-RT RIC control panel | 30091 |
| A1 Policy Management Service | 30094 /status, /a1-policy/v2/policy-types |
| SDN-R ODLUX | 30205 /odlux/index.html |
| VES collector | 30417 /healthcheck |
| rApp service manager | 31575 |
| MinIO console | $MINIO — assigned at install; smo-up prints it at the end of /var/log/smo-up.log |
| Keycloak | $KC — assigned at install; realm nonrtric-realm (/realms/nonrtric-realm) |
| TEIV topology | 32001 — it serves its API under sub-paths; / returns 404 |
Then check the pods:
kubectl get pods -A --no-headers | grep -vc Completed
kubectl get pods -A --no-headers | grep -v Completed |
awk '{split($3,a,"/"); if($4!="Running" || a[1]!=a[2]) print}'
What to look for: the first command prints 76 (pods other than completed jobs); the second
prints nothing, meaning every one of them is Running with all containers ready. To see each
namespace on its own, use kubectl get pods -n onap, -n nonrtric and -n smo.
Query the A1 Policy Management Service. The A1 PMS is a REST service. Ask it which policy types
it knows and which near-RT RICs are registered with it:
curl -s http://$CTRL:30094/a1-policy/v2/policy-types
curl -s http://$CTRL:30094/a1-policy/v2/rics
What to look for: JSON replies. The PMS learns policy types from the near-RT RICs it is
configured to manage, so the lists stay empty until a near-RT RIC with an A1 endpoint — or an A1
simulator — is registered with it (see "Connecting the SMO to the RAN" below).
Open the web interfaces from your laptop. The NodePorts are on the COSMOS control network,
which your laptop cannot reach directly. Forward them through the console. Run this on your
laptop, with <node> the server's name as the console knows it:
ssh -L 30205:<node>:30205 -L 30091:<node>:30091 <username>@console.<sandbox>.cosmos-lab.org
Leave that session open and browse to:
http://localhost:30205/odlux/index.html — SDN-R ODLUX. Its Connect view lists the NETCONFit-dep chart values (see thehttp://localhost:30091/ — the non-RT RIC control panel: the A1 policy types, policies andAdd another -L <port>:<node>:<port> for any other endpoint in the table, for example the MinIO console's
port ($MINIO above).
The SMO on its own manages nothing; its value appears when network elements are connected to it.
On COSMOS today:
| RAN element | Interface | Status |
|---|---|---|
| OCUDU gNB | O1 | An O1 adapter sidecar (ocudu/ocudu_o1_adapter) gives NETCONF configuration and fault management only; performance data stays on OCUDU's websocket metrics |
| Duranta gNB | O1 | Proven through the OAI O1 adapter |
| COSMOS near-RT RIC (O-RAN SC RIC, Docker Compose) | A1 | Not wired. That deployment has no A1 mediator. Use the A1 simulators, or add the RIC's A1 mediator (ric-plt-a1) and its RMR routes |
| LiteOn FlexFi O-RU | M-Plane | Not manageable: the LiteOn has no M-Plane |
| Benetel RAN650 O-RU | M-Plane | See the warning below |
Danger: An SMO that writes O-RU configuration conflicts with the golden O-RU rule. The
commercial O-RUs on COSMOS are shared, stateful devices: each rig restores the O-RU to its
committed golden state before it starts a DU and refuses to start on a mismatch (see
the golden O-RU rule). Any change the
SMO makes to an O-RU — including a power-state toggle from an energy-saving rApp — is drift
that the next user's bring-up will reject, or that silently breaks the downlink of a rig that
does not check. Do not point the SMO at a shared O-RU unless your reservation covers the radio
and you restore its golden state when you finish.
baseline.ndzUse this path to change what the image contains: the Kubernetes version, the cluster CIDRs, the
chart values or the SMO release. It produces the same system as Path A. Five files do the work:
| File | What it does |
|---|---|
smo-build.sh |
Top-level build, five stages; each stage logs to /root/smo-build-logs/; safe to re-run after a failure |
smo-k8s-setup.sh |
Single-node Kubernetes with the COSMOS address plan (--apply: packages, cluster, Calico, Helm; --cluster: cluster and Calico only) |
smo-up.sh |
The boot-time bring-up described in Step A2 (--rebuild forces a cluster rebuild) |
smo-up.service |
The systemd unit that runs smo-up.sh at boot, with a 45-minute start timeout |
README-SMO.md |
A short description of the image, left in /root |
The files are in /root on a server loaded with smo.ndz.
omf load -i baseline.ndz -t <node>
omf tell -a on -t <node>
omf stat -t <node>
Copy the five files into /root on the node — for example from the console, with
<smo-node> a server running smo.ndz:
scp root@<smo-node>:/root/{smo-build.sh,smo-k8s-setup.sh,smo-up.sh,smo-up.service,README-SMO.md} .
scp smo-build.sh smo-k8s-setup.sh smo-up.sh smo-up.service README-SMO.md root@<node>:/root/
ssh root@<node>
bash /root/smo-build.sh
The script prints one == HH:MM:SS <stage> line per stage:
1 kubernetes — smo-k8s-setup.sh --apply: disables swap, loads overlay andbr_netfilter, installs containerd (systemd cgroups, 20 parallel downloads), installs and holdskubeadm init on the CTRL address with pods2 it-dep n-release — clones it-dep at branch n-release into /root/dep and runs the0-setup-helm3.sh).3 images — generates the SMO's image list, pre-pulls all of it with crictl, andmissing images: if any image is absent.4 SMO install — runs 2-install-oran.sh default release. With the images pre-pulled this5 smo-up.service — installs and enables the unit, then runs smo-up.sh once, whichWhat to look for: == … done at the end, preceded by all N pods ready and the endpoint list
from smo-up.sh. Then check the endpoints and pods exactly as in Step A3.
The N-release charts name 72 container images: 29 from nexus3.onap.org, 40 from the
nexus3.o-ran-sc.org proxies, 2 from quay.io and 1 from ghcr.io; none come directly from
Docker Hub. Three of them no longer exist upstream. The build pulls a replacement and tags it
locally under the name the chart expects, so the chart runs unchanged:
| Name the chart expects | Local tag is made from |
|---|---|
nexus3.o-ran-sc.org:10001/minio/minio:RELEASE.2025-06-13T11-33-47Z |
cgr.dev/chainguard/minio:latest (a newer MinIO that runs as non-root) |
nexus3.onap.org:10001/bitnami/redis:7.4.1-debian-12-r2 |
docker.io/bitnamilegacy/redis:7.4.1-debian-12-r2 |
nexus3.onap.org:10001/bitnami/redis-sentinel:7.4.1-debian-12-r2 |
docker.io/bitnamilegacy/redis-sentinel:7.4.1-debian-12-r2 |
A local tag exists only on this node. If it disappears — kubelet image garbage collection under
disk pressure deletes local tags, and so does an image prune — a pod that needs it tries to pull
the chart's name and fails. smo-up.sh restores missing tags at every boot.
Warning: Never run
crictl rmi --pruneon the SMO server. The three local tags are the only
copies of those images; pruning removes them.
To keep a modified build, save it as described in OMF commands.
Because smo-up.service is enabled, the saved image behaves like smo.ndz: it restarts on the
same server and rebuilds on any other.
| Check | Command | Expected |
|---|---|---|
| Bring-up finished | systemctl status smo-up |
active (exited); /var/log/smo-up.log ends with the endpoint list |
| All pods Ready | kubectl get pods -A --no-headers \| grep -vc Completed and the awk filter in Step A3 |
76, and no pods listed as not Ready |
| Endpoints | the curl loop in Step A3 |
200 on every line |
| A1 PMS | curl -s http://$CTRL:30094/a1-policy/v2/policy-types |
a JSON reply |
| Keycloak realm | curl -s -o /dev/null -w '%{http_code}' http://$CTRL:$KC/realms/nonrtric-realm |
200 |
| Web interfaces | SSH tunnel from Step A4 | ODLUX and the control panel load at localhost:30205/odlux/index.html and localhost:30091 |
Close the SSH tunnel on your laptop, then turn the server off from the console:
omf tell -a offh -t <node>
Nothing needs to be saved: the next omf load of smo.ndz gives a fresh SMO. If you connected
the SMO to a shared O-RU, restore that O-RU's golden state before you release the reservation.
| Symptom | Cause | Fix |
|---|---|---|
Pods Evicted; the node reports DiskPressure; cached images disappear |
The root filesystem was not grown to the whole disk. omf save shrinks the root partition and omf load -r 0 does not always grow it back; the SMO then fills a small root, kubelet evicts pods, and image garbage collection deletes cached images |
smo-up.sh grows root at every boot; check df -h / and look for grew root from in /var/log/smo-up.log. If the root is still small, run growpart on its disk and partition and resize2fs on the partition (both are needed), then systemctl restart kubelet and bash /root/smo-up.sh |
ImagePullBackOff on a MinIO or Redis pod |
One of the three local image tags is missing (garbage-collected, or pruned) and the chart's name no longer exists upstream | Run bash /root/smo-up.sh; it restores missing tags. Never run crictl rmi --prune |
Kafka pods in CrashLoopBackOff, logs say Realm does not exist; SMO clients cannot authenticate |
Keycloak runs start-dev with no volume and lost nonrtric-realm when it restarted |
bash /root/smo-up.sh re-runs the keycloak-init job and restarts Kafka and the smo workloads. Check with curl -s -o /dev/null -w '%{http_code}' http://$CTRL:$KC/realms/nonrtric-realm → 200 |
Topology (TEIV) pods show Init:Error or restart during start-up |
Their init step waits for PostgreSQL; before Postgres is ready it fails and Kubernetes retries | Wait; they recover once Postgres is up. The init step also installs netcat from the Alpine CDN at every start, so it keeps failing if the server has no internet access |
smo-up.sh ends with not all pods ready after 20 min: |
Usually one of the causes above, or a slow first boot | Fix the listed pods using the rows above, then run bash /root/smo-up.sh again. bash /root/smo-up.sh --rebuild forces a new cluster and reinstall |
omf load never gets past check-in on an HP Gen9 server |
The server was not fully off when the load started, and missed the network-boot check-in | omf tell -a offh -t <node>, wait until omf stat -t <node> reports it off, then load again. Allow 4–6 min of POST after every power change |
omf load exits 0 but the node does not boot the SMO |
The load timed out; omf load powers the node off and still exits 0 |
Read the summary line: 1 node(s) successfully imaged is success. Keep -o 2400 -b 600 for this image |
| A1 PMS policy-type and RIC lists are empty | No near-RT RIC with an A1 endpoint is registered; the COSMOS near-RT RIC has no A1 mediator | Expected. Use the A1 simulators, or add ric-plt-a1 to the near-RT RIC |
The web interfaces do not load on localhost |
No tunnel, or the tunnel points at the wrong host | Re-open the ssh -L <port>:<node>:<port> session from Step A4 on your laptop and keep it open |
On this wiki
onos-a1t).External
smo-install — the O-RAN SCAuthor(s): COSMOS team · Last verified: 2026-09-25 · Tested image/release: smo-20260925.ndz (O-RAN SC SMO N release, it-dep n-release commit 8f797d6; Kubernetes 1.32.8, Helm 3.18.6; Ubuntu 24.04) · Tags: oran, smo, non-rt-ric, a1, o1, sdn-r, teiv, rapp, onap, kubernetes, o-ran-sc