¶ Real-time host preparation for an O-RAN 7.2 O-DU
This page explains how a COSMOS server is prepared to run the real-time part of a 5G base station:
an O-RAN 7.2 distributed unit (O-DU) or a software radio unit (O-RU) that exchanges fronthaul
packets with a radio on a sub-millisecond deadline. It is the single reference for the host
preparation that every O-RAN 7.2 tutorial in this group relies on: CPU isolation, hyperthread
siblings, interrupt pinning, hugepages, SR-IOV virtual functions, promiscuous mode, Precision Time
Protocol (PTP) timing and the jumbo-frame data plane. For each item it explains why it matters,
how to apply it, and how to prove it took effect.
Read it before your first O-RAN 7.2 tutorial. The concepts behind the fronthaul itself (split 7.2x,
eAxC, timing windows) are on the O-RAN introduction.
¶ Why this matters
A 5G New Radio (NR) cell with 30 kHz subcarrier spacing (numerology μ = 1, the value every COSMOS
7.2 cell uses) divides time into slots of 0.5 ms, each carrying 14 OFDM symbols of about
35.7 µs. In the 7.2x split the O-DU sends the radio unit one set of frequency-domain IQ samples
per symbol, and each packet must arrive inside the radio unit's receive window, a few hundred
microseconds before its air time. Uplink packets from the radio unit must likewise be picked up by
the O-DU inside its own window (Ta4). The 5G SA fundamentals
page explains numerology and slots; the O-RAN introduction
explains the windows.
A general-purpose Linux server does not guarantee anything at that scale. The scheduler moves
threads between cores, the timer tick interrupts every core, network interrupts land wherever the
driver put them, idle cores sleep in deep power states and a powersave governor clocks cores down.
Each of these adds jitter of tens to hundreds of microseconds, which is several symbols. The
consequences seen on COSMOS are concrete:
- Late packets. The O-DU logs
OFH: Real-time timing worker woke up late, skipped 3 symbols,
the radio countsRX_LATE, and a UE that camps never completes random access. - A starved host. A fronthaul O-DU busy-polls its timing and packet threads at 100 % CPU. On a
server without isolated cores it starvessshdwithin about 15–30 s:sshfails with
Connection timed out during banner exchangewhilepingstill answers in 0.2 ms. - A wedged host. On the PREEMPT_RT kernel, busy-poll threads on cores that also carry network
interrupts or kernel housekeeping threads (RCU) starve those threads. The network card's transmit
watchdog fires, the physical function resets into an unrecoverable state,ptp4ldies, and the
server needs a power cycle. - Silent throughput loss. Some mistakes produce no error at all. Worker threads sharing a
physical core with the fronthaul thread, or a fronthaul NIC that is not promiscuous, leave every
counter looking healthy while the cell underperforms or receives nothing.
Every item on this page removes one of these failure modes. None of it is optional on a 7.2 O-DU.
¶ The checklist at a glance
| Item | Why | When it is done | Who does it |
|---|---|---|---|
CPU isolation (isolcpus, nohz_full, rcu_nocbs) |
keep the scheduler, timer tick and RCU callbacks off the real-time cores | once per omf load, then reboot |
you: cosmos-ofh-isolation --apply (or the bundle via ofh-isolation-ensure.sh) |
| Hyperthread siblings / main pool | no worker thread may share a physical core with a fronthaul thread | every config render | the bring-up scripts derive affinities from the host |
Launch under taskset over all CPUs |
isolcpus hides the isolated cores from srsRAN's affinity check |
every launch | the bring-up scripts |
| Performance governor, C-states off | stop frequency scaling and wake-up latency | every boot | the bring-up / host-prep scripts |
NIC queues and IRQ pinning, irqbalance off |
keep network interrupts off busy-polled cores | every bring-up (VF creation resets affinities) | the bring-up scripts |
| Hugepages (DPDK only) | DPDK allocates packet buffers from hugepages | every boot | Duranta bring-up scripts |
SR-IOV VFs bound to vfio-pci, VF MAC set after the bind (DPDK only) |
OAI's xran library drives a virtual function from user space | every boot | Duranta bring-up scripts |
| Promiscuous fronthaul NIC (srsRAN OFH only) | srsRAN's raw socket receives nothing otherwise | every boot | cosmos-fh-ptp-guard on ocudu.ndz, and the bring-up scripts |
PTP (G.8275.1): ptp4l + phc2sys, NTP off, step once then slew |
the O-DU and O-RU must agree on time to within nanoseconds | every boot | the bring-up / host-prep scripts |
| MTU 9000 on the fronthaul NIC | a 100 MHz symbol does not fit a 1500-byte frame | image + every bring-up | cosmos-ifname in the image, and the bring-up scripts |
Figure — a prepared two-socket server: the fronthaul NIC's NUMA node holds a few housekeeping cores (which take all NIC interrupts) and the isolated cores for the fronthaul and L1 workers; each isolated physical core is reserved together with its hyperthread sibling, and the main worker pool lives on the other socket. Hugepages and the vfio-bound virtual functions serve DPDK, and ptp4l/phc2sys discipline the system clock from the NIC's hardware clock.
¶ CPU isolation
¶ What the three kernel parameters do
Three boot parameters together turn a set of cores into dedicated real-time cores:
isolcpus=<list>removes the cores from the scheduler's general load balancing. Nothing runs
there unless it is explicitly pinned there.nohz_full=<list>stops the periodic scheduler tick on those cores while a single task runs.rcu_nocbs=<list>moves the kernel's read-copy-update (RCU) callback processing off those cores.
This prevents RCU starvation: on the PREEMPT_RT kernel, DPDK poll threads on non-isolated cores
starve thercu_preemptthread, anddmesgreports
rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.before
the host wedges.
Because they are kernel command-line parameters, they live in GRUB and take effect only after a
reboot.
¶ Why the set is different on every host
The right cores are the physical cores on the same NUMA node as the fronthaul NIC (so packet
buffers and threads share a memory controller), minus a few left to the operating system, taking
both hyperthread siblings of each physical core. COSMOS servers differ in core count, in whether
simultaneous multithreading (SMT) is on, in how the kernel numbers CPUs across NUMA nodes, and in
which node the NIC sits on. Three hosts give three answers that share no cores at all:
| host | CPU | SMT | NUMA | ice NIC | correct isolcpus |
|---|---|---|---|---|---|
| srv19.osc | Xeon 6126 | ON | 2 nodes, contiguous | numa 0 | 3-11,27-35 |
| srv1-lg1.sb1 | Xeon 6226 | off | 2 nodes, odd/even interl. | numa 1 | 7,9,11,...,23 |
| srv6.rrail | AMD EPYC 9355P | off | 1 node | numa 0 | 3-31 |
Copying srv19's set to srv1-lg1 is not merely suboptimal: CPUs 27–35 do not exist on a 24-CPU
host, and 3–11 straddles both NUMA nodes, the opposite of the intent.
Warning: Never bake
isolcpusinto a disk image. GRUB is part of the.ndz, so a value
captured on one server is silently wrong on the next — the same class of defect as shipping an
-march=nativebinary. COSMOS images (ocudu.ndz,duranta.ndz) deliberately ship with no
isolcpus; you derive it on the host after everyomf load.
¶ Command 1 — cosmos-ofh-isolation: derive, inspect, apply
cosmos-ofh-isolation is installed on the RAN images. It reads the live topology: it finds the
E810 (ice) fronthaul NIC (or the one you name with --nic), reads that NIC's NUMA node and the
node's CPU list, groups logical CPUs into physical cores through
/sys/devices/system/cpu/cpu*/topology/thread_siblings_list, leaves three physical cores to the
OS (--housekeep N changes that), and isolates the rest together with their siblings. It refuses
to ever isolate CPU 0.
Without arguments it only reports; with --apply it edits GRUB and runs update-grub. The
reference procedure after loading an image:
cosmos-ofh-isolation # inspect what THIS host needs (read-only)
cosmos-ofh-isolation --apply # edit GRUB
reboot
cosmos-ofh-isolation # must say "already correct on this host"
Options: --nic NAME (force the FH NIC), --housekeep N (OS cores, default 3), --json. The
report states one of three statuses: STATUS: not isolated yet -- rerun with --apply, then reboot.,
STATUS: already correct on this host., or STATUS: differs (...). The last means the running
kernel carries a different set; the tool leaves a hand-tuned set alone unless you re-apply.
The tutorials run the same command from the console, naming the fronthaul interface. On the OCUDU
O-RU tutorial's O-DU host:
ssh root@srv1-lg1.sb1.cosmos-lab.org
cosmos-ofh-isolation --nic DATA1a --apply # writes GRUB, then:
reboot
grep -o 'isolcpus=[^ ]*' /proc/cmdline
isolcpus=7,9,11,13,15,17,19,21,23
On a two-socket Xeon Gold 6126 host with SMT on (the Foxconn tutorial's O-DU host), the same
procedure gives both siblings of nine physical cores:
ssh root@<du-node> 'cosmos-ofh-isolation --nic DATA1a --apply' && ssh root@<du-node> 'systemctl reboot'
ssh root@<du-node> 'cat /sys/devices/system/cpu/isolated; timedatectl | grep NTP; pgrep -a phc2sys'
3-11,27-35
NTP service: inactive
3187 phc2sys -s DATA1a -w -n 24 -m
What to look for: /sys/devices/system/cpu/isolated lists the set the tool derived. An empty
line means the reboot did not pick up the new command line.
--apply edits GRUB carefully, because a mistake there can leave a remote server unbootable:
/etc/default/grubcan hold more than oneGRUB_CMDLINE_LINUX=line. The file is sourced by the
shell, so the last one wins; the tool edits only that line, by number, and keeps a
time-stamped backup.- If the edit would drop
console=ttyS*(the serial console — the only recovery path on a remote
server that does not boot), the tool reverts from the backup and stops. isolcpusmay carry leading flags, e.g.isolcpus=managed_irq,domain,5-31. The tool preserves
them; droppingmanaged_irqwould push managed device interrupts back onto the isolated cores.- After any manual restore of
/etc/default/grub, re-runupdate-grub, orgrub.cfgkeeps the
old command line. Check what will actually boot withgrep vmlinuz /boot/grub/grub.cfgbefore
rebooting.
¶ Command 2 — /root/ofh-isolation-ensure.sh <nic>: the runtime check
/root/ofh-isolation-ensure.sh ships on the current ocudu.ndz and duranta.ndz. It is the check-and-apply step the tutorial bundles and bring-up flows run
before they start a DU. It asks cosmos-ofh-isolation --nic <nic> --json which CPUs this host
should isolate, compares that set with the running kernel's /sys/devices/system/cpu/isolated,
and then either reports that the host is ready or applies the set to GRUB and reports that a reboot
is needed. It prints one line:
ssh root@srv1-lg1 /root/ofh-isolation-ensure.sh DATA1a
ssh root@srv2-lg1 /root/ofh-isolation-ensure.sh DATA1a
REBOOT-NEEDED want=7,9,11,13,15,17,19,21,23 have=none
Reboot each server that printed REBOOT-NEEDED (ssh root@srv1-lg1 systemctl reboot), wait for
it, and re-run the command: it must now print ISOLATED 7,9,11,13,15,17,19,21,23. The Ansible mode
of the ocudu-72 bundle runs the same logic as a task and reboots only when it sees
REBOOT-NEEDED, so a second run on an already-prepared host costs nothing.
¶ Which command to use
cosmos-ofh-isolation |
/root/ofh-isolation-ensure.sh <nic> |
|
|---|---|---|
| Purpose | derive and explain the per-host set; write it to GRUB with --apply |
decide whether this host is ready to start a DU |
| Default action | report only (read-only) | apply the set if the running kernel differs |
| Output | a human-readable report (or --json) with the housekeeping and isolated CPUs |
one line: ISOLATED <set> or REBOOT-NEEDED want=<set> have=<set> |
| When | once after each omf load, followed by a reboot; any time you want to inspect |
at the start of every bring-up (the bundles call it) |
| Reboot | always required after --apply |
required only when it printed REBOOT-NEEDED |
Both use the same derivation, so they never disagree about the set. The housekeeping list that
cosmos-ofh-isolation --json reports (housekeep_cpus) is also what the bring-up scripts use as
the target for network interrupts (see below).
¶ Removing isolation
Isolation stays until you remove it or reload the image. Remove it before running a gNB that is
not pinned to isolated cores on the same server, such as the
OCUDU over-the-air tutorial:
that gNB then gets only the non-isolated CPUs and floods Real-time failure in RF: late under
traffic (measured: DL 4–59 Mbit/s with isolation, 59.7 / UL 59.5 without, same image and radio). On
a 24-core host it stalls.
ssh root@srv1-lg1 'sed -i -E "s/ ?(isolcpus|nohz_full|rcu_nocbs)=[^ \"]*//g" /etc/default/grub && update-grub && systemctl reboot'
¶ Hyperthread siblings and the main-pool rule
With SMT on, each physical core appears as two logical CPUs (siblings) that share the core's
execution units and caches. A busy thread on one sibling slows the other. Isolation therefore
reserves complete physical cores: on srv18.osc and srv19.osc, isolcpus = 3-11,27-35 is nine
physical cores, not eighteen independent ones (cpu3 <-> cpu27, cpu4 <-> cpu28, …,
cpu11 <-> cpu35).
An O-DU then places its threads in three groups. In OCUDU (srsRAN) the configuration names them
directly: ru_cpus for the radio-unit timing thread, txrx_cpus for the Open Fronthaul
transmit/receive busy-poll, and main_pool_cpus for the upper-PHY and upper-layer worker pool. The
first two are hard real time and belong on isolated cores near the NIC. The main pool is not hard
real time and may run on the second socket.
Warning:
main_pool_cpusmust avoid the hyperthread siblings of the isolated RU and
fronthaul cores, not just the cores themselves. Example: withru_cpus3,4 and OFHtxrx_cpus
5–8 on isolated cores, amain_pool_cpusof9..47includes 27–35, the siblings of those very
cores, and puts worker threads on the same physical cores as the fronthaul path. No counter
reports it:rx_on_time=2400,rx_late=0,nof_skipped_symbols=0all stay clean. Give the
main pool the other socket, heremain_pool_cpus: 12-23,36-47.
To check a configuration by hand, look up the sibling of every CPU in ru_cpus and txrx_cpus and
confirm it does not appear in main_pool_cpus:
cat /sys/devices/system/cpu/cpu3/topology/thread_siblings_list
On a host without SMT (the sb1 Xeon Gold 6226 servers, the AMD EPYC 9355P) every CPU is its own
physical core and the rule is satisfied automatically. The COSMOS configuration renderers
(ocudu-72_lib.py render-gnb, protoru-render, and oai72-host.py for Duranta) derive all of
these affinities from the host's isolated set, so the values differ per server by design.
Two more srsRAN rules follow from isolation:
- Launch under
tasksetover all CPUs.isolcpusremoves the isolated CPUs from every
process's default affinity mask, and srsRAN validates its configured cores against that mask. A
gNB started normally fails withInvalid CPU core selected '8'. The bring-up scripts launch it
insidetaskset -c 0-$(($(nproc --all)-1)). Usenproc --all: plainnproccounts only the
caller's mask, sotaskset -c 0-$(($(nproc)-1))shrinks to the non-isolated CPUs. - The pool size is a separate setting.
Maximum PUSCH and SRS concurrency (i.e., 10) exceeds the number of main pool threads (i.e., 9)meansexpert_execution.threads.main_pool.nof_threadsis
too small; adding CPUs tomain_pool_cpusalone does not change the thread count.
¶ Interrupts and NIC queues
The Intel E810 (ice driver) creates one transmit/receive queue per CPU and affines queue N's
interrupt to CPU N. OAI's xran library runs its fronthaul threads as SCHED_FIFO busy loops on
its pinned cores. On the PREEMPT_RT kernel, a queue whose interrupt handler lives on one of those
cores is never serviced: its transmit completions are never cleaned, the kernel's NETDEV WATCHDOG
fires after 5 s, and the physical function ends in tx_timeout recovery ... unrecoverable, taking
ptp4l with it. The stalled queues are exactly the xran cores (for example 5, 7 and 9).
Pinning the interrupts afterwards is not enough on its own. A poll (NAPI) instance already scheduled
on a busy-polled core stays stranded there. The robust order is:
- Reduce each NIC to a few queues before the virtual functions exist (
ethtool -Lis refused
once VFs are bound). - Stop
irqbalance, which would otherwise move interrupts back. - Pin every NIC interrupt to the housekeeping CPUs (the ones
cosmos-ofh-isolationdid not
isolate), after VF creation, because creating VFs resets the affinities. - Steer all kernel transmit traffic on the fronthaul port to queue 0 (XPS).
The OCUDU bring-up (ocudu-72_bringup.sh) pins the fronthaul interrupts like this:
ISO=$(grep -oE 'isolcpus=[^ ]+' /proc/cmdline | sed -E 's/isolcpus=//; s/(managed_irq|domain|nohz),?//g')
HK=$(cosmos-ofh-isolation --json 2>/dev/null | python3 -c 'import sys,json; print(json.load(sys.stdin).get("housekeep_cpus",""))' 2>/dev/null)
[ -n "$HK" ] || HK="0-1"
systemctl stop irqbalance 2>/dev/null
n=0; for irq in $(grep -E "$PTP_IF|ice-" /proc/interrupts | awk -F: '{print $1}' | tr -d ' '); do
echo "$HK" > /proc/irq/$irq/smp_affinity_list 2>/dev/null && n=$((n+1)); done
echo " isolated: ${ISO:-none}; pinned $n fronthaul IRQs to CPUs $HK"
The Duranta bring-ups (liteon-du-bringup.sh, benetel-bringup.sh) do the fuller version: they
shrink the PF's queues before creating VFs, pin the interrupts of every NIC (the Broadcom bnxt
data NICs stalled the same way), refuse to continue if any NIC interrupt is still on an isolated
core, and set XPS. The final assertion is the line to look for:
pinned <n> NIC IRQs to CPUs <housekeeping>; NIC IRQs still on isolated cores: 0
The Duranta image also blacklists the unused RDMA drivers bnxt_re and irdma
(/etc/modprobe.d/cosmos-no-rdma.conf): a bnxt_re firmware stall at boot holds the kernel's
network lock (rtnl) and leaves ethtool and ip blocked in D-state.
Verify: for each fronthaul interrupt, cat /proc/irq/<irq>/effective_affinity_list must show
only housekeeping CPUs.
¶ CPU frequency and idle states
With the powersave governor an idle core runs at about 750 MHz and needs time to ramp up, which
is enough to miss fronthaul deadlines (and, in OAI's xran, to trip an assertion). Deep idle states
(C-states) add wake-up latency. Every bring-up sets the performance governor; the ProtO-RU host
preparation also disables C-states, as ProtO-RU's hardware requirements ask:
for g in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do echo performance > "$g" 2>/dev/null; done
for s in /sys/devices/system/cpu/cpu*/cpuidle/state[1-9]/disable; do echo 1 > "$s" 2>/dev/null; done
Neither setting survives a reboot, which is why the scripts re-apply them every time.
¶ Hugepages (DPDK stacks)
The Data Plane Development Kit (DPDK), which OAI's fhi_72 transport uses, allocates its packet
buffers from hugepages: large, pinned memory pages that avoid TLB misses and page faults on the
packet path. The 24.04 real-time base image mounts /dev/hugepages for 1 GB pages only, and DPDK
here needs a 2 MB hugetlbfs mount; without it DPDK panics with
no mounted hugetlbfs found for that size or EAL: Cannot get hugepage information. The Duranta
bring-ups set this up on every boot:
mkdir -p /dev/hugepages2M; mountpoint -q /dev/hugepages2M || mount -t hugetlbfs -o pagesize=2M nodev /dev/hugepages2M
for d in /sys/devices/system/node/node[0-9]*; do echo 12288 > $d/hugepages/hugepages-2048kB/nr_hugepages 2>/dev/null || true; done
find /dev/hugepages2M -maxdepth 1 -type f -delete 2>/dev/null || true
find /dev/hugepages -maxdepth 1 -type f -delete 2>/dev/null || true
HP=$(awk '/^HugePages_Total/{print $2}' /proc/meminfo)
echo " HugePages_Total = $HP"
Verify: HugePages_Total in /proc/meminfo is greater than 0. OCUDU does not use DPDK on
COSMOS and needs no hugepages.
¶ SR-IOV virtual functions and vfio-pci (DPDK stacks)
Single-root I/O virtualisation (SR-IOV) lets one physical NIC port (the physical function, PF)
present several virtual functions (VFs), each a separate PCI device. OAI's xran library drives a VF
directly from user space through DPDK; for that the VF is detached from the kernel driver and bound
to vfio-pci, which gives a user-space process safe access to the device. VFs do not survive a
reboot, so the Duranta bring-ups create them every time:
echo 4 > /sys/class/net/$PF/device/sriov_numvfs; sleep 3
ip link set $PF mtu 9000; ip link set $PF up; sleep 1
echo vfio-pci > /sys/bus/pci/devices/$VF_PCI/driver_override
echo $VF_PCI > /sys/bus/pci/devices/$VF_PCI/driver/unbind 2>/dev/null || true
echo $VF_PCI > /sys/bus/pci/drivers/vfio-pci/bind 2>/dev/null || true
sleep 2
# The bind WIPES the VF MAC; set it afterwards or DPDK invents an f2:99:... address.
for try in 1 2 3; do
ip link set $PF vf 0 mac $DU_MAC vlan 4 spoofchk off trust on 2>/dev/null; sleep 2
got=$(ip link show $PF | awk '/vf 0/{print $4}')
[ "$got" = "$DU_MAC" ] && break
done
Warning: Set the VF MAC after the
vfio-pcibind. The bind wipes the VF's MAC, DPDK then
invents one (the same bogusf2:99:ef:ef:10:44every run, because its generator is unseeded), and
in O-DU mode OAI ignores thedu_addrin its configuration. The radio unit sends all uplink to
the MAC it was configured with, so the O-DU receives nothing:[o-du 0][rx 0 pps 0 ...]with every
error counter at 0. The VLAN (4) also comes from the VF's port VLAN, not from the OAI config.
Verify: the bring-up prints vf0 MAC = <DU MAC> (vf0 MAC 00:11:22:33:44:66 on the COSMOS
rigs), and the O-DU's first status line shows a non-zero rx ... pps.
After several warm DU restarts (kill the DU, recreate VFs, rebind) the E810 VFs can wedge: the
bring-up aborts at its VF-MAC gate and dmesg shows iavf ... Reset never finished, VF disabled.
Recover with a real power cycle of the DU host, not a soft reboot. Do not add the DPDK device
argument auto_reset=1 as a remedy: with it, OAI's fronthaul stalls at start-up (first_call).
OCUDU (srsRAN) does not use DPDK on COSMOS. Its Open Fronthaul uses a raw socket on the PF itself
(network_interface: DATA1a), so it needs no VFs, no vfio-pci and no hugepages — but it needs
promiscuous mode.
¶ Promiscuous mode (srsRAN Open Fronthaul)
srsRAN's fronthaul receiver is an AF_PACKET raw socket bound to the eCPRI ethertype, and it puts
the interface into promiscuous mode only when its own configuration says so. On COSMOS the fronthaul
frames arrive VLAN-tagged on a NIC that does hardware VLAN offload, and unless the interface is
promiscuous the O-DU's socket never sees them.
ip link set DATA1a promisc on
Warning:
tcpdumpis not evidence here. A capture tap sits before protocol demultiplexing,
sotcpdump -i DATA1ashows thousands of correct frames per second from the radio while the
O-DU receives exactly zero. The symptom is a UE that camps with excellent signal and never
registers, with no PRACH ever detected, on any radio vendor.
The one honest counter is the O-DU's own receive count. Enable the fronthaul metrics in the gNB
configuration:
metrics:
enable_log: true
layers:
enable_ru: true
ether_rx … rx_bytes = 0 means not received, and no PRACH, timing or compression change can fix
that. With promiscuous mode on, rx_bytes climbs and nof_missed_prach_occasions is 0.
On ocudu.ndz the boot-time cosmos-fh-ptp-guard service sets
promisc on, and the bring-up scripts set it again and print DATA1a promiscuous: yes. ProtO-RU,
built on the same code base, needs it on both hosts (enable_promiscuous: true in its
configuration plus the ip link command).
¶ Timing: PTP, the system clock and slewing
¶ What must be synchronised
In the 7.2 split the radio unit transmits each symbol at an absolute time, so the O-DU and O-RU
must share one time base to within tens of nanoseconds. That is the fronthaul's synchronisation
plane (S-plane). On COSMOS a GPS-disciplined grandmaster feeds the fronthaul switches using the
ITU-T G.8275.1 telecom profile (PTP over Layer 2, domain 24); the switch acts as a boundary clock,
and both the O-DU server and the O-RU follow it (the O-RAN LLS-C3 configuration).
On the server, two daemons from linuxptp do the work:
ptp4llocks the NIC's hardware clock (the PTP hardware clock, PHC) to the grandmaster.phc2syscopies the PHC into the system clock (CLOCK_REALTIME), which is what the O-DU's
timing thread reads.
NTP must be off, or systemd-timesyncd keeps pulling CLOCK_REALTIME back towards its own idea of
UTC and fights phc2sys.
The images ship the G.8275.1 configuration as /etc/linuxptp/ptp4l-g8275.conf:
[global]
dataset_comparison G.8275.x
G.8275.defaultDS.localPriority 128
maxStepsRemoved 255
logAnnounceInterval -3
logSyncInterval -4
logMinDelayReqInterval -4
announceReceiptTimeout 3
serverOnly 0
clientOnly 1
G.8275.portDS.localPriority 128
ptp_dst_mac 01:1B:19:00:00:00
network_transport L2
domainNumber 24
time_stamping hardware
tx_timestamp_timeout 100
summary_interval 0
[DATA1a]
¶ Step once, then slew
After a reboot the system clock can be far from PTP time: 37 s if it is on UTC and the PHC on TAI,
or about 144 ms after an ordinary reboot. Slewing (gradually adjusting the clock rate) would take
around 25 minutes to close even the smaller gap, during which the cell is unusable. Stepping
(jumping the clock) closes it at once, but a step while the DU runs moves every deadline the DU has
computed in the same instant. The COSMOS scripts therefore do both, in order: step once before the
DU starts, then slew only. In steady state phc2sys must never step; the bring-up scripts treat a
steady-state step as a cause of host wedges and refuse to start the DU until the offset has
converged.
The OCUDU bring-up and the ProtO-RU host preparation run:
systemctl stop systemd-timesyncd 2>/dev/null; timedatectl set-ntp false 2>/dev/null
setsid chrt -f 10 ptp4l -f "$PTP_CFG" -i "$PTP_IF" -m >/root/ptp4l.log 2>&1 </dev/null &
# step phase: -S 1 allows the one correction (e.g. the 37 s left by an earlier -O 0 run)
setsid chrt -f 10 phc2sys -s "$PTP_IF" -w -n 24 -m -S 1 >/root/phc2sys-step.log 2>&1 </dev/null &
# steady state: slew only (a step here is what wedges COSMOS nodes)
setsid chrt -f 10 phc2sys -s "$PTP_IF" -w -n 24 -m >/root/phc2sys.log 2>&1 </dev/null &
(The script polls each phase's log until the offset is under 100 µs and then 10 µs, and kills the
step-phase phc2sys before starting the slewing one.) The Duranta bring-ups use a step threshold
below the expected offset and pin the daemons to housekeeping CPUs:
setsid chrt -f 10 taskset -c $PHC_CPU phc2sys -s $PF -n 24 -O 0 -S 0.01 -m >/root/phc2sys-step.log 2>&1 </dev/null &
setsid chrt -f 10 taskset -c $PHC_CPU phc2sys -s $PF -n 24 -O 0 -S 0.0 -F 0.0 -m >/root/phc2sys.log 2>&1 </dev/null &
-S 0.0 -F 0.0 disables stepping entirely, including at start-up of that instance.
¶ Which timescale: -w or -O 0
The G.8275.1 PHC runs on the TAI timescale, 37 s ahead of UTC. phc2sys -w applies the UTC offset
that ptp4l learned, so CLOCK_REALTIME ends up on UTC; -O 0 applies no offset and leaves it on
TAI. The two stacks' scripts differ, and each is what its verified rigs use:
- OCUDU / ProtO-RU scripts use
-w. The Benetel O-RU keeps UTC. With-O 0the O-DU's clock
is 37 s ahead of the radio's, the two derive System Frame Numbers 3700 frames apart, and every
PRACH lands in a frame the O-DU has no context for. The gNB also logs
OFH woke up late … skipped N symbols. - Duranta scripts use
-O 0with the Benetel, LiteOn andnr-orurigs as documented in their
tutorials.
Do not change a script's timescale in isolation; if you write your own bring-up, match the stack's
verified script.
¶ Verify PTP
The bring-up output shows the lock and the two phases:
ptp4l locked: ptp4l[2139.061]: rms 3149 max 4739 freq -6742 +/- 390 delay
stepped: |offset| = 148 ns
slewing: |offset| = 1 ns (UTC timescale)
and, in steady state on a well-locked host:
ptp4l: ptp4l[128.920]: rms 6 max 15 freq -1134 +/- 9 delay 246 +/-
phc2sys: phc2sys[128.586]: CLOCK_REALTIME phc offset 25 s2 freq -91473 delay 4
rms in single-digit nanoseconds and a phc2sys offset of tens of nanoseconds in state s2
(locked) are what you want. To query a running ptp4l, use pmc -u -b 0 -d 24 "GET PORT_DATA_SET"
and look for portState SLAVE; pmc needs -d 24, and without it the query is silently ignored
on this domain, which looks exactly like a dead daemon.
Warning: Run exactly one
ptp4lper host. Two instances on annr-oruhost produce
millions of early and late fronthaul errors. Check withpgrep -a ptp4lbefore
starting your own.
¶ The PTP management-message storm
A single-port ptp4l re-transmits the PTP management messages it receives, and the fronthaul
switch re-floods them to every port. With two such hosts on the fabric the messages circulate
indefinitely, the management traffic grows to tens of thousands of messages per second, and a
host's ptp4l cannot reach the SLAVE state. Each ptp4l host therefore drops its own outgoing
management messages:
tc qdisc add dev DATA1a clsact
tc filter add dev DATA1a egress protocol 0x88f7 prio 1 u32 match u8 0x0d 0x0f at 0 action drop
ocudu.ndz ships this as the cosmos-fh-ptp-guard service.
Check it with tc -s filter show dev DATA1a egress.
¶ The jumbo-frame fronthaul (MTU 9000)
A 100 MHz, 273-PRB symbol for one antenna carries several kilobytes of IQ data even after block
floating-point compression, which does not fit a standard 1500-byte Ethernet frame. The COSMOS data
plane runs at MTU 9000: the images' cosmos-ifname package sets it on the data interfaces, and every
bring-up re-asserts it on the fronthaul port (ip link set "$PTP_IF" mtu 9000). See the
data-plane MTU reference for how MTU is assigned per subnet.
Radio units have their own jumbo setting: the LiteOn's jumboframe resets to 0 on every RU boot and
is re-applied by the LiteOn bring-up (see the O-RU hardware page).
¶ Putting it together
Once per image load (you):
omf loadthe image and power the server on.cosmos-ofh-isolation --nic <fronthaul NIC> --apply, then reboot (or run
/root/ofh-isolation-ensure.sh <nic>and reboot if it saysREBOOT-NEEDED).- Confirm
/sys/devices/system/cpu/isolatedmatches the derived set.
Every bring-up (the scripts): performance governor → fronthaul NIC (queues, VFs and MAC for
DPDK stacks, MTU 9000, promiscuous for srsRAN) → interrupts to housekeeping → hugepages (DPDK) →
NTP off, ptp4l, phc2sys step then slew → the core → the DU, pinned from the host's isolated set
and launched under taskset.
| Script (image / bundle) | Stack | Host-prep steps it performs |
|---|---|---|
ocudu-72_bringup.sh (ocudu-72 bundle) |
OCUDU + commercial O-RU | governor, MTU, promiscuous, IRQs, PTP step/slew, then core and gNB |
/root/proto-ru/protoru-host-prep.sh (ocudu.ndz) |
OCUDU + ProtO-RU, both hosts | governor, C-states, MTU, rings, promiscuous, socket buffers, IRQs, PTP step/slew |
/root/liteon-du-bringup.sh, /root/benetel-bringup.sh (duranta.ndz) |
Duranta + commercial O-RU | governor, NIC queues, VFs, vfio-pci, VF MAC, IRQs, XPS, hugepages, PTP step/slew, then core and DU |
/root/du-oru-bringup.sh, /root/oru-bringup.sh (duranta.ndz) |
Duranta + nr-oru |
the same host prep on the O-DU and on the soft O-RU host |
¶ Verification checklist
| Check | Command | Expected |
|---|---|---|
| Isolation active | cat /sys/devices/system/cpu/isolated |
the set cosmos-ofh-isolation derived (non-empty) |
| Host ready | /root/ofh-isolation-ensure.sh DATA1a |
ISOLATED <set> |
| No NIC IRQ on isolated CPUs | cat /proc/irq/<irq>/effective_affinity_list for the fronthaul IRQs |
housekeeping CPUs only |
| Governor | cat /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor |
performance |
| Hugepages (DPDK) | grep HugePages_Total /proc/meminfo |
greater than 0 |
| VF MAC (DPDK) | bring-up output | vf0 MAC = <DU MAC> |
| Promiscuous (srsRAN) | ip -br link show DATA1a |
flags include PROMISC |
| NTP off | timedatectl \| grep NTP |
NTP service: inactive |
| PTP locked | pmc -u -b 0 -d 24 "GET PORT_DATA_SET" |
portState SLAVE |
| Clock disciplined | tail -1 /root/phc2sys.log |
offset tens of ns, state s2 |
One ptp4l |
pgrep -a ptp4l |
exactly one process |
| O-DU actually receives (srsRAN) | gNB log, OFH sector metrics |
rx_total > 0, rx_late=0 |
¶ How this maps onto COSMOS
Eight servers carry an Intel E810 NIC on the fronthaul fabric with DPDK, PTP and VLANs 4 and 5, and
can drive any O-RU: srv17.osc, srv18.osc, srv19.osc, srv20.osc, srv6.rrail, srv7.rrail,
srv1-lg1.sb1 and srv2-lg1.sb1. They span three CPU platforms (2× Xeon Gold 6126 with SMT,
2× Xeon Gold 6226 without SMT, 1× AMD EPYC 9355P), which is why every value on this page is derived
per host. Fronthaul interfaces are named DATA1a on most hosts; Duranta configurations on sb1 may
refer to the same E810 port as ens5f0.
The tutorials that depend on this page, in learning-path order:
- OCUDU with a soft O-RU (ProtO-RU) and
Duranta with a soft O-RU (nr-oru) - OCUDU over a commercial O-RU,
Duranta over a commercial O-RU and
OCUDU over the Foxconn O-RU
¶ Common misconceptions and pitfalls
- "The image already has the right
isolcpus." COSMOS RAN images deliberately carry none; a
reloaded server has no isolation until you apply it and reboot. - "Isolating the cores is enough." The worker pool must also avoid their hyperthread siblings,
and nothing reports the contention. - "
tcpdumpshows the frames, so the DU gets them." Not for srsRAN's raw socket; read
ether_rx rx_bytes. - "A clean fronthaul means the radio works." Timing counters at zero say nothing about the radio
unit's own configuration; see the golden O-RU rule on the O-RU hardware page.
¶ Where to go next
- O-RAN introduction — the fronthaul planes, eAxC and timing windows this preparation serves.
- O-RU hardware — the radios at the other end of the fronthaul, and the golden O-RU rule.
- OCUDU with a soft O-RU — the first 7.2 tutorial, with counters on both ends of the link.
- COTS UE: the Quectel modem — the UE side of every over-the-air tutorial.
¶ Further reading
- Linux kernel parameters —
isolcpus,nohz_full,rcu_nocbsandirqaffinity. - linuxptp —
ptp4l,phc2sysandpmc. - DPDK guides — hugepages,
vfio-pcibinding and the Inteliavfpoll-mode driver. - O-RAN ALLIANCE specifications — the Open Fronthaul (WG4) control, user and synchronisation plane specification, including the LLS-C3 timing configuration.
- OAI 7.2 fronthaul tutorial — OAI's own host requirements for
fhi_72. - srsRAN Project documentation — the O-DU's
expert_executionaffinity settings and Open Fronthaul configuration.
Author(s): COSMOS team · Last reviewed: 2026-09-24 · Tags: oran, 7.2, fronthaul, real-time, isolcpus, hyperthreading, irq, hugepages, sr-iov, vfio, dpdk, promiscuous, ptp, linuxptp, mtu, concepts