Bring up a private LTE or 5G-SA network on COSMOS — core, base station, and a bank of emulated
UEs — then sweep the UE count from 1 to 64, measure how many actually register, and measure what
the link carries. The same experiment is given three ways: command by command, as an Ansible
sweep, and as a Jupyter notebook — and the base station can run on an osc or an sb1 server,
so you can see what the server's CPU does to the result.
Most cellular tutorials stop at "one UE attached." That proves the plumbing works but says nothing
about behaviour under load, which is what a testbed is for. This tutorial runs a controlled
cell — every element is yours, so the ground truth is known — and asks a quantitative question:
as you add UEs, where does it stop working?
The rig has three parts. One host runs ltemme (the unified EPC + 5GC core) and lteenb (the eNB
for LTE, gNB for 5G). A second host runs lteue, Amarisoft's UE simulator, which emulates up to 64
subscribers against a real USRP. The two radios meet over the air, not over the network.
Because every component is under your control, this rig is also the reference against which
passive decoders are validated: you know exactly which grants were scheduled, so you can measure a
receiver's recall instead of guessing at it.
The headline result on the reference rig is that the cell is not the limit: all 64 provisioned
subscribers attach on both 4G and 5G. What the sweep really teaches is how to tell a genuine
capacity limit apart from the several ways a measurement of one can be fabricated — the
Troubleshooting section is a catalogue of the ones that bit us.
After completing this tutorial you will be able to:
power_on step that is| Difficulty | Advanced |
| Estimated time | 30 min for a short sweep; ~20 min for the full 1→64 sweep |
| Domain / sandbox | sb1 UE bank + a RAN server in osc (amari_rig: osc, default) or sb1 (amari_rig: sb1) |
| Topic group | Cellular (4G / 5G / O-RAN) |
| Last verified | 2026-09-13 — three-mode parity PASS for 4G and 5G on both RAN servers (short sweep, 1 and 4 UEs). The full 1→64 sweep below is from 2026-08-28. |
Background knowledge
omf node control — see OMF commands.Account & access
This tutorial cannot be run by two people at once. Amarisoft on COSMOS is licensed through a
floating licence server (license.orbit-lab.org, a systemd unit on mgmt1) with one seat per
tag for the entire testbed. All five tags are provisioned —gNB,eNB,5gC,4gC,
ue— so no component is "unlicensed"; there is simply one of each. If someone else is running
Amarisoft anywhere, yours will not start, and the error is not a friendly "in use" message: the
client segfaults. See Troubleshooting.
Devices / nodes
| Resource | Role | Qty | Notes |
|---|---|---|---|
srv17.osc or srv1-lg1.sb1 |
Core (ltemme) + base station (lteenb) |
1 | picked by amari_rig (osc / sb1); Amarisoft under /opt/amarisoft |
sdr1-in3.sb1 |
Emulated UE bank (lteue) |
1 | needs UHD + trx_uhd.so |
USRP N310 @ 10.39.3.1 |
Base-station radio | 1 | reached over the DATA plane |
USRP N310 @ 10.38.2.1 |
UE-side radio | 1 | rx_antenna RX2, not TX/RX |
Hosts are named, not resolved by capability — see Topology. Both RAN servers drive the same
two radios and the same UE bank, so switching amari_rig changes only the server.
Disk images
| Image | Load onto | Provides |
|---|---|---|
amarisoft-20260904.ndz |
the RAN server (srv17.osc or srv1-lg1.sb1) |
lteenb, ltemme (vendor install, in-dir OpenSSL) |
lte5gue-20260905.ndz |
sdr1-in3.sb1 |
lteue, UHD 4.9, trx_uhd.so |
Nothing else has to be on the nodes. Every config the run uses — core, base station, UE bank —
is rendered from the tutorial package on the console and pushed to the nodes under a
cosmos-tut- prefix, along with the tutorial's own helper scripts
(/opt/cosmos-tutorials/amarisoft/). Verified 2026-09-13 on a freshly imaged srv1-lg1.sb1.
(Before that fix the tutorial depended on a UE-config generator that had only ever existed on one
node and was never committed, so it could not run from a clean image.)
Software components
| Component | Version | Source |
|---|---|---|
Amarisoft lteenb / ltemme / lteue |
2026-06-12 | preinstalled in the images |
| UHD | 4.9.0.0 | preinstalled (must match on both hosts) |
| Tutorial package | cosmos-experiments-amarisoft 0.2.0 |
/usr/share/cosmos-experiments/amarisoft/ on every console |
Spectrum / RF / special
Over the air inside the sandbox, on cabled USRPs. 5G-SA runs band n41 TDD at 2593.35 MHz,
40 MHz / 106 PRB, 30 kHz SCS, PCI 500. LTE runs a 10 MHz / 50 PRB carrier. Both are low-power and
cabled; do not re-point them at a live commercial band.
console.<domain> (you run everything from here)
|
| ssh root@… ssh root@…
v v
┌───────────────────┐ ┌──────────────────────┐
│ srv17.osc (osc) │ │ sdr1-in3.sb1 │
│ ltemme (core) │ │ lteue (64 UEs max) │
│ lteenb (eNB/gNB)│ │ │
│ — or — │ │ │
│ srv1-lg1.sb1 (sb1)│ │ │
└─────────┬─────────┘ └──────────┬───────────┘
│ DATA plane │ DATA plane
v v
USRP N310 10.39.3.1 ) ) ) air ( ( ( USRP N310 10.38.2.1
The two USRPs are connected over the air; the hosts never exchange user traffic directly. The
control path (your ssh) and the radio data path are different networks, and keeping them
separate is the single most important property of this rig — see Troubleshooting.
Note that the default rig spans two subdomains. That is why this tutorial does not use
OMF_NODES: cosmos-inventory.py enumerates omf stat for one domain at a time, so no single
console can resolve both hosts, and Amarisoft is licensed software pinned to specific machines.
Instead the hosts come from amarisoft_params.yml — amari_rig selects the RAN server and
amari_ue_host the UE bank — and the playbook adds them at run time. The shipped inventory/
directory holds only connection settings; cosmos-tutorial picks it up automatically.
ssh <username>@console.sb1.cosmos-lab.org
ssh root@srv17.osc.cosmos-lab.org 'cat /.omf-image-name'
ssh root@sdr1-in3.sb1.cosmos-lab.org 'cat /.omf-image-name'
amarisoft-20260904.ndz
lte5gue-20260905.ndz
/opt/cosmos-tutorials/amarisoft/amari_preflight.sh): radio subnet on the right NIC at MTU--fix repairs what is safely repairable for thesrv17 (ran): == ran-host: 18 pass, 0 fixed, 0 warn, 0 FAIL
sdr1-in3 (ue): == ue-host: 16 pass, 0 fixed, 0 warn, 0 FAIL
0 FAIL stops the run. A rig that fails preflight still runs — it just~/cosmos-experiments → system):cosmos-tutorial which amarisoft
ls $(cosmos-tutorial which amarisoft)
/usr/share/cosmos-experiments/amarisoft
amarisoft.description.yml amarisoft_parity.sh amarisoft.yml make_core_subscribers.py templates
amarisoft_lib.py amarisoft_render.py ansible.cfg node tools
amarisoft_manual.sh amarisoft_rig.yml data __pycache__
amarisoft_nbcheck.py amarisoft_teardown.yml inventory roles
amarisoft_params.yml amarisoft_tutorial.ipynb
apt-cache policy cosmos-experiments-amarisoft).amari_rig setting.sudo apt update && sudo apt install cosmos-experiments-amarisoft
cosmos-tutorial copy amarisoft makes anAll three modes read their settings from one file, amarisoft_params.yml, and render their
configs through one script, amarisoft_render.py (templates in templates/, subscribers from
data/ue_inventory.yml). Change a value there and every mode picks it up — that is what keeps
them in agreement. Two values are meant to be chosen per run: amari_rat (5g / 4g) and
amari_rig (osc / sb1).
The manual path below is the single-step version (one UE count, 5G, osc rig). The full
sweep is a loop with per-step polling, which is what the Ansible mode is for; the complete manual
script is amarisoft_manual.sh in the tutorial directory.
# 0. From the tutorial directory on the console: render this run's configs. They are
# written to a local directory; the second column is where each goes on the node.
cd $(cosmos-tutorial which amarisoft)
python3 amarisoft_render.py ran --rat 5g --rig osc --out /tmp/amari-stage
python3 amarisoft_render.py ue --rat 5g --n 4 --out /tmp/amari-stage
/tmp/amari-stage/cosmos-tut-gnb.cfg /opt/amarisoft/enb/config/cosmos-tut-gnb.cfg
/tmp/amari-stage/cosmos-tut-mme-5g.cfg /opt/amarisoft/mme/config/cosmos-tut-mme-5g.cfg
/tmp/amari-stage/cosmos-tut-ue-5g-n4.cfg /opt/amarisoft/ue/config/cosmos-tut-ue-5g-n4.cfg
# 1. Push them, plus the UE remote-API client.
scp /tmp/amari-stage/cosmos-tut-gnb.cfg root@srv17.osc.cosmos-lab.org:/opt/amarisoft/enb/config/
scp /tmp/amari-stage/cosmos-tut-mme-5g.cfg root@srv17.osc.cosmos-lab.org:/opt/amarisoft/mme/config/
scp /tmp/amari-stage/cosmos-tut-ue-5g-n4.cfg root@sdr1-in3.sb1.cosmos-lab.org:/opt/amarisoft/ue/config/
ssh root@sdr1-in3.sb1.cosmos-lab.org 'mkdir -p /opt/cosmos-tutorials/amarisoft'
scp node/amari_cmd.py root@sdr1-in3.sb1.cosmos-lab.org:/opt/cosmos-tutorials/amarisoft/
# 2. Start the core. systemd-run, NOT setsid: these binaries are interactive and
# exit on stdin EOF, and a setsid'd `tail -f` dies with your ssh session --
# which closes their stdin, so they exit CLEANLY with no error anywhere.
ssh root@srv17.osc.cosmos-lab.org '
systemctl stop cosmos-tut-lteenb cosmos-tut-ltemme 2>/dev/null; systemctl reset-failed cosmos-tut-lteenb cosmos-tut-ltemme 2>/dev/null
systemd-run --unit=cosmos-tut-ltemme -p TimeoutStopSec=20 --setenv=HOME=/root bash -c \
"cd /opt/amarisoft/mme && tail -f /dev/null | ./ltemme config/cosmos-tut-mme-5g.cfg > /tmp/mme-run.log 2>&1"'
# 3. Start the base station and wait for it to reach the core.
ssh root@srv17.osc.cosmos-lab.org '
: > /tmp/gnb0.log
systemd-run --unit=cosmos-tut-lteenb -p TimeoutStopSec=20 --setenv=HOME=/root bash -c \
"cd /opt/amarisoft/enb && tail -f /dev/null | ./lteenb config/cosmos-tut-gnb.cfg > /tmp/enb-run.log 2>&1"
for i in $(seq 1 40); do grep -aq "NG setup response" /tmp/gnb0.log && break; sleep 3; done
pgrep -c -x lteenb-avx2 # NOT -x lteenb: the launcher execs the AVX2 build
grep -a "NG setup response" /tmp/gnb0.log | tail -1'
If the count is 1 but no setup response appears within two minutes, the radio stalled on its
first launch — see Troubleshooting (kill the unit and start it once more).
# 4. Read the cell the base station actually brought up. This line is the ground
# truth -- never copy these values out of the config file.
ssh root@srv17.osc.cosmos-lab.org 'grep -a "Cell " /tmp/gnb0.log | head -1'
# Cell 0x01: nr_arfcn=518670 ul_nr_arfcn=518670 pci=500 mode=TDD rat=nr n_rb_dl=106 n_rb_ul=106 dl_mu=1 ul_mu=1 ssb_mu=1 ssb_arfcn=518670 coreset0_prb=31:48 coreset0_idx=10
# 5. Start the UE bank (4 UEs).
ssh root@sdr1-in3.sb1.cosmos-lab.org '
systemctl stop cosmos-tut-lteue 2>/dev/null; systemctl reset-failed cosmos-tut-lteue 2>/dev/null
: > /tmp/ue-run.log
systemd-run --unit=cosmos-tut-lteue -p TimeoutStopSec=20 --setenv=HOME=/root bash -c \
"cd /opt/amarisoft/ue && tail -f /dev/null | ./lteue config/cosmos-tut-ue-5g-n4.cfg > /tmp/ue-run.log 2>&1"
sleep 40
pgrep -c -x lteue-avx2'
# 6. Power the UEs on. lteue does NOT auto-attach: without this it sits powered
# off with the radio streaming and never searches for the cell.
ssh root@sdr1-in3.sb1.cosmos-lab.org '
cd /opt/cosmos-tutorials/amarisoft
for i in 1 2 3 4; do
python3 ./amari_cmd.py 9002 "{\"message\":\"power_on\",\"ue_id\":$i}" >/dev/null
done'
# 7. Count how many registered. Count MATCHES, not lines: ue_get returns every
# UE on ONE line, so `grep -c` returns 1 no matter how many attached.
ssh root@sdr1-in3.sb1.cosmos-lab.org '
python3 /opt/cosmos-tutorials/amarisoft/amari_cmd.py 9002 "{\"message\":\"ue_get\"}" \
| grep -o "\"emm_state\":\"registered\"" | wc -l'
4
You should see the count equal the number of UEs you started, within a few seconds.
This is the mode you want for the actual experiment: it walks the UE count and records what
attached at each step.
cosmos-tutorial run amarisoft --mode ansible
The launcher resolves the tutorial, chains ANSIBLE_ROLES_PATH to its roles/, and picks up the
shipped inventory/ — no -i and no OMF_NODES needed, for the reasons given under Topology. The
first play builds the host list from amarisoft_params.yml and prints it:
TASK [Report the rig] **********************************************************
ok: [localhost] => {
"msg": "rig=osc rat=5g: RAN srv17.osc.cosmos-lab.org (10.39.3.1), UE sdr1-in3.sb1.cosmos-lab.org (10.38.2.1)"
}
To modify it first: cosmos-tutorial copy amarisoft puts an editable copy in your cwd, which then
shadows the installed one.
Useful variations:
# LTE instead of 5G-SA — selects the cell AND the matching UE subscriber set
cosmos-tutorial run amarisoft --mode ansible -- -e amari_rat=4g
# run the core + base station on the sb1 server instead of the osc one
cosmos-tutorial run amarisoft --mode ansible -- -e amari_rig=sb1
# a shorter sweep
cosmos-tutorial run amarisoft --mode ansible -- -e '{"amari_sweep_counts": "1 4 16"}'
⚠️ Use JSON for multi-value extra-vars. -e "amari_sweep_counts=1 4 16" splits on whitespace and
silently keeps only 1 — the sweep then runs one step and reports a clean pass.
Per step you will see:
TASK [amarisoft_sweep : N=4: report] *******************************************
ok: [sdr1-in3.sb1.cosmos-lab.org] => {
"msg": "N=4: registered=4/4 complete=1 "
}
If the UE bank had to be relaunched — it segfaulted at start, or none of its UEs found the cell —
the step says so (N=4: CELL_SEARCH_RETRY (launch transient, retried once)) before its result.
cosmos-tutorial jupyter amarisoft # copies the system copy to cwd, prints the launch line
System tutorials are read-only, so the launcher copies before opening. Run the cells top to bottom. It performs
the same steps and plots attach rate and attach latency against offered load. It is also
papermill-runnable:
/opt/cosmos-jupyter/venv/bin/papermill \
$(cosmos-tutorial which amarisoft)/amarisoft_tutorial.ipynb /tmp/out.ipynb \
-p counts "1 4" -p rat 5g -p rig osc -p tut_dir "$(cosmos-tutorial which amarisoft)" -k python3
Every mode writes the same JSON result contract into $AMARI_RUN_DIR
(default /tmp/amarisoft-$USER):
cat /tmp/amarisoft-$USER/amari_contract_ansible.json
{
"mode": "ansible",
"rat": "5g",
"rig": "osc",
"enb_cpu": "Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz, 2x12C, 24C/48T",
"enb": "srv17", "ue": "sdr1-in3",
"cell": { "arfcn": 518670, "pci": 500, "n_rb": 106, "duplex": "TDD" },
"steps_sig": "1:1 4:4",
"steps_incomplete": 0,
"all_pass": true,
"ping_rtt_avg_ms": 48.7, "tcp_dl_mbps": 47.8, "tcp_ul_mbps": 4.63,
"udp_dl_capacity_mbps": 58.1, "udp_ul_capacity_mbps": 4.62,
"perf_plausible": true
}
enb_cpu is read from lscpu on the RAN server at run time: results on this rig are a property of
the server's CPU platform, so every contract says which one produced it. complete is the field
that matters most for the sweep. A step is scored only if the UE bank was alive and
the power-on fan-out finished; otherwise the step is recorded as not measured rather than as a
zero. A zero and an unmeasured step look identical in a results table and mean opposite things.
Run all three modes and assert they agree (RAT, RIG and UDP_RATES are passed identically to
all three):
RAT=4g RIG=osc cosmos-tutorial run amarisoft --mode parity -- "1 4"
=== three-mode parity (Amarisoft controlled cell) ===
mode rat rig cell_arfcn cell_pci cell_nrb max_ue_all_attached all_pass steps_sig ping_rtt_avg_ms udp_ul_capacity_mbps udp_dl_capacity_mbps enb_cpu
------------------------------------------------------------------------------------------------------------------------------------------------------------
ansible 4g osc 3350 1 None 4 True 1:1 4:4 35.195 17.6 46.2 Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz, 2x12C, 24C/48T
manual 4g osc 3350 1 None 4 True 1:1 4:4 36.485 16.5 37.3 Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz, 2x12C, 24C/48T
jupyter 4g osc 3350 1 None 4 True 1:1 4:4 31.065 20.0 46.7 Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz, 2x12C, 24C/48T
PARITY: PASS (same experiment in all modes; all checks green)
(Real output, 2026-09-13. cell_nrb is None on LTE because lteenb does not print the
bandwidth on its LTE cell line, and the contract records only what the base station printed.)
Parity asserts that all three modes brought up the same cell on the same server and got the
same attach result at every step. Throughput is gated per mode only — both directions must
have measured something plausible (0.5–500 Mbit/s, ping loss ≤ 5%) — and is printed, not
compared: the modes run minutes apart over the air, and on 5G the numbers can differ by several
times between such runs. See Performance by CPU platform below.
The harness deliberately uses a short sweep: parity is a statement about the three modes, not
about the capacity ceiling, and running the full sweep three times would cost an hour to prove
nothing extra.
Measured 2026-08-28 on the rig above, one UE bank against one cell:
| UEs | 4G registered | 4G attach | 5G registered | 5G attach |
|---|---|---|---|---|
| 1 | 1/1 | 6 s | 1/1 | 6 s |
| 2 | 2/2 | 6 s | 2/2 | 7 s |
| 4 | 4/4 | 6 s | 4/4 | 6 s |
| 8 | 8/8 | 6 s | 8/8 | 6 s |
| 16 | 16/16 | 6 s | 16/16 | 6 s |
| 32 | 32/32 | 6 s | 32/32 | 6 s |
| 48 | 48/48 | 39 s | 48/48 | 6 s |
| 64 | 64/64 | ~446 s | 64/64 | 6 s |
Every provisioned subscriber attaches on both RATs — 64 is the subscriber ceiling of the core
config, not a limit the cell imposed. To probe an actual limit you would have to provision more
subscribers first.
The one clear difference is attach latency at high load: LTE takes 39 s at 48 UEs and several
minutes at 64, while 5G-SA stays at 6 s throughout. Treat the LTE figures as this rig's behaviour
rather than a general result — they are single measurements, and the 64-UE LTE point in particular
is one observation.
The base station can run on either RAN server; the radios, the UE bank and the rendered configs
are identical, so the server's CPU platform is the only thing that changes. One UE, UDP capacity
(the receiver-side rate at the highest offered load with < 5% loss), measured 2026-09-13 with
amarisoft-20260904.ndz + lte5gue-20260905.ndz, Amarisoft 2026-06-12, radios cold-cycled
before each run:
| RAT | CPU platform (RAN server) | Runs | UDP DL, Mbit/s | UDP UL, Mbit/s | Ping RTT |
|---|---|---|---|---|---|
| LTE 10 MHz FDD | Intel Xeon Gold 6126 @ 2.60 GHz, 2 × 12 cores, 48 threads (HT on) | 3 | 37 – 47 | 16.5 – 20 | 31 – 36 ms |
| LTE 10 MHz FDD | Intel Xeon Gold 6226 @ 2.70 GHz, 2 × 12 cores, 24 threads (HT off) | 6 | 37 – 47 | 17 – 20 | 30 – 53 ms |
| 5G-SA n41 40 MHz TDD | Intel Xeon Gold 6126 @ 2.60 GHz, 2 × 12 cores, 48 threads (HT on) | 5 | 14 – 76 (median 57) | 3.9 – 21 (median 4.4) | 34 – 118 ms |
| 5G-SA n41 40 MHz TDD | Intel Xeon Gold 6226 @ 2.70 GHz, 2 × 12 cores, 24 threads (HT off) | 8 | 10 – 66 (median 16) | 0.6 – 17 (median 4.1) | 46 – 290 ms |
LTE: no platform effect. Both servers land in the same range, and LTE at 10 MHz sits well
below either machine's capacity.
5G: the spread within each platform is larger than the difference between them. On both
servers individual runs land either around 14–16 or around 56–76 Mbit/s downlink, minutes apart,
with nothing changed. Both platforms reach the high level (osc 76, sb1 66) and both drop to the
low one (osc 14, sb1 10); the osc runs landed high somewhat more often (3 of 5, against 3 of 8 on
sb1), but the ranges overlap completely, so this is not evidence that one CPU is better for
this cell. The base station is not CPU-bound on either machine (lteenb used 1.4–1.6 cores on
the osc server and 0.8–0.9 on the sb1 server). In the four runs instrumented for it, the slow
direction showed heavy HARQ retransmission — roughly 180–260% — at a UE SNR of about 30 dB, which
points at the link rather than at compute. The root cause is still open (under investigation:
receive-gain saturation on this short cabled path, 256-QAM link adaptation, and TX pacing on the
sb1 server's 25G→10G path to the radio). Until it is closed, treat any single 5G throughput figure
from this rig as one draw from that range, and compare platforms only across several runs.
Run the teardown — it is not optional politeness. One seat per licence tag, testbed-wide: a
base station left running blocks everybody else.
Everything the tutorial starts runs under its own systemd unit — cosmos-tut-lteenb,
cosmos-tut-ltemme, cosmos-tut-lteue — so the teardown stops exactly its own processes and
cannot touch another project's lteenb unit on a shared host. ⛔ Never blanket-pkill an Amarisoft binary on these hosts — they are
shared, and a blanket kill has already destroyed another user's experiment. Kill by PID or by
unit, never by binary name.
cosmos-tutorial run amarisoft --mode teardown # the osc rig (default)
cosmos-tutorial run amarisoft --mode teardown -- -e amari_rig=sb1 # if you ran the sb1 rig
Tear down the rig you ran: stopping the osc server while your sb1 run holds the seats releases
nothing. The parity harness tears down its own rig on every exit path, including Ctrl-C.
The teardown asserts the processes are actually gone rather than assuming the stop worked:
TASK [The seats must actually be released] *************************************
ok: [srv17.osc.cosmos-lab.org] => { "msg": "enb=0 mme=0" }
Then release your reservation as usual. The tutorial leaves its cosmos-tut-* configs and
/opt/cosmos-tutorials/amarisoft/ on the nodes; they are regenerated on every run and harmless.
Re-imaging a host with amarisoft-20260904.ndz (RAN) or lte5gue-20260905.ndz (UE) is safe — the
tutorial brings everything else itself.
| Symptom | Likely cause | Fix |
|---|---|---|
Tag <X> not found |
Usually STALE LICENCE-SERVER STATE, not a busy seat. Seen after ~6 days of server uptime, with some tags refused while others still worked. ⚠️ The error commonly names the wrong tag — it echoes the last successful lookup. | ssh root@mgmt1 systemctl restart amarisoft-license.service. This drops every connected client testbed-wide, and Amarisoft clients do not reconnect — restart them afterwards. |
lteenb/ltemme/lteue segfaults with no banner |
A licence failure of any kind — clients segfault rather than reporting it. Could be a genuinely busy seat, or the stale-server state above. | Check your own host first (pgrep -x lteenb-avx2 / ltemme / lteue-avx2). If clean, restart the licence service. ⛔ Do not blanket-pkill on a shared host — kill by PID. |
A sweep step reports 0/N while its neighbours are 100% |
The UE bank crashed at launch and the harness polled a process that was not running. | Check systemctl status cosmos-tut-lteue and tail /tmp/ue-run.log. If it never printed License server:, see the row above. The playbook now records this as incomplete, not zero. |
pgrep -x lteenb returns 0 while the base station is clearly running |
The launcher execs the AVX2 build. | Always match lteenb-avx2. Acting on the wrong name has caused a duplicate launch and a seat-contention segfault. |
A step reports 0/N and says the UE log shows No cell available (UE RSRP a sentinel such as -1.3e+23, CQI 0) |
The UE bank never heard the cell. Two known causes, which one step cannot tell apart: a launch transient — a fresh lteue sometimes never finds the cell, and a later launch of the same config against the same base station attaches in seconds (the bank is relaunched once automatically and says CELL_SEARCH_RETRY; on 2026-09-13 a launch and its relaunch were deaf and the next step attached 4/4 in 7 s); or the N310 silent-TX stall — either radio keeps streaming with no error anywhere while radiating nothing, and stays that way. Seen so far only on LTE, where it is common (4 of 9 launches in one run). The UE's log shows it did find the cell at startup (SIB found) and lost it by the time the UEs were powered on; the cause is not yet established. |
If the next step attaches, it was the transient — re-run to get a clean result. If the next step is deaf too, cold-cycle both radios from console.sb1: omf tell -a offh -t sdr1-md1,sdr1-s1-lg1, wait 15 s, omf tell -a on -t … (retry if a node answers ERROR), wait a minute, re-run. The deaf UE log is kept on the UE host as /tmp/cosmos-tut-ue-deaf-*.log (the step prints DEAF_LOG). rx_gain 40 on the UE is the tested value (CQI 15 on both RATs, 2026-09-13). |
Base station runs, but never logs S1 setup response / NG setup response |
The first lteenb launch after the radio was last used by another host or another RAT can stall alive — no licence banner, no setup, CPU busy. |
Handled: the tutorial relaunches it once and reports first lteenb launch did not reach S1/NG setup … relaunching once. If the relaunch stalls too, cold-cycle the eNB radio. |
cosmos-tutorial run amarisoft says no hosts matched for every play |
An old copy of the tutorial whose inventory was inventory/amarisoft_hosts.ini: Ansible's directory inventory loader silently skips *.ini files. |
Use the current package (inventory is YAML and the hosts come from amarisoft_params.yml). |
UEs never attach; Authentication reject in the core log |
Wrong SIM algorithm. The 64-subscriber set uses xor with no opc; the two milenage entries need one. |
Use the xor credentials; state sim_algo explicitly — the UE does not infer it. |
| Everything worked yesterday, nothing works today, no one changed anything | systemd-networkd re-applied its own config on DHCP renewal and reverted hand-made routes/MTU — within hours, no reboot, nothing in the journal. |
The tutorial's preflight (amari_preflight.sh, run by every mode) asserts and repairs the radio path each run. A second NIC on the radio subnet (e.g. DATA2a on srv1-lg1) is downed for the boot. |
| Radio opens but throughput is poor / frames are dropped | Frame size exceeds the path MTU. Note (2026-08-29): the "L3 fronthaul is capped at 8000" belief was disproved — it came from path-testing against a USRP-2974, which answers at a false ~8000 ceiling from UHD overhead on the radio side. Measured server-to-server, the osc fabric carries 9000 (see Data-plane MTU). Hosts now come up at MTU 9000. | Verify the path with ping -M do -s 8972 <non-USRP peer> before lowering anything. recv_frame_size=7900,send_frame_size=7900 remains the tested-good setting for this rig; raising it is a separate tuning exercise. |
uhd::io_error: socket closed, or No devices found right after a UE kill |
N310 wedge. Two distinct kinds: the MPM timeout, and an sfp0 ARP-wedge where the radio's 10G address goes ARP-INCOMPLETE while its management address still answers. | ARP-wedge: ssh root@<n310-mgmt-ip> reboot (~40 s; mgmt is its 10.37.x address), then wait ~1 min for MPM. MPM only: systemctl restart usrp-hwd on the radio. See the usrp-n3xx skill for the full ladder. |
lteue runs, radio streams, but no UE ever searches for the cell |
power_on was never sent. lteue does not auto-attach. |
Send {"message":"power_on","ue_id":N} per UE via amari_cmd.py. |
| Attach count stuck at exactly 1 regardless of UE count | grep -c on ue_get, which returns every UE on one line, so it counts lines. |
Count matches: grep -o '"emm_state":"registered"' \| wc -l. |
Large sweeps report a phantom 0/64 |
64 serial power_on calls take ~386 s and can outlive an ssh timeout, so no UE is ever powered on. |
Fan out in batches (the bundle uses 16) and bound every remote call with timeout. |
lteenb.doc, lteue.doc, ltemme.doc; the bundled copies under /opt/amarisoft/<release>/doc/ match the installed version.sim_algo) shared by Amarisoft, OCUDU and Duranta, and the generator that writes each core's config. Access-restricted: you must be logged in.Author(s): Ivan Seskar · Last verified: 2026-09-13 · Tested image/release: amarisoft-20260904.ndz + lte5gue-20260905.ndz, Amarisoft 2026-06-12 · Tags: amarisoft, lte, 5g, sa, ue-simulator, capacity