Measure latency, TCP and UDP throughput between two COSMOS nodes over a dedicated data interface — and learn why the numbers come out the way they do. The same experiment is presented three ways, from simplest to most capable:
ansible-playbook network-performance/network-performance.yml.All three read the same parameters and are parsed by the same library, so they produce the same result — a one-command parity harness proves it (see Same experiment, same result below).
| Difficulty | Beginner |
| Estimated time | ~25 min (most of it imaging) |
| Domain / sandbox | grid (any domain with a shared data plane works — see the topology note) |
| Path measured | dedicated 1 GbE data interface, node-to-node |
| Transmits RF? | No — wired only |
| Last verified | 2026-08-28 on grid (node1-1 server / node1-2 client, baseline.ndz) |
iperf3 in server and client mode to measure TCP and UDP throughput, and ping -I for latency — bound to the interface under test so results can't silently ride another network.systemd-run --unit) so cleanup stops exactly that unit — the safe pattern for anything you start on testbed infrastructure. console.<domain>.cosmos-lab.org
| (control plane -- drives the nodes, carries NO measurement traffic)
+---------------------------+
| |
node A (server) node B (client)
enp0s25: 192.168.200.1 ==== enp0s25: 192.168.200.2
1 GbE shared data switch
Grid nodes carry dark data NICs besides the control interface. On row-1 grid nodes the shared-L2 data port is enp0s25 (enp3s0 lands on a different segment — the tutorial checks carrier and fails clearly if the port is wrong). Other rows/domains may cable a different port: set np_data_iface in network-performance_params.yml. Domains with named data planes (DATA1/DATA2) can use those interfaces directly — some are 10 GbE, and the ceilings scale accordingly.
ssh <username>@console.<domain>.cosmos-lab.org
cd ~ && wget -q https://www.cosmos-lab.org/files/cosmos-tutorials-wifi.zip
unzip -o cosmos-tutorials-wifi.zip # extracts into ~/tutorials/ (bundle contains all tutorials)
cd tutorials
File (in network-performance/) |
What it is |
|---|---|
network-performance_params.yml |
single source of truth — iface, addresses, durations, UDP offered load |
network-performance_manual.sh |
Manual mode, scripted |
network-performance.yml |
Ansible mode (preflight first) |
network-performance_tutorial.ipynb |
Jupyter mode |
network-performance_lib.py |
shared parser + result contract + parity comparison |
network-performance_parity.sh |
runs all three modes and checks they agree |
network-performance_teardown.yml |
clean teardown (stops the supervised unit) |
Reserve two nodes and image them once:
export OMF_NODES="node1-1,node1-2" # first = iperf3 server, second = client
omf load -i baseline.ndz -t node1-1,node1-2
omf tell on -t node1-1,node1-2 # load leaves nodes powered OFF!
Substitute your two nodes for node1-1 (server) and node1-2 (client).
# 1. Configure the data interface on both nodes (server .1, client .2)
ssh root@node1-1 "apt-get install -y iperf3; ip link set enp0s25 up; ip addr replace 192.168.200.1/24 dev enp0s25"
ssh root@node1-2 "apt-get install -y iperf3; ip link set enp0s25 up; ip addr replace 192.168.200.2/24 dev enp0s25"
# 2. Start the server SUPERVISED -- systemd-run gives it a handle so cleanup
# stops exactly this unit (never a blanket pkill)
ssh root@node1-1 "systemd-run --unit=np-iperf3 --collect /usr/bin/iperf3 -s -p 5201"
# 3. Measure from the client, BOUND to the data path
ssh root@node1-2 "ping -c 5 -I enp0s25 192.168.200.1" # latency
ssh root@node1-2 "iperf3 -c 192.168.200.1 -t 10" # TCP
ssh root@node1-2 "iperf3 -c 192.168.200.1 -u -b 1G -t 10" # UDP at 1G offered
Or run the same steps as a script, which also builds the result contract:
bash network-performance/network-performance_manual.sh node1-1 node1-2
export OMF_NODES="node1-1,node1-2" # first node = server, second = client
ansible-playbook network-performance/network-performance.yml
A preflight play verifies both nodes are reachable and booted a real image; the play then configures the data interfaces, starts the supervised server, measures, and writes the result contract to $NETWORK_PERFORMANCE_RUN_DIR/np_contract_ansible.json (default /tmp/network-performance-$USER/). Change durations, offered load, or the interface for every mode at once in network-performance_params.yml.
Open network-performance/network-performance_tutorial.ipynb with the kernel on the console (SSH tunnel or VS Code Remote-SSH — see the bundle README), set OMF_NODES before launching, run top to bottom. The last cell plots TCP vs UDP throughput against the line-rate ceiling, plus latency and loss.
bash network-performance/network-performance_parity.sh node1-1 node1-2
=== network-performance parity ===
mode ping_rtt_ms tcp_mbps udp_mbps udp_loss_pct node1 node2
-------------------------------------------------------------------
ansible 0.268 934.0 949.0 0.0 node1-1 node1-2
manual 0.273 934.0 949.0 0.0 node1-1 node1-2
jupyter 0.259 934.0 949.0 0.0 node1-1 node1-2
PARITY: PASS (same experiment in all modes; all checks green)
Reading the numbers: on 1 GbE, TCP lands at ~934–941 Mbit/s (line rate minus TCP/IP header overhead), while UDP reports ~949 with 0% loss at 1G offered (smaller per-packet overhead). If UDP loss is substantial, the offered load exceeds what the path can carry — walk -b down to find the true ceiling. The wired path is deterministic, which is why the three modes agree to three significant figures.
ansible-playbook network-performance/network-performance_teardown.yml # stops the np-iperf3 unit, unconfigures ifaces
omf tell offh -t node1-1,node1-2
| Symptom | Likely cause / fix |
|---|---|
| Preflight: "no /.omf-image-name" | Node booted the PXE stage image — omf tell offh, wait 20 s, omf tell on |
no carrier on the data NIC |
Wrong port for this row/domain — check ip -br link on the node, set np_data_iface |
ping -I fails but plain ping works |
The two nodes' data ports are on different segments (e.g. enp3s0 on grid row 1) — use the shared one |
| TCP ~940 but you expected more | That is 1 GbE line rate. 10 G paths need a 10 G interface — see the variant below |
iperf3: unable to connect |
Server not running — systemctl status np-iperf3 on the server node |
The original ORBIT-era version of this tutorial measured ~9.4 Gbit/s TCP / ~8 Gbit/s UDP across a Calient S320 optical circuit between srv1-lg1 and srv4-lg1 (10 GbE through the optical fabric). The methodology above is identical — only the interface and the ceiling change. For setting up such a path, see the optical network tutorial; run this tutorial's measurement phase over the resulting interfaces.
Orchestration: Ansible + Jupyter (per-console kernel); imaging: omf. Bundle: cosmos-tutorials-wifi.zip. Last verified: 2026-08-28 on grid (node1-1/node1-2, baseline.ndz), three-mode parity PASS (spread 1.00). Tags: networking, iperf3, ansible, jupyter.