Measure latency, TCP and UDP throughput between two COSMOS nodes over a dedicated data interface — and learn why the numbers come out the way they do. The same experiment is presented three ways, from simplest to most capable:
cosmos-tutorial run network-performance.All three read the same parameters and are parsed by the same library, so they produce the same result — a one-command parity harness proves it (see Same experiment, same result below).
| Difficulty | Beginner |
| Estimated time | ~25 min (most of it imaging) |
| Domain / sandbox | grid (any domain with a shared data plane works — see the topology note) |
| Path measured | dedicated 1 GbE data interface, node-to-node |
| Transmits RF? | No — wired only |
| Last verified | 2026-11-01 on grid (node2-12 server / node2-13 client, baseline.ndz), all three modes |
iperf3 in server and client mode to measure TCP and UDP throughput, and ping -I for latency — bound to the interface under test so results can't silently ride another network.systemd-run --unit) so cleanup stops exactly that unit — the safe pattern for anything you start on testbed infrastructure. console.<domain>.cosmos-lab.org
| (control plane -- drives the nodes, carries NO measurement traffic)
+---------------------------+
| |
node A (server) node B (client)
enp0s25: 192.168.200.1 ==== enp0s25: 192.168.200.2
1 GbE shared data switch
Grid nodes carry dark data NICs besides the control interface. On row-1 grid nodes the shared-L2 data port is enp0s25 (enp3s0 lands on a different segment — the tutorial checks carrier and fails clearly if the port is wrong). Other rows/domains may cable a different port: set np_data_iface in network-performance_params.yml. Domains with named data planes (DATA1/DATA2) can use those interfaces directly — some are 10 GbE, and the ceilings scale accordingly.
| Domains | any domain - the node checks below decide |
| Nodes | 2, named in OMF_NODES |
| On the console | python3, ansible-playbook, ssh |
cosmos-tutorial check network-performance verifies, on each node:
The tutorial is installed on the console. There is nothing to download.
export OMF_NODES=<node>,<node>
cosmos-tutorial check network-performance
cosmos-tutorial run network-performance # ansible
cosmos-tutorial run network-performance --mode manual # manual
cosmos-tutorial jupyter network-performance # jupyter
cosmos-tutorial run network-performance --mode parity # parity
cosmos-tutorial run network-performance --mode teardown # teardown
Change any of these with cosmos-tutorial run network-performance --params my.yml. An unknown key or a wrong type is refused. cosmos-tutorial params network-performance prints the file.
| Key | Default | Meaning |
|---|---|---|
np_data_iface |
auto |
network-performance: canonical parameters — the SINGLE source for all three modes. |
np_server_ip |
192.168.200.1 |
nodes[0] / first node = iperf3 server |
np_client_ip |
192.168.200.2 |
nodes[1] / second node = client (measurements run here) |
np_netmask |
24 |
|
np_port |
5201 |
|
np_iperf_secs |
10 |
|
np_udp_bandwidth |
auto |
UDP offered load. 'auto' = the negotiated link speed, so the |
np_ping_count |
5 |
Reserve two nodes and image them once:
export OMF_NODES="node2-12,node2-13" # first = iperf3 server, second = client
omf load -i baseline.ndz -t $OMF_NODES
omf tell on -t $OMF_NODES # load leaves nodes powered OFF
cosmos-tutorial check network-performance
Substitute your two nodes for node1-1 (server) and node1-2 (client).
# 1. Configure the data interface on both nodes (server .1, client .2)
ssh root@node1-1 "apt-get install -y iperf3; ip link set enp0s25 up; ip addr replace 192.168.200.1/24 dev enp0s25"
ssh root@node1-2 "apt-get install -y iperf3; ip link set enp0s25 up; ip addr replace 192.168.200.2/24 dev enp0s25"
# 2. Start the server SUPERVISED -- systemd-run gives it a handle so cleanup
# stops exactly this unit (never a blanket pkill)
ssh root@node1-1 "systemd-run --unit=np-iperf3 --collect /usr/bin/iperf3 -s -p 5201"
# 3. Measure from the client, BOUND to the data path
ssh root@node1-2 "ping -c 5 -I enp0s25 192.168.200.1" # latency
ssh root@node1-2 "iperf3 -c 192.168.200.1 -t 10" # TCP
ssh root@node1-2 "iperf3 -c 192.168.200.1 -u -b 1G -t 10" # UDP at 1G offered
Or run the same steps as a script, which also builds the result contract:
cosmos-tutorial run network-performance --mode manual
export OMF_NODES="node1-1,node1-2" # first node = server, second = client
cosmos-tutorial run network-performance
A preflight play verifies both nodes are reachable and booted a real image; the play then configures the data interfaces, starts the supervised server, measures, and writes the result contract to $NETWORK_PERFORMANCE_RUN_DIR/np_contract_ansible.json (default /tmp/network-performance-$USER/). Change durations, offered load, or the interface for every mode at once in network-performance_params.yml.
Run cosmos-tutorial jupyter network-performance and open the notebook with the kernel on the console (SSH tunnel or VS Code Remote-SSH), set OMF_NODES before launching, run top to bottom. The last cell plots TCP vs UDP throughput against the line-rate ceiling, plus latency and loss.
cosmos-tutorial run network-performance --mode parity
mode link_mbps ping_rtt_ms tcp_mbps udp_mbps udp_loss_pct node1 node2
ansible 1000 0.139 941.0 956.0 0.0 node2-12 node2-13
manual 1000 0.124 941.0 956.0 0.0 node2-12 node2-13
jupyter 1000 0.117 941.0 956.0 0.0 node2-12 node2-13
PARITY: PASS (same experiment in all modes; all checks green)
The link speed is read at both ends and the slower one counts, because a link runs at the speed of its slower port. On a pair where one port came up at 100 Mbit/s the same run measured 94 Mbit/s TCP and passed, because that is the right answer for that link.
Reading the numbers: on 1 GbE, TCP lands at ~934–941 Mbit/s (line rate minus TCP/IP header overhead), while UDP reports ~949 with 0% loss at 1G offered (smaller per-packet overhead). If UDP loss is substantial, the offered load exceeds what the path can carry — walk -b down to find the true ceiling. The wired path is deterministic, which is why the three modes agree to three significant figures.
cosmos-tutorial run network-performance --mode teardown
| Symptom | Likely cause / fix |
|---|---|
| Preflight: "no /.omf-image-name" | Node booted the PXE stage image — omf tell offh, wait 20 s, omf tell on |
no carrier on the data NIC |
Wrong port for this row/domain — check ip -br link on the node, set np_data_iface |
ping -I fails but plain ping works |
The two nodes' data ports are on different segments (e.g. enp3s0 on grid row 1) — use the shared one |
TCP ~94 Mbit/s and link_mbps is 100 |
One data port negotiated 100 Mbit/s. Check cat /sys/class/net/<iface>/speed on both nodes and use another node |
| TCP ~940 but you expected more | That is 1 GbE line rate. 10 G paths need a 10 G interface — see the variant below |
iperf3: unable to connect |
Server not running — systemctl status np-iperf3 on the server node |
The original ORBIT-era version of this tutorial measured ~9.4 Gbit/s TCP / ~8 Gbit/s UDP across a Calient S320 optical circuit between srv1-lg1 and srv4-lg1 (10 GbE through the optical fabric). The methodology above is identical — only the interface and the ceiling change. For setting up such a path, see the optical network tutorial; run this tutorial's measurement phase over the resulting interfaces.
Package cosmos-experiments-network-performance 0.1.0-15 - on every console it is released to (cosmos-tutorial list shows the installed version). The code is installed on the console in /usr/share/cosmos-experiments/network-performance/. To change anything, cosmos-tutorial copy network-performance puts an editable copy in the current directory, and cosmos-tutorial run network-performance then runs that copy instead of the installed one (cosmos-tutorial which network-performance shows which will run). Repository (GitLab account needed, not needed to run or change it): packages/cosmos-experiments-network-performance/usr/share/cosmos-experiments/network-performance.