Real messages, real causes. Every entry below has actually happened; several
cost hours before the cause was found, which is why the wording is preserved
verbatim — searching for the text you are looking at should land here.
'x' cannot run on <sandbox> right now'needs-impossible' cannot run on grid right now.
63 powered-on node(s) were considered.
- worker: no candidate provides gpu (model=)
needed 3, found 0
each requirement, counted independently:
gpu: satisfied by 0 of 63 candidate(s) <-- nothing provides this
Working as designed. The arrow marks the requirement at fault — with four
requirements and one impossible, this points straight at it. Either ask for
less, or move sandbox.
pinned nodes belong to another subdomain- tx: pinned nodes belong to another subdomain
needed 1, found 0
node18-2.grid.cosmos-lab.org
-> in grid, not sb4
This experiment is pinned to specific hardware and does not run
anywhere else. Run it from console.grid.cosmos-lab.org.
You are on the wrong console. The message names the right one. Waiting will not
help — pinned means pinned.
does not match '^[a-z][a-z0-9-]*$'The top-level name takes hyphens; resource and parameter names take
underscores. needs_impossible is invalid as a name, valid as a resource.
No description at <name>, and no such file under: …The name is relative to ~/cosmos-experiments then
/usr/share/cosmos-experiments. Both are listed in the error. Check the
subdirectory — it is sdr/ofdm-ber-sb5.description.yml, not
ofdm-ber-sb5.description.yml.
N node(s) excluded: held by another run on this subdomain- rx: no candidate provides sdr.usrp (model=b210),
1 node(s) excluded: held by another run on this subdomain
The sandbox is busy, not lacking the hardware. Several experiments can run
on one subdomain at once, and each leases what it resolved onto so two runs
cannot land on the same radio. Your run claimed nothing — leases are
all-or-nothing, so a blocked run never sits on hardware someone else could use.
Wait for the other run, move to another subdomain, or check
cosmos-run --status if you think it has already finished. A lease whose run no
longer exists is reclaimed automatically, including when a run ended because a
reservation expired — there is nothing to clean up by hand.
Shared instruments — the XY table, the attenuator matrix — are leased whole,
not per port, because two runs on different ports still share the instrument's
switch state.
--status says diedThe run claimed to be running and its process is gone, so nothing ever recorded
an outcome. The stored state was stale and is reported as such rather than
repeated. Its log is still there:
cosmos-run --status <run-id> # detail plus the tail of the log
The usual causes are the console rebooting and the supervisor being killed
without a chance to record why.
--status says killedNormally your reservation expired and the console stopped your processes.
This is the expected end for a long run, not a crash — the experiment's teardown
was given a chance to run first, so shared hardware was restored.
Results are not lost: a sweep writes each point as it is taken, so
everything measured before the stop is in the results file and is valid CSV.
--status says running?The run was started on a different console. Process IDs are per-machine, so
that console's claim cannot be verified from this one — the tool says so rather
than guessing. Log into the console named in the WHERE column to get a
definite answer.
--status reads ~/cosmos-artifacts, which is shared across every console, so a
run started anywhere should appear anywhere. If it does not, it was started
without --detach — a foreground run records nothing until it finishes.
WARNING - using ~/cosmos-experiments/… which DIFFERS from the packaged …cosmos-run: WARNING - using /home/seskar/cosmos-experiments/sdr/ofdm-ber-sb5.description.yml
which DIFFERS from the packaged /usr/share/cosmos-experiments/sdr/ofdm-ber-sb5.description.yml
Delete your copy to use the packaged one.
Take this seriously. Your home copy wins, and it goes on winning after the
package is updated. A stale copy kept a transmit gain of 70 after the package
moved to 89, and the resulting mismatch was diagnosed for hours as failing
hardware. If you did not mean to keep a fork, delete your copy.
Nothing failed, nothing warned, and the old behaviour persists. Almost always
the artifact running is not the artifact you built. Check in this order:
cosmos-run --version
It prints the version and the path in use, which is the answer to "which
copy am I actually running". The collection can be installed twice — the package
in /usr/share/ansible/collections and a per-user copy in
~/.ansible/collections that comes first on the search path.
| What looks fine but is not | How to tell |
|---|---|
apt reported "already the newest version" |
The package version did not change, so nothing was installed — on every console, with no error. Compare cosmos-run --version against what you expect. |
| A home copy of the description | See the WARNING - using … which DIFFERS entry above. It is the same disease. |
| A per-user collection shadowing the package | cosmos-run --version names it under shadowing:. Delete it unless you are deliberately testing a build. |
The reason this class of failure costs so much time is that everything reports
success. Prove the artifact is current before you start debugging the
behaviour.
Two forms exist and they behave differently:
cosmos-run x.yml -e amplitude=0.06 # sets the variable
cosmos-run x.yml -e '{"cosmos_parameters": {"amplitude": 0.06}}' # goes through the validator
Both work. The second is checked against choices, min and max; the first
is not. Prefer the second when the value matters.
UHD is not available on <node>UHD is not available on node3-20.grid.cosmos-lab.org. This experiment needs
an SDR image (for example ubuntu2404-uhd4.9-gr3.10.ndz or a baseline-sdr
image). The node is running something else - re-image it, or let node_ready
do so with an sdr.usrp requirement.
Someone re-imaged the node, or it never had the right image. Check whether
they are still using it before taking it back — judge on recent omf load
activity, not on who, since sessions stay attached for weeks.
Connection refused right after imagingThe node is almost certainly still in the PXE stage image. A successful
omf load does not prove the node booted what you loaded.
ssh root@node1-2 cat /.omf-image-name # what it actually booted
omf tell -a offh -t node1-2 && sleep 25 && omf tell -a on -t node1-2
No UHD Devices Found on a node that definitely has a radioOn sb6, each X310 is on a private point-to-point 10 GbE run whose port comes
up with no address, so UHD cannot see it:
ip addr add 192.168.40.1/24 dev enp1s0 && ip link set enp1s0 up mtu 9000
uhd_find_devices --args="addr=192.168.40.2"
On the grid, the X310s are rfdevice resources in MIMO racks, not attached
to any node — omf load does not apply to them. Probe the rack addresses
(10.10.23.x, 10.10.24.x) and pin the one you want.
Both of these kill it at construction, so the symptom surfaces several
tasks later as "the transmitter is not running", pointing at nothing.
internal is not a valid time sourceRuntimeError: AssertionError: assertion failed:
internal is not a valid time source.
possible values are: [none, external, _external_, mimo].
A USRP2/N210 rejects internal, though GNU Radio Companion offers it.
Board 0 may not be getting a PPS signalset_time_unknown_pps() blocks waiting for a pulse that a radio with no
external reference will never produce. GRC emits it whenever the block's sync
option is "Unknown PPS". Two radios measuring a BER need no common time base —
use "don't sync".
This is its own discipline; the short version, in the order to check.
Transmit overdrive. Compression preserves power while destroying the
constellation, so the received level and the computed effective SNR both look
perfect and only the BER is wrong. Halve the drive.
Measured on sb5, QPSK: amplitude 0.06 gives 3.8e-6, amplitude 0.15 gives
7.5e-2 — and both report ~24 dB effective SNR.
Ask whether you reached a floor or the end of the axis. 64-QAM needs about
25.6 dB for 1e-5; a sweep stopping at 20 dB shows a "floor" that is the end of
the sweep.
Compare your asked SNR against the dB above floor the run prints. Beyond that
the link has no more SNR to give, every point measures the same thing, and the
scatter is the noise of a saturated axis. More samples never fix this.
--check fails on something that has not happened yetSteps that cannot be simulated are skipped, and steps that depend on their
output are skipped with them. If you see a genuine failure under --check,
that is a bug worth reporting — a dry run should report, not fail.
cosmos-run x.yml -v # more; repeat for more still
cosmos-run x.yml --version # which copy is actually running
cosmos-run x.yml -- --step # anything after -- goes to ansible-playbook
--version is the first thing to check when a fix does not seem to have taken
effect: it names the copy in use and any it is shadowing.
Last verified: 2026-08-04 on console.sb5.cosmos-lab.org.