Skip to content

Exit device-module-test early if we detect other running netlab instances#3606

Draft
snuffy22 wants to merge 4 commits into
ipspace:devfrom
snuffy22:device-module-test-early-exit
Draft

Exit device-module-test early if we detect other running netlab instances#3606
snuffy22 wants to merge 4 commits into
ipspace:devfrom
snuffy22:device-module-test-early-exit

Conversation

@snuffy22

@snuffy22 snuffy22 commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

This provides a early escape when trying to run tests and you have another 'default' netlab lab already running.

Mentioned in #3604

Conditional that it only cares if user is trying to run the up case where it would clash with an existing lab.

(py3-snuff) netlab@netlab:~/snuffy-netlab/tests/integration$ ./device-module-test -p clab -d frr  --steps create bgp
Pre-test cleanup:
17:09:48 bgp/01-ebgp-session.yml: create(ok) up(skip) config(skip) validate(skip) cleanup(ok)
17:09:49 bgp/02-ibgp-ebgp-session.yml: create(ok) up(skip) config(skip) validate(skip) cleanup(ok)
(py3-snuff) netlab@netlab:~/snuffy-netlab/tests/integration$ ./device-module-test -p clab -d frr  bgp
Running netlab lab 'default' instance detected:
Active lab instance(s)

id           directory                                    status     providers
------------ -------------------------------------------- ---------- --------------------
default      ...me/netlab/snuffy-netlab/tests/integration started    clab

[FATAL]   Refusing to launch integration test while another 'default' netlab lab is running

@snuffy22

Copy link
Copy Markdown
Contributor Author

@ipspace maybe i'm solving this in the wrong place..
I think fixing in the netlab up cli is probably better.. thoughts?

@snuffy22

snuffy22 commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

mm netlab up seems to have that protection.. I should have tested before I spoke

But one thing I did notice, the running of pre-test cleanup breaks the output of the currently running lab that is still up.

This is without my patch..

My running default lab, before trying device-module-test

(py3-snuff) netlab@netlab:~/snuffy-netlab/tests/integration$ netlab status --all --format json
Active lab instance(s)

┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┓
┃ id      ┃ directory                                                ┃ status  ┃ providers ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━┩
│ default │ /home/netlab/snuffy-netlab/tests/integration/ospf/ospfv2 │ started │ clab      │
└─────────┴──────────────────────────────────────────────────────────┴─────────┴───────────┘

After doing device-module-test in normal netlab status --all

py3-snuff) netlab@netlab:~/snuffy-netlab/tests/integration$ ./device-module-test -p clab -d frr bgp
[sudo] password for netlab:
Pre-test cleanup:
19:53:33 bgp/01-ebgp-session.yml: create(ok) up(FAIL) cleanup(ok)
19:53:34 bgp/02-ibgp-ebgp-session.yml: create(ok) up(FAIL) cleanup(ok)
19:53:36 bgp/03-ibgp-rr.yml: create(ok) up(FAIL) cleanup(ok)
19:53:37 bgp/04-originate.yml: ^C

[FATAL]   netlab down --cleanup failed, check the log files in /tmp/netlab_log

(py3-snuff) netlab@netlab:~/snuffy-netlab/tests/integration$ netlab status --all 
Active lab instance(s)

┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ id      ┃ directory                                                ┃ status                 ┃ providers ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ default │ /home/netlab/snuffy-netlab/tests/integration/ospf/ospfv2 │ stopping clab provider │ clab      │
└─────────┴──────────────────────────────────────────────────────────┴────────────────────────┴───────────┘

After with json output

(py3-snuff) netlab@netlab:~/snuffy-netlab/tests/integration$ netlab status --all --format json
{
  "default": {
    "dir": "/home/netlab/snuffy-netlab/tests/integration/ospf/ospfv2",
    "providers": [
      "clab"
    ],
    "log_line": "OK: sudo -E containerlab destroy --cleanup -t clab.yml",
    "log": [
      "2026-07-10T18:09:46.788605-04:00: OK: containerlab version",
      "2026-07-10T18:09:46.803499-04:00: OK: bash -c [[ `containerlab version|awk '/version/ {print $2}'` > '0.74.99' ]] && echo OK",
      "2026-07-10T18:09:46.842268-04:00: OK: docker image ls --format json quay.io/frrouting/frr:10.6.1",
      "2026-07-10T18:09:46.843868-04:00: restarting lab",
      "2026-07-10T18:09:46.846365-04:00: starting provider clab",
      "2026-07-10T18:09:48.707586-04:00: OK: sudo -E containerlab deploy --reconfigure -t clab.yml",
      "2026-07-10T18:09:48.709698-04:00: clab workload started",
      "2026-07-10T18:09:48.711610-04:00: deploying initial configuration",
      "2026-07-10T18:09:48.847956-04:00: deploying configuration: complete configuration",
      "2026-07-10T18:09:52.800956-04:00: configuration deployment complete",
      "2026-07-10T18:09:52.832012-04:00: OK: netlab initial --no-message --deploy",
      "2026-07-10T18:09:52.834547-04:00: initial configuration complete",
      "2026-07-10T18:09:52.836745-04:00: started",
      "2026-07-10T19:53:34.798429-04:00: lab shutdown requested in conflicting directory",
      "2026-07-10T19:53:34.813251-04:00: OK: containerlab version",
      "2026-07-10T19:53:34.830172-04:00: OK: bash -c [[ `containerlab version|awk '/version/ {print $2}'` > '0.74.99' ]] && echo OK",
      "2026-07-10T19:53:34.833622-04:00: stopping clab provider",
      "2026-07-10T19:53:34.880416-04:00: OK: sudo -E containerlab destroy --cleanup -t clab.yml",
      "2026-07-10T19:53:36.020868-04:00: lab shutdown requested in conflicting directory",
      "2026-07-10T19:53:36.035809-04:00: OK: containerlab version",
      "2026-07-10T19:53:36.052394-04:00: OK: bash -c [[ `containerlab version|awk '/version/ {print $2}'` > '0.74.99' ]] && echo OK",
      "2026-07-10T19:53:36.056043-04:00: stopping clab provider",
      "2026-07-10T19:53:36.104211-04:00: OK: sudo -E containerlab destroy --cleanup -t clab.yml",
      "2026-07-10T19:53:37.294479-04:00: lab shutdown requested in conflicting directory",
      "2026-07-10T19:53:37.310224-04:00: OK: containerlab version",
      "2026-07-10T19:53:37.328416-04:00: OK: bash -c [[ `containerlab version|awk '/version/ {print $2}'` > '0.74.99' ]] && echo OK",
      "2026-07-10T19:53:37.333138-04:00: stopping clab provider",
      "2026-07-10T19:53:37.382554-04:00: OK: sudo -E containerlab destroy --cleanup -t clab.yml"
    ],
    "timestamp": "2026-07-10 23:53:37.382554+00:00",
    "status": "stopping clab provider",
    "name": "ospfv2"
  }
}

@ipspace ipspace left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two major changes:

  • Don't run a subprocess. Use the utils.status.read_status function
  • Don't assume you have to check the default instance. I'm using the multilab plugin, so the integration tests don't interfere with other stuff I'm doing. That instance needs to be checked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants