enhance the fleettest tool

.github/workflows/fleettest.yml

@@ -0,0 +1,64 @@
+name: Test fleettest harness
+
+# Bitrot check for testsuite/fleettest.py (the developer fleet CI harness).
+# fleettest is meant to be run by developers on a modern Ubuntu box, so this
+# job runs only on ubuntu-latest: it stands up a one-host "fleet" of two
+# targets that both ssh to localhost and runs a real fleettest pass against it.
+# It does not run on the BSD/Solaris/macOS/Cygwin matrix.
+
+on:
+  push:
+    branches: [ master ]
+    paths:
+      - 'testsuite/fleettest.py'
+      - '.github/workflows/fleettest.yml'
+  pull_request:
+    branches: [ master ]
+    paths:
+      - 'testsuite/fleettest.py'
+      - '.github/workflows/fleettest.yml'
+  workflow_dispatch:
+  schedule:
+    - cron: '17 7 * * 1'
+
+jobs:
+  fleettest:
+    runs-on: ubuntu-latest
+    name: fleettest against localhost
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        fetch-depth: 0
+    - name: prep
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y gcc g++ gawk autoconf automake \
+          acl libacl1-dev attr libattr1-dev liblz4-dev libzstd-dev libxxhash-dev \
+          python3-cmarkgfm openssl rsync openssh-server
+    - name: set up ssh to localhost
+      run: |
+        mkdir -p ~/.ssh && chmod 700 ~/.ssh
+        ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_ed25519
+        cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys
+        chmod 600 ~/.ssh/authorized_keys
+        sudo systemctl start ssh || sudo service ssh start
+        # fleettest connects with `ssh -o BatchMode=yes localhost`, which won't
+        # answer a host-key prompt -- so pre-trust localhost in known_hosts.
+        ssh-keyscan -H localhost 127.0.0.1 >> ~/.ssh/known_hosts 2>/dev/null
+        ssh -o BatchMode=yes -o ConnectTimeout=15 localhost 'echo ssh-to-localhost-ok'
+    - name: write localhost fleet config
+      run: |
+        cat > fleettest-ci.json <<'EOF'
+        { "targets": [
+          { "name": "local-a", "ssh_host": "localhost", "workflow": "none.yml",
+            "configure_flags": [], "builddir": "rsync-citest-a", "privilege": "sudo" },
+          { "name": "local-b", "ssh_host": "localhost", "workflow": "none.yml",
+            "configure_flags": [], "builddir": "rsync-citest-b", "privilege": "sudo" }
+        ] }
+        EOF
+    - name: fleettest --list (config sanity)
+      run: python3 testsuite/fleettest.py --fleet fleettest-ci.json --list
+    - name: run fleettest against localhost
+      # Two targets both on localhost exercise the parallel multi-target path
+      # and the per-run dir / port isolation; exit 0 iff every cell is OK.
+      run: python3 testsuite/fleettest.py --fleet fleettest-ci.json --timing

testsuite/README.md

@@ -133,6 +133,9 @@ cp testsuite/fleettest.json.example testsuite/fleettest.json   # then edit
 # (or symlink it, or point elsewhere with --fleet PATH)
 ```
 
+The config is looked up in order: `~/.fleettest.json` first, then
+`testsuite/fleettest.json`, unless overridden with `--fleet PATH`.
+
 Each entry names an ssh host (`null` to run locally), the workflow it mirrors,
 and its configure flags, plus optional per-target settings (`make`, `privilege`,
 `env_prefix`, …). See the comments in `fleettest.json.example`.
@@ -150,10 +153,26 @@ Run it from inside a checkout (it builds the current directory's HEAD; use
 ```sh
 python3 testsuite/fleettest.py                       # whole fleet, both transports
 python3 testsuite/fleettest.py --list                # list configured targets
-python3 testsuite/fleettest.py --targets NAME[,NAME] --clean
+python3 testsuite/fleettest.py --targets NAME[,NAME]
 python3 testsuite/fleettest.py --fleet other.json --transport pipe
+python3 testsuite/fleettest.py --timing              # per-target wall-clock breakdown
 ```
 
+`--timing` adds a per-target breakdown after the report — total wall-clock plus
+the push / build / pipe / tcp / nonroot phases, sorted slowest-first. Targets
+run in parallel, so the whole run is gated by the slowest one; the phase columns
+show whether that target's hold-up is the push, the build, or a test pass.
+
+Each run gets its own randomly-named build dir on every target
+(`<builddir>-<run_id>`), so two or three runs can share the same fleet without
+interfering. The dir is removed when the run ends — on success or failure, and
+best-effort on Ctrl-C/kill; pass `--keep` to retain it for inspection. A hard
+kill (`SIGKILL`), or a signal arriving mid-push, can leave a stray
+`<builddir>-<id>` behind; sweep leftovers with
+`python3 testsuite/fleettest.py --cleanup` (scope it with `--targets`, and only
+run it when no other fleet runs are active, since it removes *all* matching run
+dirs on the selected targets).
+
 Each target must be provisioned with the build toolchain its workflow installs
 (autoconf, automake, a C compiler, perl, a python3 markdown module such as
 cmarkgfm or commonmark unless the flags pass `--disable-md2man`, and the dev