Skip to content

Commit 45a2508

Browse files
maxisbeymaxisbey
committed
De-flake conformance CI: solo re-verification, spawn-storm reduction, result artifacts
The client conformance suite launches every scenario's client subprocess concurrently; on a 2-vCPU runner the resulting contention intermittently pushes sse-retry's reconnect-timing measurement past its tolerance and fails the job (~4-5% of runs, including pushes to main). - .github/actions/conformance/run-client.sh: wraps the client suite legs. On failure, scenarios listed as unexpected failures are re-run alone on the then-quiet runner: a real failure fails again; a contention artifact passes and the job goes green with a FLAKE_RESCUED marker saved into the results directory. Stale-baseline errors and infra failures are never retried, and an unparseable failure list falls back to the original exit code, so the wrapper cannot green-wash anything but solo-verified passes. - conformance.yml: uv sync gains --compile-bytecode and the editable sources are pre-compiled, so ~40 concurrent interpreters stop racing to byte-compile the same modules during the measurement window; the client command execs the synced venv's interpreter directly instead of paying uv's lockfile re-check in every spawn; both jobs save --output-dir results and upload them when the job fails or a flake was rescued. - everything-server: test_sampling passes related_request_id so the sampling request rides the originating tools/call SSE stream instead of racing the client's standalone GET stream (a dropped request there hangs tools-call-sampling to the 60s client timeout).
1 parent 0b200ef commit 45a2508

3 files changed

Lines changed: 161 additions & 6 deletions

File tree

‎.github/actions/conformance/run-client.sh‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
#!/bin/bash
2+
# Run a client conformance suite, re-verifying unexpected failures solo.
3+
#
4+
# Suite mode launches every scenario's client subprocess concurrently; on a
5+
# 2-vCPU runner that contention can push scenarios with real-time waits (the
6+
# SSE reconnect timing in sse-retry) past their tolerances. So a scenario the
7+
# suite run flags as an unexpected failure is re-run alone on the then-quiet
8+
# runner: a real failure fails again and the job stays red; a contention
9+
# artifact passes and the job goes green, with a FLAKE_RESCUED marker written
10+
# into the --output-dir so the artifact upload preserves the evidence.
11+
# Failures that only reproduce under concurrency are deliberately traded
12+
# away - the suite asserts spec compliance, not behavior under parallel load.
13+
set -uo pipefail
14+
15+
: "${CONFORMANCE_PKG:?set CONFORMANCE_PKG (pinned in .github/workflows/conformance.yml)}"
16+
SOLO_ATTEMPTS="${CONFORMANCE_SOLO_ATTEMPTS:-2}"
17+
18+
# Relative paths in the arguments (the client command, --output-dir) resolve
19+
# from the repo root, same contract as run-server.sh.
20+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
21+
cd "$SCRIPT_DIR/../../.." || exit 1
22+
23+
log="$(mktemp)"
24+
trap 'rm -f "$log"' EXIT
25+
26+
npx --yes "$CONFORMANCE_PKG" client "$@" 2>&1 | tee "$log"
27+
rc=${PIPESTATUS[0]}
28+
if [ "$rc" -eq 0 ]; then
29+
exit 0
30+
fi
31+
32+
plain="$(sed 's/\x1b\[[0-9;]*m//g' "$log")"
33+
34+
# Scenarios listed under "Unexpected failures (not in baseline):". Anything
35+
# else behind the nonzero exit (stale baseline entries, harness or infra
36+
# errors) is not retried. The extraction is coupled to the pinned harness's
37+
# summary wording and print order; if a pin bump changes either, the list
38+
# comes up empty and the original failure passes through - never a false
39+
# green.
40+
mapfile -t scenarios < <(
41+
printf '%s\n' "$plain" |
42+
sed -n '/^Unexpected failures (not in baseline):$/,/^$/p' |
43+
sed -n 's/^ ✗ //p'
44+
)
45+
if [ "${#scenarios[@]}" -eq 0 ]; then
46+
exit "$rc"
47+
fi
48+
for scenario in "${scenarios[@]}"; do
49+
if ! [[ "$scenario" =~ ^[A-Za-z0-9/_-]+$ ]]; then
50+
echo "Extracted unexpected-failure name '${scenario}' does not look like a scenario name; passing the suite failure through." >&2
51+
exit "$rc"
52+
fi
53+
done
54+
55+
# A stale baseline entry is a configuration error a solo rerun cannot excuse.
56+
if printf '%s\n' "$plain" | grep -q '^Stale baseline entries'; then
57+
echo "Suite also reported stale baseline entries; not retrying." >&2
58+
exit "$rc"
59+
fi
60+
61+
# Reuse the suite invocation's arguments for the solo runs, minus the flags
62+
# that only make sense for a suite (--scenario replaces --suite; single runs
63+
# are judged directly, not against the baseline). Solo results are saved next
64+
# to the suite's so the uploaded artifact carries both.
65+
rerun_args=()
66+
output_dir=""
67+
skip_next=0
68+
expect_output_dir=0
69+
for arg in "$@"; do
70+
if [ "$skip_next" -eq 1 ]; then
71+
if [ "$expect_output_dir" -eq 1 ]; then
72+
output_dir="$arg"
73+
fi
74+
skip_next=0
75+
expect_output_dir=0
76+
continue
77+
fi
78+
case "$arg" in
79+
--output-dir)
80+
skip_next=1
81+
expect_output_dir=1
82+
;;
83+
--suite | --expected-failures) skip_next=1 ;;
84+
--output-dir=*) output_dir="${arg#--output-dir=}" ;;
85+
--suite=* | --expected-failures=*) ;;
86+
*) rerun_args+=("$arg") ;;
87+
esac
88+
done
89+
if [ -n "$output_dir" ]; then
90+
rerun_args+=(--output-dir "${output_dir}-solo")
91+
fi
92+
93+
for scenario in "${scenarios[@]}"; do
94+
passed=0
95+
for attempt in $(seq 1 "$SOLO_ATTEMPTS"); do
96+
echo ""
97+
echo "Re-running '${scenario}' solo (attempt ${attempt}/${SOLO_ATTEMPTS})..."
98+
if npx --yes "$CONFORMANCE_PKG" client --scenario "$scenario" "${rerun_args[@]}"; then
99+
passed=1
100+
break
101+
fi
102+
done
103+
if [ "$passed" -ne 1 ]; then
104+
echo "'${scenario}' still fails when run alone: real failure, not suite contention." >&2
105+
exit 1
106+
fi
107+
done
108+
109+
if [ -n "$output_dir" ]; then
110+
mkdir -p "$output_dir"
111+
printf '%s\n' "${scenarios[@]}" > "$output_dir/FLAKE_RESCUED"
112+
fi
113+
echo "All ${#scenarios[@]} unexpected failure(s) passed when re-run solo; the suite failures were parallel-run contention."
114+
exit 0

‎.github/workflows/conformance.yml‎

Lines changed: 42 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -64,17 +64,20 @@ jobs:
6464
./.github/actions/conformance/run-server.sh
6565
--suite active
6666
--expected-failures ./.github/actions/conformance/expected-failures.yml
67+
--output-dir conformance-results/server-active
6768
- name: Run server conformance (draft suite)
6869
run: >-
6970
./.github/actions/conformance/run-server.sh
7071
--suite draft
7172
--expected-failures ./.github/actions/conformance/expected-failures.yml
73+
--output-dir conformance-results/server-draft
7274
- name: Run server conformance (2026-07-28 wire, all suite)
7375
run: >-
7476
./.github/actions/conformance/run-server.sh
7577
--suite all
7678
--spec-version 2026-07-28
7779
--expected-failures ./.github/actions/conformance/expected-failures.2026-07-28.yml
80+
--output-dir conformance-results/server-2026-07-28
7881
- name: Run server conformance (all suite, extension scenarios)
7982
# A bare `--suite all` (no --spec-version) selects every scenario
8083
# shipped with the pinned harness — including the extension-tagged
@@ -91,6 +94,16 @@ jobs:
9194
./.github/actions/conformance/run-server.sh
9295
--suite all
9396
--expected-failures ./.github/actions/conformance/expected-failures.yml
97+
--output-dir conformance-results/server-all
98+
- name: Upload conformance results
99+
# The suite summary only prints counts for warning-level findings; the
100+
# per-check measurements live in the checks.json files saved above.
101+
if: failure()
102+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
103+
with:
104+
name: server-conformance-results
105+
path: conformance-results/
106+
if-no-files-found: ignore
94107

95108
client-conformance:
96109
runs-on: ubuntu-latest
@@ -118,22 +131,46 @@ jobs:
118131
echo "CONFORMANCE_PKG=file:/tmp/conformance.tgz" >> "$GITHUB_ENV"
119132
;;
120133
esac
121-
- run: uv sync --frozen --all-extras --package mcp
134+
# --compile-bytecode: the harness spawns every scenario's client
135+
# concurrently; without pre-compiled site-packages, ~40 fresh
136+
# interpreters race to byte-compile the same modules on a 2-core
137+
# runner, saturating it for the first ~20s of the suite — exactly when
138+
# timing-sensitive scenarios take their measurements.
139+
- run: uv sync --frozen --all-extras --package mcp --compile-bytecode
140+
- name: Pre-compile bytecode (editable sources)
141+
run: uv run --frozen python -m compileall -q src .github/actions/conformance
122142
- name: Run client conformance (all suite)
123143
# The harness runs all scenarios via unbounded Promise.all; with 40
124144
# scenarios on a 2-core runner the slowest one (sse-retry, which has a
125145
# real-time SSE reconnect wait) needs more than the 30s default budget.
146+
# The client command execs the synced venv's interpreter directly:
147+
# `uv run` would re-check the lockfile in every one of the ~40
148+
# concurrent spawns, compounding the startup storm. run-client.sh
149+
# re-verifies unexpected failures solo before failing the job.
126150
run: >-
127-
npx --yes "$CONFORMANCE_PKG" client
128-
--command 'uv run --frozen python .github/actions/conformance/client.py'
151+
./.github/actions/conformance/run-client.sh
152+
--command '.venv/bin/python .github/actions/conformance/client.py'
129153
--suite all
130154
--timeout 60000
131155
--expected-failures ./.github/actions/conformance/expected-failures.yml
156+
--output-dir conformance-results/client-all
132157
- name: Run client conformance (2026-07-28 wire, all suite)
133158
run: >-
134-
npx --yes "$CONFORMANCE_PKG" client
135-
--command 'uv run --frozen python .github/actions/conformance/client.py'
159+
./.github/actions/conformance/run-client.sh
160+
--command '.venv/bin/python .github/actions/conformance/client.py'
136161
--suite all
137162
--timeout 60000
138163
--spec-version 2026-07-28
139164
--expected-failures ./.github/actions/conformance/expected-failures.2026-07-28.yml
165+
--output-dir conformance-results/client-2026-07-28
166+
- name: Upload conformance results
167+
# The suite summary only prints counts for warning-level findings; the
168+
# per-check measurements live in the checks.json files saved above.
169+
# Also upload when run-client.sh rescued a flake (job green, but the
170+
# contention evidence should not be discarded).
171+
if: failure() || hashFiles('conformance-results/**/FLAKE_RESCUED') != ''
172+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
173+
with:
174+
name: client-conformance-results
175+
path: conformance-results/
176+
if-no-files-found: ignore

‎examples/servers/everything-server/mcp_everything_server/server.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -191,10 +191,14 @@ async def test_tool_with_progress(ctx: Context) -> str:
191191
async def test_sampling(prompt: str, ctx: Context) -> str:
192192
"""Tests server-initiated sampling (LLM completion request)"""
193193
try:
194-
# Request sampling from client
194+
# Request sampling from client. related_request_id routes the request onto
195+
# the originating tools/call SSE stream, which exists for the whole handler;
196+
# without it the request targets the standalone GET stream and is dropped if
197+
# the client has not finished opening that stream yet.
195198
result = await ctx.session.create_message( # pyright: ignore[reportDeprecated]
196199
messages=[SamplingMessage(role="user", content=TextContent(type="text", text=prompt))],
197200
max_tokens=100,
201+
related_request_id=ctx.request_id,
198202
)
199203

200204
# Since we're not passing tools param, result.content is single content

0 commit comments

Comments
 (0)