Skip to content

fix(playwright): pin merge-multiple=false on results download#30470

Merged
chirag-madlani merged 2 commits into
mainfrom
fix/playwright-summary-single-shard-download
Jul 24, 2026
Merged

fix(playwright): pin merge-multiple=false on results download#30470
chirag-madlani merged 2 commits into
mainfrom
fix/playwright-summary-single-shard-download

Conversation

@chirag-madlani

@chirag-madlani chirag-madlani commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

Follow-up to #30468. Single-shard PR runs (spec-only mode, one chromium lane) fail `playwright-summary` with:

```
Shard chromium-01 did not upload a usable Playwright results artifact.
Shard chromium-01 did not upload its execution status.
```

Seen on PR #30437 run 30088248354.

Root cause

Verified against the run's artifacts API — `playwright-results-json-chromium-01` was uploaded correctly, archive contents:

```
results.json (441110 bytes)
ci-status.json (470 bytes)
```

The download step reports `success`, but `render_playwright_summary.cjs` expects the on-disk layout:

```
results/playwright-results-json-/results.json
```

That layout is what `pattern: playwright-results-json-*` produces when multiple artifacts match. When only one artifact matches (single-shard runs), some `download-artifact@v7` configurations flatten the contents directly into `path:` (`results/results.json`) — breaking the `readdirSync`-based shard discovery in the renderer.

Multi-shard runs never hit this because there's always >1 artifact matched — which is why we only see this on impact-selected / spec-only PRs like #30437.

Change

One-line addition to the `Download all results JSON` step:

```yaml
merge-multiple: false
```

Locks the per-artifact subdirectory layout regardless of match count, so single-shard runs render exactly the same way multi-shard runs do. Also added an inline comment explaining the failure mode + linking the reference run.

Diff scope

```
.github/workflows/playwright-postgresql-e2e.yml | 11 +++++++++++
1 file changed, 11 insertions(+)
```

Pure workflow config. No script changes.

Not in scope (deferred)

A more resilient fix would harden `render_playwright_summary.cjs` to handle either layout (loose files at `results/` root vs. per-shard subdirs). That's Option B from the earlier analysis — can be a follow-up if this pin turns out not to help on single-shard runs.

Test plan

  • This PR's own run: single-shard scenario should pass. If it doesn't, we know the pin isn't enough and need Option B.
  • Multi-shard runs (merge queue, full-suite dispatch): unaffected — behaviour was already correct there.

🤖 Generated with Claude Code

Greptile Summary

Adds an explicit merge-multiple: false setting to preserve per-artifact Playwright result directories and logs the downloaded layout for diagnosing single-shard summary failures.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failures eligible for this follow-up review remain.

No blocking failure remains.

Important Files Changed

Filename Overview
.github/workflows/playwright-postgresql-e2e.yml Pins the Playwright artifact download layout and adds temporary diagnostics for downloaded files and expected shard IDs.

Reviews (2): Last reviewed commit: "add diagnostic ls -R for downloaded resu..." | Re-trigger Greptile

Single-shard PR runs (spec-only mode, one chromium lane) have been
failing playwright-summary with:

  Shard chromium-01 did not upload a usable Playwright results artifact.
  Shard chromium-01 did not upload its execution status.

The chromium-01 shard actually uploaded playwright-results-json-chromium-01
successfully (verified against the run's artifacts API — archive contains
results.json + ci-status.json at the root). The download step then reports
`success` too. But render_playwright_summary.cjs expects the on-disk
layout to be:

  results/playwright-results-json-<shardId>/results.json

That layout is what `pattern: playwright-results-json-*` produces when
multiple artifacts match. When only one artifact matches, some
download-artifact@v7 configurations flatten the contents directly into
`path:` (results/results.json), which breaks the readdirSync-based
shard discovery and makes the renderer report the shard as absent.

Multi-shard runs never hit this because there's always >1 artifact
matched — that's why we only see the failure on impact-selected /
spec-only PRs. Pinning merge-multiple=false explicitly locks the
per-artifact subdirectory layout regardless of match count, so
single-shard runs render the same way multi-shard runs do.

Seen on https://github.com/open-metadata/OpenMetadata/actions/runs/30088248354/job/89474166397?pr=30437

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 24, 2026 13:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions

Copy link
Copy Markdown
Contributor

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

  • No GitHub issue is linked. Link an issue in the Development section of the PR (or add Fixes #12345 to the description). For a same-org cross-repo issue, add Fixes open-metadata/<repo>#123 to the description.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

@github-actions github-actions Bot added safe to test Add this label to run secure Github workflows on PRs UI UI specific issues labels Jul 24, 2026
The explicit merge-multiple: false pin added in the previous commit
matches what should be the documented default for pattern-based
downloads on actions/download-artifact@v7 — so if single-shard runs
STILL report the same "Shard chromium-01 did not upload a usable
Playwright results artifact" error, the flatten hypothesis was wrong
and the real bug is elsewhere.

Add a diagnostic step that runs unconditionally on non-skipped
playwright-ci-postgresql outcomes and dumps:
  - `ls -laR results/` — the actual on-disk layout the renderer sees
  - the shardId values from plan-playwright.matrix (so we can diff
    "expected" vs "present")

If the pin works, this step just adds a few log lines. If it doesn't,
the log tells us precisely whether files landed at
  results/playwright-results-json-<shardId>/results.json  (expected)
or somewhere else (e.g., results/results.json flat) so we can pick
the right follow-up without more speculation. Comment notes it's
removable once single-shard runs are consistently green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 24, 2026 13:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@chirag-madlani

Copy link
Copy Markdown
Collaborator Author

Pushed 9c79d87391 — added a diagnostic ls -laR results/ step right after the download.

Reason: after re-reading the actions/download-artifact release notes, the merge-multiple: false pin in the previous commit matches what should already be the default for pattern-based downloads. If single-shard runs still fail the same way after this PR merges, my earlier flatten hypothesis was wrong. Rather than speculate more, the diagnostic step will print the actual on-disk layout the renderer sees so we can pick the right follow-up with data instead of guesses.

Comment on the step notes it's removable once single-shard runs are consistently green.

Also — to answer @chirag Madlani's question about mixing upload-artifact@v6 with download-artifact@v7: no compatibility issue there. Both are on the same post-v4 storage backend (v4 introduced the immutable-artifacts model); download@v7 handles v6-uploaded zips identically. The v6→v7 upload change was purely additive (an archive: false opt-in for skipping the zip on single-file uploads) — v6-uploaded zips have identical structure to what v7 would produce.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Playwright Results — workflow succeeded

Validated commit 9c79d8739183f991d43e1e4f0ed35c603c6629ab in Playwright run 30097627375, attempt 1.

✅ 536 passed · ❌ 0 failed · 🟡 1 flaky · ⏭️ 5 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 49m 54s

⏱️ Max setup 2m 56s · max shard execution 18m 22s · max shard-job elapsed before upload 22m 3s · reporting 4s

🌐 202.70 requests/attempt · 2.87 app boots/UI scenario · 17.70% common-shard skew

Optimization targets still in progress:

  • Common shard skew was 17.7% (convergence target: at most 15%).
  • Browser traffic was 202.7 requests per attempt (convergence target: fewer than 200).
  • Application boot ratio was 2.87 per UI scenario (1604 boots / 559 scenarios; convergence target: at most 1).
Shard Passed Failed Flaky Skipped Lifecycle failed Lifecycle flaky
✅ Shard chromium-01 143 0 0 3 0 0
🟡 Shard chromium-02 132 0 1 0 0 0
✅ Shard chromium-03 123 0 0 0 0 0
✅ Shard data-asset-rules-01 61 0 0 0 0 0
✅ Shard domain-isolation-01 14 0 0 0 0 0
✅ Shard global-state-01 23 0 0 0 0 0
✅ Shard ingestion-01 1 0 0 0 0 0
✅ Shard reindex-01 2 0 0 0 0 0
✅ Shard search-01 10 0 0 0 0 0
✅ Shard search-rbac-01 27 0 0 2 0 0
🟡 1 flaky test(s) (passed on retry)
  • Pages/Entity.spec.tsDomain Propagation (shard chromium-02, 1 retry)

📦 Download artifacts

How to debug locally
# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

@chirag-madlani
chirag-madlani merged commit 8a5c540 into main Jul 24, 2026
74 of 76 checks passed
@chirag-madlani
chirag-madlani deleted the fix/playwright-summary-single-shard-download branch July 24, 2026 14:29
@gitar-bot

gitar-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown
Code Review ✅ Approved

Pins merge-multiple=false on the Playwright results download step to preserve per-shard subdirectory structures during single-shard runs. No issues found.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

safe to test Add this label to run secure Github workflows on PRs UI UI specific issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants