From 73e26aa58b2ec2b3680dec5e9c67762d17a3f3db Mon Sep 17 00:00:00 2001 From: "Ariznawl@163.com" Date: Tue, 28 Jul 2026 19:39:34 +0800 Subject: [PATCH 01/16] docs: design tenant-parallel BVT orchestration --- .../2026-07-28-tenant-parallel-bvt-design.md | 287 ++++++++++++++++++ 1 file changed, 287 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md diff --git a/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md b/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md new file mode 100644 index 0000000..353abf3 --- /dev/null +++ b/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md @@ -0,0 +1,287 @@ +# Tenant-Parallel BVT Design + +## Status + +Approved direction: run one MatrixOne deployment on one GitHub Actions runner, then run multiple `mo-tester` processes against different MatrixOne tenants on that deployment. + +This design targets the active PR coverage workflows: + +- `.github/workflows/e2e-compose-parallel.yaml` +- `.github/workflows/e2e-standalone-parallel.yaml` + +The first rollout keeps the existing complementary BVT group assignment across the two workflows. Each workflow parallelizes only its assigned group inside its own runner. Checkin Regression adoption is a later rollout after the PR workflow is stable. + +The CI repository baseline for this design is commit: + +`940ee29eea1ee59eed5d3bec69cae7586b9a1162` + +## Goals + +1. Reduce BVT wall-clock time without adding one runner per tenant. +2. Preserve the current union/disjoint guarantees of BVT groups 0 and 1. +3. Keep global-state and cross-account tests serial. +4. Produce an explainable per-script classification and a merged report. +5. Make the new path opt-in until shadow validation proves it stable. +6. Leave the existing serial invocation available as an immediate fallback. + +## Non-goals + +- Enabling the dormant in-process parallel mode in `mo-tester`. +- Running one MatrixOne deployment per tenant. +- Moving or renaming MatrixOne BVT case directories. +- Converting every serial test into a parallel test in the first rollout. +- Changing Checkin Regression in the first rollout. + +## Considered approaches + +### CI-managed processes on one runner — selected + +The CI workflow starts one MatrixOne deployment and creates two test tenants. It launches one `mo-tester` process per tenant, using isolated working and resource directories. + +This keeps the implementation in `matrixorigin/CI`, does not depend on the disabled `mo-tester` parallel switch, and matches the requested same-cluster tenant isolation model. + +### In-process scheduling inside `mo-tester` + +This would provide tighter report integration but requires coordinated changes in `matrixorigin/mo-tester`. Its existing implementation is fixed to one extra tenant, lacks before/after barriers, and is disabled. It is not selected for the first rollout. + +### One GitHub Actions runner per group + +This is the current complementary-group model. It isolates failures well, but it creates separate MatrixOne deployments rather than using tenants on one deployment. It remains the outer deployment model but is not used for per-tenant parallelism. + +## Current inventory + +The baseline scan is anchored to MatrixOne commit: + +`d17b5a1f8e83cee4999181b9509af6126517985c` + +The full `test/distributed/cases` tree contains 1,155 `.sql` and `.test` scripts. The current PR coverage selection excludes paths containing `optimistic`, leaving 1,133 scripts. + +The conservative initial classification is: + +| Phase | Scripts | Meaning | +|---|---:|---| +| `serial-before` | 29 | Ordered observability producers and verifiers that must run before tenant-generated traffic | +| `parallel-candidate` | 749 | No known global-state rule matched; requires shadow validation | +| `serial-after` | 355 | Cross-account, cluster-global, explicit-user, recovery, or other high-risk behavior | + +The 355 serial-after scripts are selected by exclusive first-match reason: + +| Reason | Scripts | +|---|---:| +| Serial suite override | 256 | +| `mo_ctl` use | 41 | +| Account DDL | 37 | +| Explicit session user/password | 11 | +| `SET GLOBAL` | 6 | +| Sys account ID assumption | 1 | +| `mo_catalog.mo_account` access | 1 | +| Account restore | 1 | +| `system_metrics` access | 1 | + +These numbers describe static candidates, not a claim that all 749 scripts are already safe. Shadow runs can only move scripts from parallel to serial unless a reviewed policy change explicitly relaxes a rule. + +## Classification policy + +The policy is stored as data in `scripts/bvt_tenant_policy.json`. The planner emits `plan.json` and `inventory.tsv`, including the selected phase and every matching reason for every script. + +### Serial-before suites + +The following suites run in their existing lexical order as sys before test tenants are created: + +- `log` +- `result_count` +- `sql_source_type` +- `statement_query_type` +- `zz_statement_query_type` + +They inspect statement, log, and result metadata. Running them after tenant workers would expose them to parallel test traffic. + +### Serial-after suite overrides + +The following suites are serial even when an individual file does not match a content rule: + +- `feature_limit` +- `git4data` +- `mo_cloud` +- `pitr` +- `publication_subscription` +- `snapshot` +- `sql_inject` +- `system` +- `system_variable` +- `task` +- `tenant` +- `tenxcloud_xx` +- `zz_accesscontrol` + +These suites exercise cross-account state, account recovery, global feature configuration, background tasks, failpoints, or external environments. + +### Serial content rules + +A script is serial-after when it contains any of: + +- `CREATE ACCOUNT`, `DROP ACCOUNT`, or `ALTER ACCOUNT` +- `RESTORE ACCOUNT` +- `SHOW ACCOUNTS` +- an `@session` directive with an explicit user or password +- `mo_ctl(...)` +- `mo_feature_registry_*` +- `SET GLOBAL` +- an `@system` command +- `system_metrics` or `mo_debug` +- an explicit `account_id = 0` assumption +- `current_account_id()` or `current_account_name()` +- `mo_catalog.mo_account` +- `KILL CONNECTION` or `KILL QUERY` + +Matching ignores case. Comment matches are intentionally conservative in the first rollout. + +### Explicit overrides + +The policy supports exact-path overrides with a mandatory reason. Precedence is: + +1. non-overridable hard blockers; +2. exact serial overrides; +3. reviewed exact parallel overrides; +4. suite and remaining content rules. + +- `serial-before` and `serial-after` overrides handle hidden ordering or isolation dependencies found in shadow runs. +- A `parallel` override requires review. +- Account DDL, account restore, explicit credentials, and system commands are hard blockers and cannot be overridden to parallel. + +### Affinity groups + +Most parallel scripts are independent scheduling units. Multi-file workloads that depend on lexical execution order are assigned as one unit. The initial affinity rule keeps `benchmark/tpch/**` on one worker and preserves path order. + +Additional affinity groups are added only with an identified producer/consumer dependency. + +## Selection and planning + +MatrixOne's `optools/run_bvt_group.sh` remains the source of truth for complementary groups 0 and 1. + +The orchestrator captures its `-i` selection by invoking it with a temporary no-op `mo-tester/run.sh`. The captured paths are passed to the planner. This avoids copying the group mapping into the CI repository. + +The planner then: + +1. validates every selected path exists below the case root; +2. classifies every path; +3. verifies the three phases are disjoint and their union equals the selected group; +4. creates affinity units; +5. assigns parallel units to workers using longest-first balancing; +6. uses an optional `--timings ` input when available and file size as the deterministic fallback weight; +7. writes per-phase and per-worker include lists. + +Unknown newly added suites are classified by content rules. They are never omitted. The plan records that they used the fallback policy. + +## Runtime architecture + +Each workflow job continues to consume one runner and start one MatrixOne deployment. + +```text +runner +├── MatrixOne deployment +├── serial-before mo-tester (sys) +├── tenant worker 0 mo-tester +├── tenant worker 1 mo-tester +└── serial-after mo-tester (sys) +``` + +The default worker count is two and is configurable from one to four. + +### Worker isolation + +Each worker receives: + +- a unique MatrixOne account, `bvtw_`; +- a unique `mo-tester` working directory; +- a copied 12 MiB MatrixOne resource directory; +- its own `mo.yml`, logs, reports, and pprof directory; +- the shared case tree as read-only input. + +The worker's default JDBC user is `:admin`. The sys credentials remain available to `mo-tester` for its internal sync-commit connection, but classification prevents test scripts with explicit credentials or known global SQL from entering a tenant worker. + +### Phase order + +1. Capture the outer BVT group and build the plan. +2. Run serial-before as sys. +3. Create test tenants. +4. Run all tenant workers and wait for every worker. +5. Collect reports and statuses. +6. Drop all test tenants. +7. Run serial-after as sys if MatrixOne is reachable. +8. Merge reports and return failure when any phase failed or was unexpectedly skipped. + +Test tenants are removed before serial-after so account enumeration and recovery tests see the same account state as the serial baseline. + +## Failure handling + +- A serial-before failure prevents tenant workers from starting. +- Parallel worker assertion failures do not terminate sibling workers; all available reports are collected. +- If MatrixOne remains reachable, serial-after still runs after worker failures to maximize diagnostic coverage. +- If MatrixOne is unreachable, serial-after is recorded as skipped and the job fails. +- A shell trap attempts tenant cleanup on success, failure, timeout, and cancellation. +- Cleanup targets only the exact `bvtw_` accounts created by the current process. +- The final cleanup check queries `mo_catalog.mo_account`; any leaked worker account fails the job. + +Artifacts include: + +- `plan.json` +- `inventory.tsv` +- per-phase and per-worker logs +- original `mo-tester` reports +- merged summary and timing table +- cleanup status + +## Workflow integration and rollout + +The reusable workflows gain: + +- `tenant_parallel_enabled`, default `false` +- `tenant_parallel_workers`, default `2` + +When disabled, the existing `run_bvt_group.sh` invocation is unchanged. When enabled, the workflow calls `scripts/run_bvt_tenant_parallel.sh`. + +Validation uses a MatrixOne workflow pinned to the CI branch commit SHA. The serial baseline and tenant-parallel candidate run in separate jobs and separate MatrixOne deployments. + +Rollout gates: + +1. planner and shell tests pass; +2. a curated integration subset proves phase barriers, tenant isolation, failure reporting, and cleanup; +3. full shadow runs complete at least ten times with no new concurrency-caused failure; +4. selected-script union and result coverage match the serial baseline; +5. no worker account or resource output leaks; +6. no new MatrixOne crash, OOM, or restart; +7. median BVT wall-clock time improves by at least 30%. + +After the gates pass, callers enable tenant parallelism. The disabled path remains available for immediate fallback. + +## Test strategy + +`scripts/test_bvt_tenant_plan.py` covers: + +- phase classification and reason reporting; +- exact override precedence; +- union/disjoint validation; +- unknown-suite handling; +- affinity preservation; +- deterministic worker balancing; +- malformed policy and path rejection. + +Shell integration tests use fake `mysql` and `mo-tester` commands to verify phase order, all-worker wait behavior, exit aggregation, artifacts, and cleanup. + +Repository validation runs: + +- Python unit tests +- `shellcheck` +- `actionlint` +- planner `--dry-run` against a MatrixOne checkout + +The cross-repository validation workflow then runs the curated subset and full shadow comparison. + +## Security and trust + +The workflows already execute the checked-out MatrixOne `optools/run_bvt_group.sh`. Capturing its selection does not broaden that trust boundary. + +Secrets are not written to `plan.json`, inventory, or logs. Generated `mo.yml` files are included in failure artifacts only after password fields are redacted. + +Account and filesystem cleanup use explicit generated paths and account names; no recursive cleanup accepts an empty or unresolved root. From c8ba3e430055d5843ca5c75dee3807cb7f606779 Mon Sep 17 00:00:00 2001 From: "Ariznawl@163.com" Date: Wed, 29 Jul 2026 00:01:44 +0800 Subject: [PATCH 02/16] docs: classify tenant BVT by case directory --- .../2026-07-28-tenant-parallel-bvt-design.md | 167 +++++++++++------- 1 file changed, 100 insertions(+), 67 deletions(-) diff --git a/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md b/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md index 353abf3..c94964e 100644 --- a/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md +++ b/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md @@ -20,7 +20,7 @@ The CI repository baseline for this design is commit: 1. Reduce BVT wall-clock time without adding one runner per tenant. 2. Preserve the current union/disjoint guarantees of BVT groups 0 and 1. 3. Keep global-state and cross-account tests serial. -4. Produce an explainable per-script classification and a merged report. +4. Produce an explainable per-directory classification and a merged report. 5. Make the new path opt-in until shadow validation proves it stable. 6. Leave the existing serial invocation available as an immediate fallback. @@ -54,39 +54,29 @@ The baseline scan is anchored to MatrixOne commit: `d17b5a1f8e83cee4999181b9509af6126517985c` -The full `test/distributed/cases` tree contains 1,155 `.sql` and `.test` scripts. The current PR coverage selection excludes paths containing `optimistic`, leaving 1,133 scripts. +The full `test/distributed/cases` tree contains 1,155 `.sql` and `.test` scripts in 73 top-level directories. The current PR coverage selection excludes the `optimistic` directory, leaving 1,133 scripts in 72 top-level directories. The conservative initial classification is: -| Phase | Scripts | Meaning | -|---|---:|---| -| `serial-before` | 29 | Ordered observability producers and verifiers that must run before tenant-generated traffic | -| `parallel-candidate` | 749 | No known global-state rule matched; requires shadow validation | -| `serial-after` | 355 | Cross-account, cluster-global, explicit-user, recovery, or other high-risk behavior | +| Phase | Directories | Scripts | Meaning | +|---|---:|---:|---| +| `serial-before` | 5 | 29 | Ordered observability producers and verifiers that must run before tenant-generated traffic | +| `parallel-candidate` | 29 | 215 | Every script in the directory passed the conservative scan; requires shadow validation | +| `serial-after` | 38 | 889 | At least one script in the directory has cross-account, cluster-global, explicit-user, recovery, or other high-risk behavior | -The 355 serial-after scripts are selected by exclusive first-match reason: +The directory is the smallest scheduling and policy unit. The planner never divides scripts from the same top-level directory between phases or workers. -| Reason | Scripts | -|---|---:| -| Serial suite override | 256 | -| `mo_ctl` use | 41 | -| Account DDL | 37 | -| Explicit session user/password | 11 | -| `SET GLOBAL` | 6 | -| Sys account ID assumption | 1 | -| `mo_catalog.mo_account` access | 1 | -| Account restore | 1 | -| `system_metrics` access | 1 | - -These numbers describe static candidates, not a claim that all 749 scripts are already safe. Shadow runs can only move scripts from parallel to serial unless a reviewed policy change explicitly relaxes a rule. +This changes the expected optimization ceiling: only 215 of 1,133 scripts are initially parallel candidates. The simpler commands and safer maintenance are preferred over file-level parallel coverage; the actual wall-clock benefit must be established by shadow runs. ## Classification policy -The policy is stored as data in `scripts/bvt_tenant_policy.json`. The planner emits `plan.json` and `inventory.tsv`, including the selected phase and every matching reason for every script. +The policy is stored as data in `scripts/bvt_tenant_policy.json`. It lists each top-level directory, its phase, and its reason. The planner emits `plan.json` and `inventory.tsv` with one record per directory. + +The policy is explicit rather than reclassifying individual files at runtime. A newly added directory defaults to `serial-after` and is reported as unreviewed, so it is exercised but cannot enter a tenant worker without a policy review. -### Serial-before suites +### Serial-before directories -The following suites run in their existing lexical order as sys before test tenants are created: +The following directories run in their existing lexical order as sys before test tenants are created: - `log` - `result_count` @@ -96,29 +86,90 @@ The following suites run in their existing lexical order as sys before test tena They inspect statement, log, and result metadata. Running them after tenant workers would expose them to parallel test traffic. -### Serial-after suite overrides - -The following suites are serial even when an individual file does not match a content rule: - +### Parallel directories + +The following directories are the initial parallel candidates: + +- `analyze` +- `auto_increment` +- `benchmark` +- `charset_collation` +- `comment` +- `cte` +- `dataXtest` +- `distinct` +- `dtype` +- `expression` +- `fake_pk` +- `fulltext` +- `geo` +- `keyword` +- `operator` +- `pg_cast` +- `plan_cache` +- `plugin` +- `procedure` +- `qexec` +- `recursive_cte` +- `replace_statement` +- `sample` +- `sequence` +- `time_window` +- `udf` +- `union` +- `view` +- `window` + +Each directory is assigned to exactly one tenant worker. `benchmark` remains one unit, which preserves the lexical DDL, load, query, and cleanup order below `benchmark/tpch`. + +### Serial-after directories + +The following directories run as sys after tenant workers: + +- `array` +- `database` +- `ddl` +- `disttae` +- `dml` - `feature_limit` +- `foreign_key` +- `function` - `git4data` +- `hint` +- `iceberg` +- `join` +- `load_data` +- `metadata` - `mo_cloud` +- `optimizer` +- `pessimistic_transaction` - `pitr` +- `prepare` - `publication_subscription` +- `query_result` +- `save_query_result` +- `security` +- `set` - `snapshot` - `sql_inject` +- `stage` +- `subquery` - `system` - `system_variable` +- `table` - `task` +- `temporary` - `tenant` - `tenxcloud_xx` +- `util` +- `vector` - `zz_accesscontrol` -These suites exercise cross-account state, account recovery, global feature configuration, background tasks, failpoints, or external environments. +These directories exercise cross-account state, account recovery, global feature configuration, background tasks, failpoints, external environments, or contain at least one script matching a global-state rule. ### Serial content rules -A script is serial-after when it contains any of: +A directory is classified as serial-after during policy review when any script below it contains: - `CREATE ACCOUNT`, `DROP ACCOUNT`, or `ALTER ACCOUNT` - `RESTORE ACCOUNT` @@ -134,44 +185,26 @@ A script is serial-after when it contains any of: - `mo_catalog.mo_account` - `KILL CONNECTION` or `KILL QUERY` -Matching ignores case. Comment matches are intentionally conservative in the first rollout. - -### Explicit overrides - -The policy supports exact-path overrides with a mandatory reason. Precedence is: - -1. non-overridable hard blockers; -2. exact serial overrides; -3. reviewed exact parallel overrides; -4. suite and remaining content rules. - -- `serial-before` and `serial-after` overrides handle hidden ordering or isolation dependencies found in shadow runs. -- A `parallel` override requires review. -- Account DDL, account restore, explicit credentials, and system commands are hard blockers and cannot be overridden to parallel. - -### Affinity groups - -Most parallel scripts are independent scheduling units. Multi-file workloads that depend on lexical execution order are assigned as one unit. The initial affinity rule keeps `benchmark/tpch/**` on one worker and preserves path order. - -Additional affinity groups are added only with an identified producer/consumer dependency. +Matching ignores case. Comment matches are intentionally conservative in the first rollout. The scan explains why a directory is serial; it does not split safe-looking files out of that directory. ## Selection and planning MatrixOne's `optools/run_bvt_group.sh` remains the source of truth for complementary groups 0 and 1. -The orchestrator captures its `-i` selection by invoking it with a temporary no-op `mo-tester/run.sh`. The captured paths are passed to the planner. This avoids copying the group mapping into the CI repository. +The orchestrator captures its `-i` selection by invoking it with a temporary no-op `mo-tester/run.sh`. The captured paths are reduced to their top-level directories and passed to the planner. This avoids copying the group mapping into the CI repository. The planner then: -1. validates every selected path exists below the case root; -2. classifies every path; -3. verifies the three phases are disjoint and their union equals the selected group; -4. creates affinity units; -5. assigns parallel units to workers using longest-first balancing; -6. uses an optional `--timings ` input when available and file size as the deterministic fallback weight; -7. writes per-phase and per-worker include lists. +1. validates every selected directory exists immediately below the case root; +2. verifies that every selected script maps to exactly one selected top-level directory; +3. looks up each directory in the explicit policy; +4. defaults an unknown directory to `serial-after` and reports it as unreviewed; +5. verifies the three phases are disjoint and their directory union equals the selected group; +6. assigns whole parallel directories to workers using longest-first balancing; +7. uses an optional `--timings ` input when available and aggregate directory file size as the deterministic fallback weight; +8. writes per-phase and per-worker directory include lists. -Unknown newly added suites are classified by content rules. They are never omitted. The plan records that they used the fallback policy. +Every `mo-tester -i` argument is therefore a comma-separated list of directories, not hundreds of individual scripts. ## Runtime architecture @@ -205,7 +238,7 @@ The worker's default JDBC user is `:admin`. The sys credentials remain 1. Capture the outer BVT group and build the plan. 2. Run serial-before as sys. 3. Create test tenants. -4. Run all tenant workers and wait for every worker. +4. Run all tenant workers, each with a directory include list, and wait for every worker. 5. Collect reports and statuses. 6. Drop all test tenants. 7. Run serial-after as sys if MatrixOne is reachable. @@ -251,7 +284,7 @@ Rollout gates: 4. selected-script union and result coverage match the serial baseline; 5. no worker account or resource output leaks; 6. no new MatrixOne crash, OOM, or restart; -7. median BVT wall-clock time improves by at least 30%. +7. the measured wall-clock improvement justifies enabling the feature; the previous 30% target is no longer assumed because directory-level classification leaves only 215 scripts parallel. After the gates pass, callers enable tenant parallelism. The disabled path remains available for immediate fallback. @@ -259,12 +292,12 @@ After the gates pass, callers enable tenant parallelism. The disabled path remai `scripts/test_bvt_tenant_plan.py` covers: -- phase classification and reason reporting; -- exact override precedence; -- union/disjoint validation; -- unknown-suite handling; -- affinity preservation; -- deterministic worker balancing; +- directory phase lookup and reason reporting; +- rejection of file-level policy entries; +- directory union/disjoint validation; +- unknown-directory serial fallback; +- preservation of all scripts below each selected directory; +- deterministic whole-directory worker balancing; - malformed policy and path rejection. Shell integration tests use fake `mysql` and `mo-tester` commands to verify phase order, all-worker wait behavior, exit aggregation, artifacts, and cleanup. From da7007976e286151e79e6b75eab2fa6a4b7434aa Mon Sep 17 00:00:00 2001 From: "Ariznawl@163.com" Date: Wed, 29 Jul 2026 10:23:40 +0800 Subject: [PATCH 03/16] docs: plan tenant-parallel BVT implementation --- .../plans/2026-07-29-tenant-parallel-bvt.md | 517 ++++++++++++++++++ 1 file changed, 517 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md diff --git a/docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md b/docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md new file mode 100644 index 0000000..67da0cb --- /dev/null +++ b/docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md @@ -0,0 +1,517 @@ +# Tenant-Parallel BVT Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Run the directory-safe portion of each MatrixOne PR BVT group concurrently in two tenant accounts on one deployment, while preserving sys-account serial phases and the existing fallback. + +**Architecture:** A Python planner converts the MatrixOne group runner's selected script list into whole top-level directory units using an explicit JSON policy. A shell orchestrator creates isolated `mo-tester` copies and tenant accounts, runs serial-before, tenant workers, cleanup, and serial-after in order, and records one artifact directory. The two reusable workflows checkout the matching CI revision and select the old or new path through an opt-in input. + +**Tech Stack:** Python 3 standard library, Bash, MySQL CLI, `mo-tester`, GitHub Actions YAML. + +## Global Constraints + +- The smallest scheduling unit is one immediate child directory of `test/distributed/cases`; no file-level policy or worker assignment is allowed. +- The selected directories are exactly the current MatrixOne group intersected with the policy; no selected script may be omitted or assigned twice. +- Unknown top-level directories run in `serial-after` and are reported as unreviewed. +- The default worker count is `2`; accepted values are `1` through `4`. +- Each workflow job continues to use one runner and one MatrixOne deployment. +- Worker accounts are removed before `serial-after`. +- `tenant_parallel_enabled` defaults to `false`, preserving the existing `run_bvt_group.sh` path. +- Test code uses only the Python standard library and fake local executables; repository tests must not require a live MatrixOne deployment. + +--- + +### Task 1: Directory Policy and Planner + +**Files:** +- Create: `scripts/bvt_tenant_policy.json` +- Create: `scripts/bvt_tenant_plan.py` +- Create: `scripts/test_bvt_tenant_plan.py` + +**Interfaces:** +- Consumes: MatrixOne case root, newline-delimited selected script paths captured from `run_bvt_group.sh`, worker count, and JSON policy. +- Produces: `plan.json`, `inventory.tsv`, `serial-before.include`, `worker-N.include`, and `serial-after.include`. +- Python API: + - `load_policy(path: pathlib.Path) -> dict[str, PolicyEntry]` + - `build_plan(case_root: pathlib.Path, selected_file: pathlib.Path, policy: dict[str, PolicyEntry], workers: int) -> dict` + - `write_plan(plan: dict, output_dir: pathlib.Path) -> None` + +- [ ] **Step 1: Write failing policy-validation and directory-planning tests** + +Create `scripts/test_bvt_tenant_plan.py` with `unittest`. Fixtures must use literal expected directories and script counts: + +```python +def test_build_plan_assigns_whole_directories_and_preserves_selection(self): + self.case("alpha/a.sql", "x") + self.case("alpha/b.sql", "xx") + self.case("beta/a.sql", "xxx") + self.case("global/a.sql", "xxxx") + selected = self.selected("alpha/a.sql", "alpha/b.sql", "beta/a.sql", "global/a.sql") + policy = { + "alpha": PolicyEntry("parallel", "safe"), + "beta": PolicyEntry("parallel", "safe"), + "global": PolicyEntry("serial-after", "global state"), + } + + plan = build_plan(self.case_root, selected, policy, workers=2) + + self.assertEqual(plan["serial_after"], [str(self.case_root / "global")]) + self.assertEqual( + sorted(path for worker in plan["workers"] for path in worker["directories"]), + sorted([str(self.case_root / "alpha"), str(self.case_root / "beta")]), + ) + self.assertEqual(plan["selected_script_count"], 4) +``` + +Also cover: + +- duplicate directory entries rejected by the JSON object loader; +- invalid phase rejected; +- `workers=0` and `workers=5` rejected; +- a selected file outside the case root rejected; +- a partial selection from one top-level directory rejected; +- an unknown directory assigned to `serial-after` with `reviewed=false`; +- serial-before, worker, and serial-after sets are disjoint; +- longest-first directory weighting is deterministic; +- emitted include files contain absolute directory paths, never `.sql` or `.test` paths. + +- [ ] **Step 2: Run the planner tests and verify RED** + +Run: + +```bash +python3 -m unittest scripts/test_bvt_tenant_plan.py -v +``` + +Expected: `ModuleNotFoundError` for `scripts.bvt_tenant_plan`. + +- [ ] **Step 3: Add the explicit 72-directory policy** + +Create schema version 1 with one object per directory: + +```json +{ + "schema_version": 1, + "directories": { + "log": {"phase": "serial-before", "reason": "statement and log metadata ordering"}, + "result_count": {"phase": "serial-before", "reason": "statement result metadata ordering"}, + "sql_source_type": {"phase": "serial-before", "reason": "statement source metadata ordering"}, + "statement_query_type": {"phase": "serial-before", "reason": "statement query metadata producer"}, + "zz_statement_query_type": {"phase": "serial-before", "reason": "statement query metadata verifier"}, + + "analyze": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "auto_increment": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "benchmark": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker; keep the entire directory on one worker"}, + "charset_collation": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "comment": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "cte": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "dataXtest": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "distinct": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "dtype": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "expression": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "fake_pk": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "fulltext": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "geo": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "keyword": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "operator": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "pg_cast": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "plan_cache": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "plugin": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "procedure": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "qexec": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "recursive_cte": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "replace_statement": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "sample": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "sequence": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "time_window": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "udf": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "union": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "view": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "window": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + + "array": {"phase": "serial-after", "reason": "contains mo_ctl"}, + "database": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, + "ddl": {"phase": "serial-after", "reason": "contains account DDL, mo_ctl, or SET GLOBAL"}, + "disttae": {"phase": "serial-after", "reason": "contains account DDL or mo_ctl"}, + "dml": {"phase": "serial-after", "reason": "contains account DDL, mo_ctl, or global state"}, + "feature_limit": {"phase": "serial-after", "reason": "global feature registry and account state"}, + "foreign_key": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, + "function": {"phase": "serial-after", "reason": "contains mo_ctl, account DDL, or account identity functions"}, + "git4data": {"phase": "serial-after", "reason": "branch, account, debug, and global state"}, + "hint": {"phase": "serial-after", "reason": "contains account DDL, SET GLOBAL, or system metrics"}, + "iceberg": {"phase": "serial-after", "reason": "contains explicit account sessions and SET GLOBAL"}, + "join": {"phase": "serial-after", "reason": "contains mo_ctl or account DDL"}, + "load_data": {"phase": "serial-after", "reason": "contains account DDL or mo_catalog.mo_account"}, + "metadata": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, + "mo_cloud": {"phase": "serial-after", "reason": "external environment and system metrics"}, + "optimizer": {"phase": "serial-after", "reason": "contains mo_ctl or current account assumptions"}, + "pessimistic_transaction": {"phase": "serial-after", "reason": "account, restore, debug, and transaction-global state"}, + "pitr": {"phase": "serial-after", "reason": "account recovery and debug state"}, + "prepare": {"phase": "serial-after", "reason": "contains account DDL or SET GLOBAL"}, + "publication_subscription": {"phase": "serial-after", "reason": "cross-account publication state"}, + "query_result": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, + "save_query_result": {"phase": "serial-after", "reason": "contains account enumeration and DDL"}, + "security": {"phase": "serial-after", "reason": "explicit account sessions and SET GLOBAL"}, + "set": {"phase": "serial-after", "reason": "contains account DDL or SET GLOBAL"}, + "snapshot": {"phase": "serial-after", "reason": "account snapshot and restore state"}, + "sql_inject": {"phase": "serial-after", "reason": "cluster-global failpoint state"}, + "stage": {"phase": "serial-after", "reason": "contains account DDL or mo_ctl"}, + "subquery": {"phase": "serial-after", "reason": "contains mo_ctl"}, + "system": {"phase": "serial-after", "reason": "system account behavior"}, + "system_variable": {"phase": "serial-after", "reason": "global system variables and account sessions"}, + "table": {"phase": "serial-after", "reason": "cluster table, account, and system metrics state"}, + "task": {"phase": "serial-after", "reason": "background task and account state"}, + "temporary": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, + "tenant": {"phase": "serial-after", "reason": "tenant and cross-account behavior"}, + "tenxcloud_xx": {"phase": "serial-after", "reason": "external environment"}, + "util": {"phase": "serial-after", "reason": "contains mo_ctl"}, + "vector": {"phase": "serial-after", "reason": "contains mo_ctl or SET GLOBAL"}, + "zz_accesscontrol": {"phase": "serial-after", "reason": "account, password, and global access-control state"} + } +} +``` + +- [ ] **Step 4: Implement the minimal planner** + +Implement duplicate-key detection with `json.load(..., object_pairs_hook=...)`, normalize every selected path with `Path.resolve()`, map it to `relative.parts[0]`, and reject partial directory selection by comparing selected `.sql`/`.test` files with a recursive discovery of that directory. + +Balance only `parallel` directories: + +```python +units = sorted(parallel_units, key=lambda unit: (-unit["weight_bytes"], unit["name"])) +worker_units = [{"index": index, "weight_bytes": 0, "directories": []} for index in range(workers)] +for unit in units: + target = min(worker_units, key=lambda worker: (worker["weight_bytes"], worker["index"])) + target["directories"].append(unit["path"]) + target["weight_bytes"] += unit["weight_bytes"] +``` + +Write include files as a single comma-separated line of absolute directory paths. Empty phases produce empty files. + +- [ ] **Step 5: Run planner tests and verify GREEN** + +Run: + +```bash +python3 -m unittest scripts/test_bvt_tenant_plan.py -v +``` + +Expected: all tests pass. + +- [ ] **Step 6: Dry-run the planner against MatrixOne main** + +Capture both group selections with a fake `mo-tester/run.sh`, plan both groups, and verify the union contains 72 directories and 1,133 scripts. + +Expected aggregate: + +```text +serial-before: 5 directories, 29 scripts +parallel: 29 directories, 215 scripts +serial-after: 38 directories, 889 scripts +``` + +- [ ] **Step 7: Commit the planner** + +```bash +git add scripts/bvt_tenant_policy.json scripts/bvt_tenant_plan.py scripts/test_bvt_tenant_plan.py +git commit -m "feat: plan BVT execution by case directory" +``` + +--- + +### Task 2: Tenant-Parallel Shell Orchestrator + +**Files:** +- Create: `scripts/run_bvt_tenant_parallel.sh` +- Create: `scripts/test_run_bvt_tenant_parallel.sh` + +**Interfaces:** +- Consumes: + - `--tester-dir PATH` + - `--case-root PATH` + - `--group-runner PATH` + - `--group 0|1` + - `--policy PATH` + - `--planner PATH` + - `--output-dir PATH` + - `--workers 1..4` + - optional `--resource-dir PATH` + - optional MySQL flags defaulting to `127.0.0.1:6001`, `dump/111` +- Produces: planner artifacts, per-phase logs and reports, `summary.tsv`, and a nonzero aggregate exit status. + +- [ ] **Step 1: Write the failing shell integration test** + +Use a temporary fixture containing: + +- a fake group runner that invokes its tester with a literal four-file include list; +- a fake original `mo-tester` tree with `run.sh`, `mo.yml`, `run.yml`, `lib/`, `log/`, and `report/`; +- a fake `mysql` executable that appends SQL to `${FAKE_MYSQL_LOG}`; +- a fake planner implementing the real planner CLI output contract. + +Assert observable behavior: + +```bash +assert_order serial-before create-account worker-0 worker-1 drop-account serial-after +assert_contains "$FAKE_MYSQL_LOG" "create account" +assert_contains "$FAKE_MYSQL_LOG" "drop account" +assert_file "$output_dir/plan.json" +assert_file "$output_dir/summary.tsv" +assert_not_contains "$output_dir/worker-0.include" ".sql" +``` + +Add separate cases proving: + +- sibling workers finish even if worker 0 fails; +- worker failure makes the final script fail; +- serial-after still runs after a worker assertion failure; +- serial-before failure prevents account creation and workers; +- account cleanup occurs on worker failure; +- an empty parallel phase skips account creation; +- generated worker `mo.yml` uses `bvtw_g_w:admin`; +- resource directories are copied per phase when `--resource-dir` is supplied. + +- [ ] **Step 2: Run the shell test and verify RED** + +Run: + +```bash +bash scripts/test_run_bvt_tenant_parallel.sh +``` + +Expected: fail because `scripts/run_bvt_tenant_parallel.sh` does not exist. + +- [ ] **Step 3: Implement argument validation and group capture** + +Use `set -uo pipefail`, resolve all required paths, validate workers and group, create only children below the explicit output directory, and install a trap that drops only names recorded in an in-memory `created_accounts` array. + +The capture tester parses the group runner's `-i` argument and writes one selected script per line to `${output_dir}/selected-files.txt`. + +- [ ] **Step 4: Implement isolated tester preparation** + +For every phase copy only: + +```text +run.sh +run.yml +mo.yml +kafka.yml +log4j.properties +pprof.sh +lib/ +``` + +Create fresh `log/` and `report/`. Copy the MatrixOne resource directory separately for every phase because `mo-tester` removes configured output paths during startup and cleanup. + +For workers, replace only the YAML `user.name` and `user.password` values; retain `sysuser` and `syspass`. + +- [ ] **Step 5: Implement phase execution and status aggregation** + +Run serial-before synchronously. Create accounts with: + +```sql +create account if not exists `bvtw_g0_w0` admin_name 'admin' identified by '111'; +``` + +Run every tenant worker in a background subshell with `set -o pipefail` and `tee`. Wait for all PIDs without exiting on the first failure. Drop every created account, then run serial-after when MySQL is reachable. + +Write `summary.tsv` columns: + +```text +phase name status include_file log_file +``` + +Exit nonzero when a required phase failed, cleanup failed, or serial-after was skipped because MatrixOne was unreachable. + +- [ ] **Step 6: Run shell tests and verify GREEN** + +Run: + +```bash +bash scripts/test_run_bvt_tenant_parallel.sh +``` + +Expected: all cases print `ok` and the script exits 0. + +- [ ] **Step 7: Run planner and shell tests together** + +Run: + +```bash +python3 -m unittest scripts/test_bvt_tenant_plan.py -v +bash scripts/test_run_bvt_tenant_parallel.sh +bash -n scripts/run_bvt_tenant_parallel.sh +``` + +Expected: all pass. + +- [ ] **Step 8: Commit the orchestrator** + +```bash +git add scripts/run_bvt_tenant_parallel.sh scripts/test_run_bvt_tenant_parallel.sh +git commit -m "feat: orchestrate tenant-parallel BVT phases" +``` + +--- + +### Task 3: Reusable Workflow Integration + +**Files:** +- Modify: `.github/workflows/e2e-compose-parallel.yaml` +- Modify: `.github/workflows/e2e-standalone-parallel.yaml` +- Create: `scripts/test_bvt_workflow_contract.py` + +**Interfaces:** +- New workflow inputs: + - `tenant_parallel_enabled`: boolean, default `false` + - `tenant_parallel_workers`: number, default `2` + - `ci_ref`: string, default `main` +- Existing caller contracts and fallback commands remain unchanged. + +- [ ] **Step 1: Write a failing executable workflow-contract test** + +Load both YAML files as text and use a small structural extractor to assert: + +- all three inputs exist with exact defaults; +- the active BVT job checks out `matrixorigin/CI` at `${{ inputs.ci_ref }}` into `.ci/tenant-parallel`; +- the old `run_bvt_group.sh` command is guarded by disabled tenant parallelism; +- the orchestrator command is guarded by enabled tenant parallelism; +- compose passes its case root without a resource directory; +- standalone passes `head/test/distributed/resources`; +- artifact upload includes the orchestrator output directory. + +The test must fail on the current workflows because the new inputs and guarded orchestrator call do not exist. + +- [ ] **Step 2: Run the workflow test and verify RED** + +Run: + +```bash +python3 -m unittest scripts/test_bvt_workflow_contract.py -v +``` + +Expected: failures naming missing `tenant_parallel_enabled`. + +- [ ] **Step 3: Add opt-in inputs and CI checkout** + +Add the three inputs to both `workflow_call.inputs`. In each active job, checkout: + +```yaml +- name: Checkout tenant-parallel BVT scripts + if: ${{ inputs.tenant_parallel_enabled }} + uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 + with: + repository: matrixorigin/CI + ref: ${{ inputs.ci_ref }} + path: .ci/tenant-parallel +``` + +- [ ] **Step 4: Guard fallback and tenant-parallel execution** + +Keep the current group assignment and manifest generation. Replace only the final test invocation branch: + +```bash +if [ '${{ inputs.tenant_parallel_enabled }}' = 'true' ]; then + bash "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/run_bvt_tenant_parallel.sh" \ + --tester-dir "$GITHUB_WORKSPACE/mo-tester" \ + --case-root "$GITHUB_WORKSPACE/test/distributed/cases" \ + --group-runner "${bvt_runner}" \ + --group "${bvt_group}" \ + --workers '${{ inputs.tenant_parallel_workers }}' \ + --policy "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/bvt_tenant_policy.json" \ + --planner "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/bvt_tenant_plan.py" \ + --output-dir "${RUNNER_TEMP}/bvt-tenant-compose" +else + bash "${bvt_runner}" \ + "$GITHUB_WORKSPACE/mo-tester" \ + "$GITHUB_WORKSPACE/test/distributed/cases" \ + "${bvt_group}" +fi +``` + +The standalone branch uses `head/test/distributed/cases`, adds `--resource-dir "$GITHUB_WORKSPACE/head/test/distributed/resources"`, and writes to `bvt-tenant-pessimistic`. + +- [ ] **Step 5: Extend artifact paths and summaries** + +Upload the existing combined log plus the corresponding tenant output directory. Add plan phase counts and `summary.tsv` to `$GITHUB_STEP_SUMMARY` when present. + +- [ ] **Step 6: Run contract and script tests** + +Run: + +```bash +python3 -m unittest scripts/test_bvt_tenant_plan.py scripts/test_bvt_workflow_contract.py -v +bash scripts/test_run_bvt_tenant_parallel.sh +bash -n scripts/run_bvt_tenant_parallel.sh +ruby -e 'require "yaml"; ARGV.each { |path| YAML.load_file(path, aliases: true) }' \ + .github/workflows/e2e-compose-parallel.yaml \ + .github/workflows/e2e-standalone-parallel.yaml +``` + +Expected: every command exits 0. + +- [ ] **Step 7: Commit workflow integration** + +```bash +git add .github/workflows/e2e-compose-parallel.yaml .github/workflows/e2e-standalone-parallel.yaml scripts/test_bvt_workflow_contract.py +git commit -m "ci: add opt-in tenant-parallel BVT execution" +``` + +--- + +### Task 4: Full Verification and Trial Handoff + +**Files:** +- Modify if verification finds a tested defect: only files introduced or named above. + +**Interfaces:** +- Produces a local CI commit SHA suitable for pinning from a MatrixOne trial branch. + +- [ ] **Step 1: Run the complete local verification** + +```bash +python3 -m unittest scripts/test_bvt_tenant_plan.py scripts/test_bvt_workflow_contract.py -v +bash scripts/test_run_bvt_tenant_parallel.sh +bash -n scripts/run_bvt_tenant_parallel.sh +git diff --check +``` + +- [ ] **Step 2: Run a real MatrixOne-main planning dry-run** + +Use `origin/main:optools/run_bvt_group.sh` from `/Users/ariznawl/weilu/matrixone` and the real case tree exported from that ref. Verify both groups together produce: + +```text +72 selected directories +1133 selected scripts +5 serial-before directories +29 parallel directories +38 serial-after directories +``` + +- [ ] **Step 3: Inspect the final diff** + +Confirm: + +- no MatrixOne or `mo-tester` source file changed; +- workflow defaults preserve serial behavior; +- no password appears in `plan.json`, `inventory.tsv`, or summary output; +- cleanup targets only `bvtw_g_w`; +- no generated test artifact is staged. + +- [ ] **Step 4: Prepare the MatrixOne trial configuration** + +The caller pins both reusable workflows to the final CI SHA and passes: + +```yaml +with: + tenant_parallel_enabled: true + tenant_parallel_workers: 2 + ci_ref: codex/tenant-parallel-bvt +``` + +For a non-experimental rollout, replace the branch in both the reusable-workflow `uses:` reference and `ci_ref` with the same value printed by `git rev-parse HEAD`. + +Run the Compose + Proxy and Launch + Pessimistic jobs once. Compare BVT result coverage, leaked accounts, MatrixOne crashes/restarts, and wall-clock time with the serial baseline. + +- [ ] **Step 5: Record final status** + +Report the final local branch, commit SHA, verification commands, exact trial inputs, and the remaining push-permission limitation. From 41587b1da04cf31ea432c51abcd28316231c93f3 Mon Sep 17 00:00:00 2001 From: "Ariznawl@163.com" Date: Wed, 29 Jul 2026 10:28:12 +0800 Subject: [PATCH 04/16] feat: plan BVT execution by case directory --- scripts/bvt_tenant_plan.py | 231 +++++++++++++++++++++++++++++++ scripts/bvt_tenant_policy.json | 79 +++++++++++ scripts/test_bvt_tenant_plan.py | 236 ++++++++++++++++++++++++++++++++ 3 files changed, 546 insertions(+) create mode 100644 scripts/bvt_tenant_plan.py create mode 100644 scripts/bvt_tenant_policy.json create mode 100644 scripts/test_bvt_tenant_plan.py diff --git a/scripts/bvt_tenant_plan.py b/scripts/bvt_tenant_plan.py new file mode 100644 index 0000000..5167c6b --- /dev/null +++ b/scripts/bvt_tenant_plan.py @@ -0,0 +1,231 @@ +#!/usr/bin/env python3 + +import argparse +import csv +import json +from dataclasses import dataclass +from pathlib import Path + + +VALID_PHASES = {"serial-before", "parallel", "serial-after"} + + +@dataclass(frozen=True) +class PolicyEntry: + phase: str + reason: str + + +def _reject_duplicate_keys(pairs): + result = {} + for key, value in pairs: + if key in result: + raise ValueError(f"duplicate JSON key: {key}") + result[key] = value + return result + + +def load_policy(path: Path) -> dict[str, PolicyEntry]: + with path.open() as policy_file: + raw = json.load(policy_file, object_pairs_hook=_reject_duplicate_keys) + if raw.get("schema_version") != 1: + raise ValueError("policy schema_version must be 1") + directories = raw.get("directories") + if not isinstance(directories, dict): + raise ValueError("policy directories must be an object") + + policy = {} + for name, value in directories.items(): + if not name or "/" in name or "\\" in name: + raise ValueError(f"invalid top-level directory name: {name}") + if not isinstance(value, dict): + raise ValueError(f"policy entry for {name} must be an object") + phase = value.get("phase") + reason = value.get("reason") + if phase not in VALID_PHASES: + raise ValueError(f"invalid phase for {name}: {phase}") + if not isinstance(reason, str) or not reason.strip(): + raise ValueError(f"policy reason for {name} must be non-empty") + policy[name] = PolicyEntry(phase=phase, reason=reason.strip()) + return policy + + +def _scripts_below(directory: Path) -> set[Path]: + return { + path.resolve() + for path in directory.rglob("*") + if path.is_file() and path.suffix in {".sql", ".test"} + } + + +def _selected_by_directory(case_root: Path, selected_file: Path) -> dict[str, set[Path]]: + selected = {} + for line_number, raw_line in enumerate(selected_file.read_text().splitlines(), 1): + if not raw_line.strip(): + continue + path = Path(raw_line.strip()).resolve() + try: + relative = path.relative_to(case_root) + except ValueError as error: + raise ValueError( + f"selected file outside case root at line {line_number}: {path}" + ) from error + if len(relative.parts) < 2: + raise ValueError(f"selected path is not below a top-level directory: {path}") + if not path.is_file() or path.suffix not in {".sql", ".test"}: + raise ValueError(f"selected path is not a .sql or .test file: {path}") + selected.setdefault(relative.parts[0], set()).add(path) + if not selected: + raise ValueError("selected file list is empty") + return selected + + +def build_plan( + case_root: Path, + selected_file: Path, + policy: dict[str, PolicyEntry], + workers: int, +) -> dict: + if workers < 1 or workers > 4: + raise ValueError("workers must be between 1 and 4") + case_root = case_root.resolve() + if not case_root.is_dir(): + raise ValueError(f"case root is not a directory: {case_root}") + selected_by_directory = _selected_by_directory(case_root, selected_file) + + units = [] + for name in sorted(selected_by_directory): + directory = (case_root / name).resolve() + if directory.parent != case_root or not directory.is_dir(): + raise ValueError(f"selected top-level directory is invalid: {name}") + discovered = _scripts_below(directory) + selected = selected_by_directory[name] + if selected != discovered: + missing = sorted(str(path) for path in discovered - selected) + extra = sorted(str(path) for path in selected - discovered) + detail = f"missing={missing[:3]}, extra={extra[:3]}" + raise ValueError(f"partial directory selection for {name}: {detail}") + + entry = policy.get(name) + reviewed = entry is not None + if entry is None: + entry = PolicyEntry( + phase="serial-after", + reason="unreviewed directory defaults to serial-after", + ) + units.append( + { + "name": name, + "path": str(directory), + "phase": entry.phase, + "reason": entry.reason, + "reviewed": reviewed, + "script_count": len(discovered), + "weight_bytes": sum(path.stat().st_size for path in discovered), + "worker": None, + } + ) + + worker_units = [ + {"index": index, "weight_bytes": 0, "directories": []} + for index in range(workers) + ] + parallel_units = sorted( + (unit for unit in units if unit["phase"] == "parallel"), + key=lambda unit: (-unit["weight_bytes"], unit["name"]), + ) + for unit in parallel_units: + target = min( + worker_units, + key=lambda worker: (worker["weight_bytes"], worker["index"]), + ) + target["directories"].append(unit["path"]) + target["weight_bytes"] += unit["weight_bytes"] + unit["worker"] = target["index"] + + return { + "schema_version": 1, + "case_root": str(case_root), + "selected_script_count": sum(unit["script_count"] for unit in units), + "selected_directory_count": len(units), + "serial_before": [ + unit["path"] for unit in units if unit["phase"] == "serial-before" + ], + "workers": worker_units, + "serial_after": [ + unit["path"] for unit in units if unit["phase"] == "serial-after" + ], + "directories": units, + } + + +def _write_include(path: Path, directories: list[str]) -> None: + path.write_text(",".join(directories) + ("\n" if directories else "")) + + +def write_plan(plan: dict, output_dir: Path) -> None: + output_dir.mkdir(parents=True, exist_ok=True) + (output_dir / "plan.json").write_text( + json.dumps(plan, indent=2, sort_keys=True) + "\n" + ) + _write_include(output_dir / "serial-before.include", plan["serial_before"]) + for worker in plan["workers"]: + _write_include( + output_dir / f"worker-{worker['index']}.include", + worker["directories"], + ) + _write_include(output_dir / "serial-after.include", plan["serial_after"]) + + with (output_dir / "inventory.tsv").open("w", newline="") as inventory_file: + writer = csv.writer(inventory_file, delimiter="\t", lineterminator="\n") + writer.writerow( + [ + "directory", + "phase", + "worker", + "reviewed", + "script_count", + "weight_bytes", + "reason", + ] + ) + for unit in plan["directories"]: + writer.writerow( + [ + unit["name"], + unit["phase"], + "" if unit["worker"] is None else unit["worker"], + str(unit["reviewed"]).lower(), + unit["script_count"], + unit["weight_bytes"], + unit["reason"], + ] + ) + + +def main() -> None: + parser = argparse.ArgumentParser( + description="Plan tenant-parallel BVT execution by top-level case directory." + ) + parser.add_argument("--case-root", required=True, type=Path) + parser.add_argument("--selected-files", required=True, type=Path) + parser.add_argument("--policy", required=True, type=Path) + parser.add_argument("--workers", required=True, type=int) + parser.add_argument("--output-dir", required=True, type=Path) + args = parser.parse_args() + + policy = load_policy(args.policy) + plan = build_plan(args.case_root, args.selected_files, policy, args.workers) + write_plan(plan, args.output_dir) + print( + "BVT directory plan: " + f"{plan['selected_directory_count']} directories, " + f"{plan['selected_script_count']} scripts, " + f"{len(plan['serial_before'])} serial-before, " + f"{sum(len(worker['directories']) for worker in plan['workers'])} parallel, " + f"{len(plan['serial_after'])} serial-after" + ) + + +if __name__ == "__main__": + main() diff --git a/scripts/bvt_tenant_policy.json b/scripts/bvt_tenant_policy.json new file mode 100644 index 0000000..0b2bdf3 --- /dev/null +++ b/scripts/bvt_tenant_policy.json @@ -0,0 +1,79 @@ +{ + "schema_version": 1, + "directories": { + "log": {"phase": "serial-before", "reason": "statement and log metadata ordering"}, + "result_count": {"phase": "serial-before", "reason": "statement result metadata ordering"}, + "sql_source_type": {"phase": "serial-before", "reason": "statement source metadata ordering"}, + "statement_query_type": {"phase": "serial-before", "reason": "statement query metadata producer"}, + "zz_statement_query_type": {"phase": "serial-before", "reason": "statement query metadata verifier"}, + + "analyze": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "auto_increment": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "benchmark": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker; keep the entire directory on one worker"}, + "charset_collation": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "comment": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "cte": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "dataXtest": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "distinct": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "dtype": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "expression": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "fake_pk": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "fulltext": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "geo": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "keyword": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "operator": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "pg_cast": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "plan_cache": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "plugin": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "procedure": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "qexec": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "recursive_cte": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "replace_statement": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "sample": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "sequence": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "time_window": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "udf": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "union": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "view": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + "window": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, + + "array": {"phase": "serial-after", "reason": "contains mo_ctl"}, + "database": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, + "ddl": {"phase": "serial-after", "reason": "contains account DDL, mo_ctl, or SET GLOBAL"}, + "disttae": {"phase": "serial-after", "reason": "contains account DDL or mo_ctl"}, + "dml": {"phase": "serial-after", "reason": "contains account DDL, mo_ctl, or global state"}, + "feature_limit": {"phase": "serial-after", "reason": "global feature registry and account state"}, + "foreign_key": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, + "function": {"phase": "serial-after", "reason": "contains mo_ctl, account DDL, or account identity functions"}, + "git4data": {"phase": "serial-after", "reason": "branch, account, debug, and global state"}, + "hint": {"phase": "serial-after", "reason": "contains account DDL, SET GLOBAL, or system metrics"}, + "iceberg": {"phase": "serial-after", "reason": "contains explicit account sessions and SET GLOBAL"}, + "join": {"phase": "serial-after", "reason": "contains mo_ctl or account DDL"}, + "load_data": {"phase": "serial-after", "reason": "contains account DDL or mo_catalog.mo_account"}, + "metadata": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, + "mo_cloud": {"phase": "serial-after", "reason": "external environment and system metrics"}, + "optimizer": {"phase": "serial-after", "reason": "contains mo_ctl or current account assumptions"}, + "pessimistic_transaction": {"phase": "serial-after", "reason": "account, restore, debug, and transaction-global state"}, + "pitr": {"phase": "serial-after", "reason": "account recovery and debug state"}, + "prepare": {"phase": "serial-after", "reason": "contains account DDL or SET GLOBAL"}, + "publication_subscription": {"phase": "serial-after", "reason": "cross-account publication state"}, + "query_result": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, + "save_query_result": {"phase": "serial-after", "reason": "contains account enumeration and DDL"}, + "security": {"phase": "serial-after", "reason": "explicit account sessions and SET GLOBAL"}, + "set": {"phase": "serial-after", "reason": "contains account DDL or SET GLOBAL"}, + "snapshot": {"phase": "serial-after", "reason": "account snapshot and restore state"}, + "sql_inject": {"phase": "serial-after", "reason": "cluster-global failpoint state"}, + "stage": {"phase": "serial-after", "reason": "contains account DDL or mo_ctl"}, + "subquery": {"phase": "serial-after", "reason": "contains mo_ctl"}, + "system": {"phase": "serial-after", "reason": "system account behavior"}, + "system_variable": {"phase": "serial-after", "reason": "global system variables and account sessions"}, + "table": {"phase": "serial-after", "reason": "cluster table, account, and system metrics state"}, + "task": {"phase": "serial-after", "reason": "background task and account state"}, + "temporary": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, + "tenant": {"phase": "serial-after", "reason": "tenant and cross-account behavior"}, + "tenxcloud_xx": {"phase": "serial-after", "reason": "external environment"}, + "util": {"phase": "serial-after", "reason": "contains mo_ctl"}, + "vector": {"phase": "serial-after", "reason": "contains mo_ctl or SET GLOBAL"}, + "zz_accesscontrol": {"phase": "serial-after", "reason": "account, password, and global access-control state"} + } +} diff --git a/scripts/test_bvt_tenant_plan.py b/scripts/test_bvt_tenant_plan.py new file mode 100644 index 0000000..5286259 --- /dev/null +++ b/scripts/test_bvt_tenant_plan.py @@ -0,0 +1,236 @@ +import json +import tempfile +import unittest +from pathlib import Path + +from scripts.bvt_tenant_plan import PolicyEntry, build_plan, load_policy, write_plan + + +class PlannerFixture(unittest.TestCase): + def setUp(self): + self.temp_dir = tempfile.TemporaryDirectory() + self.root = Path(self.temp_dir.name) + self.case_root = self.root / "cases" + self.case_root.mkdir() + + def tearDown(self): + self.temp_dir.cleanup() + + def case(self, relative_path, content="select 1;"): + path = self.case_root / relative_path + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(content) + return path.resolve() + + def selected(self, *relative_paths): + path = self.root / "selected-files.txt" + path.write_text( + "".join(f"{(self.case_root / relative).resolve()}\n" for relative in relative_paths) + ) + return path + + +class TestBuildPlan(PlannerFixture): + def test_assigns_whole_directories_and_preserves_selection(self): + self.case("alpha/a.sql", "x") + self.case("alpha/b.sql", "xx") + self.case("beta/a.sql", "xxx") + self.case("global/a.sql", "xxxx") + selected = self.selected("alpha/a.sql", "alpha/b.sql", "beta/a.sql", "global/a.sql") + policy = { + "alpha": PolicyEntry("parallel", "safe"), + "beta": PolicyEntry("parallel", "safe"), + "global": PolicyEntry("serial-after", "global state"), + } + + plan = build_plan(self.case_root, selected, policy, workers=2) + + self.assertEqual(plan["serial_after"], [str((self.case_root / "global").resolve())]) + self.assertEqual( + sorted(path for worker in plan["workers"] for path in worker["directories"]), + sorted( + [ + str((self.case_root / "alpha").resolve()), + str((self.case_root / "beta").resolve()), + ] + ), + ) + self.assertEqual(plan["selected_script_count"], 4) + self.assertEqual(plan["selected_directory_count"], 3) + + def test_rejects_partial_top_level_directory_selection(self): + self.case("alpha/a.sql") + self.case("alpha/b.sql") + selected = self.selected("alpha/a.sql") + + with self.assertRaisesRegex(ValueError, "partial directory selection.*alpha"): + build_plan( + self.case_root, + selected, + {"alpha": PolicyEntry("parallel", "safe")}, + workers=2, + ) + + def test_rejects_selected_file_outside_case_root(self): + outside = self.root / "outside.sql" + outside.write_text("select 1;") + selected = self.root / "selected-files.txt" + selected.write_text(f"{outside.resolve()}\n") + + with self.assertRaisesRegex(ValueError, "outside case root"): + build_plan(self.case_root, selected, {}, workers=2) + + def test_unknown_directory_defaults_to_unreviewed_serial_after(self): + self.case("new_suite/a.sql") + selected = self.selected("new_suite/a.sql") + + plan = build_plan(self.case_root, selected, {}, workers=2) + + self.assertEqual(plan["serial_after"], [str((self.case_root / "new_suite").resolve())]) + self.assertEqual( + plan["directories"], + [ + { + "name": "new_suite", + "path": str((self.case_root / "new_suite").resolve()), + "phase": "serial-after", + "reason": "unreviewed directory defaults to serial-after", + "reviewed": False, + "script_count": 1, + "weight_bytes": len("select 1;"), + "worker": None, + } + ], + ) + + def test_worker_count_must_be_between_one_and_four(self): + self.case("alpha/a.sql") + selected = self.selected("alpha/a.sql") + policy = {"alpha": PolicyEntry("parallel", "safe")} + + for workers in (0, 5): + with self.subTest(workers=workers): + with self.assertRaisesRegex(ValueError, "workers must be between 1 and 4"): + build_plan(self.case_root, selected, policy, workers=workers) + + def test_longest_first_balancing_is_deterministic(self): + self.case("alpha/a.sql", "a" * 10) + self.case("beta/a.sql", "b" * 7) + self.case("gamma/a.sql", "g" * 6) + selected = self.selected("alpha/a.sql", "beta/a.sql", "gamma/a.sql") + policy = { + name: PolicyEntry("parallel", "safe") for name in ("alpha", "beta", "gamma") + } + + plan = build_plan(self.case_root, selected, policy, workers=2) + + self.assertEqual( + plan["workers"], + [ + { + "index": 0, + "weight_bytes": 10, + "directories": [str((self.case_root / "alpha").resolve())], + }, + { + "index": 1, + "weight_bytes": 13, + "directories": [ + str((self.case_root / "beta").resolve()), + str((self.case_root / "gamma").resolve()), + ], + }, + ], + ) + + def test_write_plan_emits_directory_only_include_files(self): + self.case("before/a.sql") + self.case("alpha/a.sql") + self.case("after/a.test") + selected = self.selected("before/a.sql", "alpha/a.sql", "after/a.test") + policy = { + "before": PolicyEntry("serial-before", "ordered"), + "alpha": PolicyEntry("parallel", "safe"), + "after": PolicyEntry("serial-after", "global"), + } + output_dir = self.root / "plan" + + plan = build_plan(self.case_root, selected, policy, workers=2) + write_plan(plan, output_dir) + + self.assertEqual( + (output_dir / "serial-before.include").read_text().strip(), + str((self.case_root / "before").resolve()), + ) + self.assertEqual( + (output_dir / "worker-0.include").read_text().strip(), + str((self.case_root / "alpha").resolve()), + ) + self.assertEqual((output_dir / "worker-1.include").read_text(), "") + self.assertEqual( + (output_dir / "serial-after.include").read_text().strip(), + str((self.case_root / "after").resolve()), + ) + for include_file in output_dir.glob("*.include"): + content = include_file.read_text() + self.assertNotIn(".sql", content) + self.assertNotIn(".test", content) + written_plan = json.loads((output_dir / "plan.json").read_text()) + self.assertEqual(written_plan, plan) + inventory = (output_dir / "inventory.tsv").read_text().splitlines() + self.assertEqual( + inventory[0], + "directory\tphase\tworker\treviewed\tscript_count\tweight_bytes\treason", + ) + self.assertEqual(len(inventory), 4) + + +class TestLoadPolicy(PlannerFixture): + def test_rejects_duplicate_directory_keys(self): + policy_path = self.root / "policy.json" + policy_path.write_text( + """{ + "schema_version": 1, + "directories": { + "alpha": {"phase": "parallel", "reason": "one"}, + "alpha": {"phase": "serial-after", "reason": "two"} + } +} +""" + ) + + with self.assertRaisesRegex(ValueError, "duplicate JSON key: alpha"): + load_policy(policy_path) + + def test_rejects_invalid_phase(self): + policy_path = self.root / "policy.json" + policy_path.write_text( + json.dumps( + { + "schema_version": 1, + "directories": { + "alpha": {"phase": "sometimes", "reason": "invalid"} + }, + } + ) + ) + + with self.assertRaisesRegex(ValueError, "invalid phase.*sometimes"): + load_policy(policy_path) + + def test_repository_policy_has_expected_directory_counts(self): + policy = load_policy(Path(__file__).with_name("bvt_tenant_policy.json")) + counts = { + phase: sum(entry.phase == phase for entry in policy.values()) + for phase in ("serial-before", "parallel", "serial-after") + } + + self.assertEqual(len(policy), 72) + self.assertEqual( + counts, + {"serial-before": 5, "parallel": 29, "serial-after": 38}, + ) + + +if __name__ == "__main__": + unittest.main() From b4d6945cc55cfa1a5193b62ff8231f0e324e5b70 Mon Sep 17 00:00:00 2001 From: "Ariznawl@163.com" Date: Wed, 29 Jul 2026 10:32:48 +0800 Subject: [PATCH 05/16] feat: orchestrate tenant-parallel BVT phases --- scripts/run_bvt_tenant_parallel.sh | 445 ++++++++++++++++++++++++ scripts/test_run_bvt_tenant_parallel.sh | 285 +++++++++++++++ 2 files changed, 730 insertions(+) create mode 100644 scripts/run_bvt_tenant_parallel.sh create mode 100644 scripts/test_run_bvt_tenant_parallel.sh diff --git a/scripts/run_bvt_tenant_parallel.sh b/scripts/run_bvt_tenant_parallel.sh new file mode 100644 index 0000000..9870eaf --- /dev/null +++ b/scripts/run_bvt_tenant_parallel.sh @@ -0,0 +1,445 @@ +#!/usr/bin/env bash + +set -uo pipefail + +usage() { + cat >&2 <<'USAGE' +Usage: + run_bvt_tenant_parallel.sh \ + --tester-dir PATH \ + --case-root PATH \ + --group-runner PATH \ + --group 0|1 \ + --policy PATH \ + --planner PATH \ + --output-dir PATH \ + --workers 1..4 \ + [--resource-dir PATH] \ + [--mysql-host HOST] \ + [--mysql-port PORT] \ + [--mysql-user USER] \ + [--mysql-password PASSWORD] \ + [--tenant-password PASSWORD] +USAGE +} + +die() { + echo "tenant-parallel BVT: $*" >&2 + exit 2 +} + +absolute_directory() { + local path=$1 + (cd "${path}" 2>/dev/null && pwd -P) +} + +absolute_file() { + local path=$1 + local directory + directory=$(cd "$(dirname "${path}")" 2>/dev/null && pwd -P) || return 1 + printf '%s/%s\n' "${directory}" "$(basename "${path}")" +} + +tester_dir="" +case_root="" +group_runner="" +group="" +policy="" +planner="" +output_dir="" +workers="" +resource_dir="" +mysql_host="127.0.0.1" +mysql_port="6001" +mysql_user="dump" +mysql_password="111" +tenant_password="111" + +while (( $# > 0 )); do + case "$1" in + --tester-dir|--case-root|--group-runner|--group|--policy|--planner|--output-dir|--workers|--resource-dir|--mysql-host|--mysql-port|--mysql-user|--mysql-password|--tenant-password) + (( $# >= 2 )) || die "missing value for $1" + option=$1 + value=$2 + shift 2 + case "${option}" in + --tester-dir) tester_dir=${value} ;; + --case-root) case_root=${value} ;; + --group-runner) group_runner=${value} ;; + --group) group=${value} ;; + --policy) policy=${value} ;; + --planner) planner=${value} ;; + --output-dir) output_dir=${value} ;; + --workers) workers=${value} ;; + --resource-dir) resource_dir=${value} ;; + --mysql-host) mysql_host=${value} ;; + --mysql-port) mysql_port=${value} ;; + --mysql-user) mysql_user=${value} ;; + --mysql-password) mysql_password=${value} ;; + --tenant-password) tenant_password=${value} ;; + esac + ;; + -h|--help) + usage + exit 0 + ;; + *) + die "unknown argument: $1" + ;; + esac +done + +[[ -n "${tester_dir}" ]] || die "--tester-dir is required" +[[ -n "${case_root}" ]] || die "--case-root is required" +[[ -n "${group_runner}" ]] || die "--group-runner is required" +[[ -n "${group}" ]] || die "--group is required" +[[ -n "${policy}" ]] || die "--policy is required" +[[ -n "${planner}" ]] || die "--planner is required" +[[ -n "${output_dir}" ]] || die "--output-dir is required" +[[ -n "${workers}" ]] || die "--workers is required" +[[ "${group}" =~ ^[01]$ ]] || die "--group must be 0 or 1" +[[ "${workers}" =~ ^[1-4]$ ]] || die "--workers must be between 1 and 4" +[[ "${mysql_port}" =~ ^[0-9]+$ ]] || die "--mysql-port must be numeric" +[[ "${tenant_password}" =~ ^[A-Za-z0-9._-]+$ ]] || + die "--tenant-password may contain only letters, digits, dot, underscore, and hyphen" + +tester_dir=$(absolute_directory "${tester_dir}") || + die "tester directory does not exist" +case_root=$(absolute_directory "${case_root}") || + die "case root does not exist" +group_runner=$(absolute_file "${group_runner}") || + die "group runner does not exist" +policy=$(absolute_file "${policy}") || + die "policy does not exist" +planner=$(absolute_file "${planner}") || + die "planner does not exist" +[[ -f "${group_runner}" ]] || die "group runner does not exist: ${group_runner}" +[[ -f "${policy}" ]] || die "policy does not exist: ${policy}" +[[ -f "${planner}" ]] || die "planner does not exist: ${planner}" + +if [[ -z "${resource_dir}" && -d "$(dirname "${case_root}")/resources" ]]; then + resource_dir="$(dirname "${case_root}")/resources" +fi +if [[ -n "${resource_dir}" ]]; then + resource_dir=$(absolute_directory "${resource_dir}") || + die "resource directory does not exist" +fi + +if [[ -e "${output_dir}" ]]; then + [[ -d "${output_dir}" ]] || die "output path is not a directory" + if find "${output_dir}" -mindepth 1 -print -quit | grep -q .; then + die "output directory must be empty: ${output_dir}" + fi +else + mkdir -p "${output_dir}" || die "cannot create output directory" +fi +output_dir=$(absolute_directory "${output_dir}") || + die "cannot resolve output directory" + +summary_file="${output_dir}/summary.tsv" +printf 'phase\tname\tstatus\tinclude_file\tlog_file\n' > "${summary_file}" + +record_summary() { + local phase=$1 + local name=$2 + local status=$3 + local include_file=$4 + local log_file=$5 + printf '%s\t%s\t%s\t%s\t%s\n' \ + "${phase}" "${name}" "${status}" "${include_file}" "${log_file}" \ + >> "${summary_file}" +} + +mysql_exec() { + local sql=$1 + MYSQL_PWD="${mysql_password}" mysql \ + --host="${mysql_host}" \ + --port="${mysql_port}" \ + --user="${mysql_user}" \ + --batch \ + --skip-column-names \ + --execute "${sql}" +} + +declare -a created_accounts=() + +cleanup_accounts() { + (( ${#created_accounts[@]} > 0 )) || return 0 + local failed=0 + local account + for account in "${created_accounts[@]}"; do + if ! mysql_exec "DROP ACCOUNT IF EXISTS \`${account}\`;"; then + echo "failed to drop worker account ${account}" >&2 + failed=1 + fi + done + (( failed == 0 )) || return 1 + + local quoted="" + for account in "${created_accounts[@]}"; do + if [[ -n "${quoted}" ]]; then + quoted+="," + fi + quoted+="'${account}'" + done + local leak_count + leak_count=$(mysql_exec \ + "SELECT count(*) FROM mo_catalog.mo_account WHERE account_name IN (${quoted});") || + return 1 + leak_count=$(printf '%s' "${leak_count}" | tr -d '[:space:]') + [[ "${leak_count}" =~ ^[0-9]+$ ]] || { + echo "invalid worker account cleanup result: ${leak_count}" >&2 + return 1 + } + if (( leak_count != 0 )); then + echo "${leak_count} worker account(s) remain after cleanup" >&2 + return 1 + fi + created_accounts=() +} + +cleanup_on_exit() { + local status=$? + trap - EXIT + if (( ${#created_accounts[@]} > 0 )); then + cleanup_accounts || status=1 + fi + exit "${status}" +} + +trap cleanup_on_exit EXIT +trap 'exit 130' INT TERM + +capture_tester="${output_dir}/capture-tester" +mkdir -p "${capture_tester}" +cat > "${capture_tester}/run.sh" <<'CAPTURE' +#!/usr/bin/env bash +set -euo pipefail +include="" +while getopts ":p:m:t:r:i:e:s:ogfnch" opt; do + if [[ "${opt}" == "i" ]]; then + include=${OPTARG} + fi +done +[[ -n "${include}" ]] +printf '%s\n' "${include}" | tr ',' '\n' > "${BVT_CAPTURE_FILE}" +CAPTURE +chmod +x "${capture_tester}/run.sh" + +selected_files="${output_dir}/selected-files.txt" +capture_log="${output_dir}/group-capture.log" +group_command=( + bash "${group_runner}" + "${capture_tester}" + "${case_root}" + "${group}" +) +if [[ -n "${resource_dir}" ]]; then + group_command+=("${resource_dir}") +fi +if ! BVT_CAPTURE_FILE="${selected_files}" \ + "${group_command[@]}" > "${capture_log}" 2>&1; then + cat "${capture_log}" >&2 + die "failed to capture BVT group ${group}" +fi +[[ -s "${selected_files}" ]] || die "captured BVT group is empty" + +if ! python3 "${planner}" \ + --case-root "${case_root}" \ + --selected-files "${selected_files}" \ + --policy "${policy}" \ + --workers "${workers}" \ + --output-dir "${output_dir}"; then + die "failed to build BVT directory plan" +fi + +prepare_phase() { + local phase_name=$1 + local account_user=$2 + local phase_root="${output_dir}/phases/${phase_name}" + local phase_tester="${phase_root}/tester" + mkdir -p "${phase_tester}/log" "${phase_tester}/report" + + local required + for required in run.sh run.yml mo.yml log4j.properties pprof.sh; do + [[ -f "${tester_dir}/${required}" ]] || + die "mo-tester is missing ${required}" + cp -a "${tester_dir}/${required}" "${phase_tester}/${required}" + done + [[ -d "${tester_dir}/lib" ]] || die "mo-tester is missing lib/" + cp -a "${tester_dir}/lib" "${phase_tester}/lib" + if [[ -f "${tester_dir}/kafka.yml" ]]; then + cp -a "${tester_dir}/kafka.yml" "${phase_tester}/kafka.yml" + fi + chmod +x "${phase_tester}/run.sh" "${phase_tester}/pprof.sh" + + prepared_resource="" + if [[ -n "${resource_dir}" ]]; then + cp -a "${resource_dir}" "${phase_root}/resources" + prepared_resource="${phase_root}/resources" + fi + + if [[ -n "${account_user}" ]]; then + python3 - "${phase_tester}/mo.yml" "${account_user}" "${tenant_password}" <<'PY' +import sys +from pathlib import Path + +path = Path(sys.argv[1]) +name = sys.argv[2] +password = sys.argv[3] +lines = path.read_text().splitlines() +name_count = 0 +password_count = 0 +for index, line in enumerate(lines): + if line.startswith(" name:"): + lines[index] = f' name: "{name}"' + name_count += 1 + elif line.startswith(" password:"): + lines[index] = f' password: "{password}"' + password_count += 1 +if name_count != 1 or password_count != 1: + raise SystemExit( + f"expected one user name and password in {path}; " + f"found name={name_count}, password={password_count}" + ) +path.write_text("\n".join(lines) + "\n") +PY + fi + + prepared_tester="${phase_tester}" + prepared_log="${phase_root}/${phase_name}.log" +} + +run_phase() { + local phase_name=$1 + local include_file=$2 + local account_user=$3 + local include + include=$(<"${include_file}") + [[ -n "${include}" ]] || return 0 + + prepare_phase "${phase_name}" "${account_user}" + local -a tester_args=(-n -g -o -p "${case_root}" -i "${include}") + if [[ -n "${prepared_resource}" ]]; then + tester_args+=(-s "${prepared_resource}") + fi + ( + set -o pipefail + cd "${prepared_tester}" + ./run.sh "${tester_args[@]}" 2>&1 | tee "${prepared_log}" + ) +} + +serial_before_include="${output_dir}/serial-before.include" +serial_after_include="${output_dir}/serial-after.include" + +if [[ -s "${serial_before_include}" ]]; then + if run_phase "serial-before" "${serial_before_include}" ""; then + record_summary \ + "serial" "serial-before" "passed" \ + "${serial_before_include}" \ + "${output_dir}/phases/serial-before/serial-before.log" + else + record_summary \ + "serial" "serial-before" "failed" \ + "${serial_before_include}" \ + "${output_dir}/phases/serial-before/serial-before.log" + exit 1 + fi +else + record_summary "serial" "serial-before" "skipped" "${serial_before_include}" "" +fi + +declare -a active_workers=() +worker_index=0 +while (( worker_index < workers )); do + worker_include="${output_dir}/worker-${worker_index}.include" + if [[ -s "${worker_include}" ]]; then + active_workers+=("${worker_index}") + else + record_summary "parallel" "worker-${worker_index}" "skipped" "${worker_include}" "" + fi + ((worker_index+=1)) +done + +account_setup_failed=0 +for worker_index in "${active_workers[@]}"; do + account="bvtw_g${group}_w${worker_index}" + if mysql_exec \ + "CREATE ACCOUNT IF NOT EXISTS \`${account}\` ADMIN_NAME 'admin' IDENTIFIED BY '${tenant_password}';"; then + created_accounts+=("${account}") + else + echo "failed to create worker account ${account}" >&2 + account_setup_failed=1 + break + fi +done +if (( account_setup_failed != 0 )); then + record_summary "parallel" "account-setup" "failed" "" "" + exit 1 +fi + +declare -a worker_pids=() +declare -a launched_workers=() +for worker_index in "${active_workers[@]}"; do + account="bvtw_g${group}_w${worker_index}:admin" + run_phase \ + "worker-${worker_index}" \ + "${output_dir}/worker-${worker_index}.include" \ + "${account}" & + worker_pids+=("$!") + launched_workers+=("${worker_index}") +done + +final_status=0 +pid_index=0 +while (( pid_index < ${#worker_pids[@]} )); do + worker_index=${launched_workers[pid_index]} + worker_include="${output_dir}/worker-${worker_index}.include" + worker_log="${output_dir}/phases/worker-${worker_index}/worker-${worker_index}.log" + if wait "${worker_pids[pid_index]}"; then + record_summary \ + "parallel" "worker-${worker_index}" "passed" "${worker_include}" "${worker_log}" + else + record_summary \ + "parallel" "worker-${worker_index}" "failed" "${worker_include}" "${worker_log}" + final_status=1 + fi + ((pid_index+=1)) +done + +cleanup_ok=1 +if (( ${#created_accounts[@]} > 0 )); then + if cleanup_accounts; then + record_summary "cleanup" "worker-accounts" "passed" "" "" + else + record_summary "cleanup" "worker-accounts" "failed" "" "" + cleanup_ok=0 + final_status=1 + fi +fi + +if (( cleanup_ok == 0 )); then + record_summary "serial" "serial-after" "skipped" "${serial_after_include}" "" +elif ! mysql_exec "SELECT 1;" >/dev/null; then + echo "MatrixOne is unreachable; serial-after cannot run" >&2 + record_summary "serial" "serial-after" "skipped" "${serial_after_include}" "" + final_status=1 +elif [[ -s "${serial_after_include}" ]]; then + if run_phase "serial-after" "${serial_after_include}" ""; then + record_summary \ + "serial" "serial-after" "passed" \ + "${serial_after_include}" \ + "${output_dir}/phases/serial-after/serial-after.log" + else + record_summary \ + "serial" "serial-after" "failed" \ + "${serial_after_include}" \ + "${output_dir}/phases/serial-after/serial-after.log" + final_status=1 + fi +else + record_summary "serial" "serial-after" "skipped" "${serial_after_include}" "" +fi + +exit "${final_status}" diff --git a/scripts/test_run_bvt_tenant_parallel.sh b/scripts/test_run_bvt_tenant_parallel.sh new file mode 100644 index 0000000..2ce32b6 --- /dev/null +++ b/scripts/test_run_bvt_tenant_parallel.sh @@ -0,0 +1,285 @@ +#!/usr/bin/env bash + +set -euo pipefail + +script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +orchestrator="${script_dir}/run_bvt_tenant_parallel.sh" +planner="${script_dir}/bvt_tenant_plan.py" + +fail() { + echo "not ok - $*" >&2 + exit 1 +} + +assert_file() { + [[ -f "$1" ]] || fail "expected file: $1" +} + +assert_contains() { + local path=$1 + local expected=$2 + grep -F -- "${expected}" "${path}" >/dev/null || + fail "expected '${expected}' in ${path}" +} + +assert_not_contains() { + local path=$1 + local unexpected=$2 + if grep -F -- "${unexpected}" "${path}" >/dev/null; then + fail "did not expect '${unexpected}' in ${path}" + fi +} + +line_number() { + local path=$1 + local pattern=$2 + grep -n -m1 -F -- "${pattern}" "${path}" | cut -d: -f1 +} + +assert_order() { + local path=$1 + shift + local previous=0 + local pattern current + for pattern in "$@"; do + current=$(line_number "${path}" "${pattern}") + [[ -n "${current}" ]] || fail "missing event '${pattern}'" + (( current > previous )) || + fail "event '${pattern}' was out of order in ${path}" + previous=${current} + done +} + +setup_fixture() { + fixture_root=$(mktemp -d) + case_root="${fixture_root}/cases" + tester_dir="${fixture_root}/mo-tester" + resource_dir="${fixture_root}/resources" + fake_bin="${fixture_root}/bin" + event_log="${fixture_root}/events.log" + mysql_log="${fixture_root}/mysql.log" + output_dir="${fixture_root}/output" + policy="${fixture_root}/policy.json" + group_runner="${fixture_root}/run_bvt_group.sh" + + mkdir -p \ + "${case_root}/before" \ + "${case_root}/alpha" \ + "${case_root}/beta" \ + "${case_root}/after" \ + "${tester_dir}/lib" \ + "${tester_dir}/log" \ + "${tester_dir}/report" \ + "${resource_dir}/nested" \ + "${fake_bin}" + printf 'select 1;\n' > "${case_root}/before/a.sql" + printf 'select 2;\n' > "${case_root}/alpha/a.sql" + printf 'select 3;\n' > "${case_root}/beta/a.sql" + printf 'select 4;\n' > "${case_root}/after/a.test" + printf 'resource\n' > "${resource_dir}/nested/payload.txt" + : > "${tester_dir}/lib/fake.jar" + : > "${event_log}" + : > "${mysql_log}" + + printf '%s\n' \ + 'jdbc:' \ + ' server:' \ + ' - addr: "127.0.0.1:6001"' \ + 'user:' \ + ' name: "dump"' \ + ' password: "111"' \ + ' sysuser: "dump"' \ + ' syspass: "111"' > "${tester_dir}/mo.yml" + printf '%s\n' 'method: "run"' > "${tester_dir}/run.yml" + printf '%s\n' 'log4j.rootLogger=INFO' > "${tester_dir}/log4j.properties" + printf '%s\n' 'bootstrap.servers: localhost:9092' > "${tester_dir}/kafka.yml" + printf '%s\n' '#!/usr/bin/env bash' 'exit 0' > "${tester_dir}/pprof.sh" + chmod +x "${tester_dir}/pprof.sh" + + cat > "${tester_dir}/run.sh" <<'TESTER' +#!/usr/bin/env bash +set -euo pipefail +phase=$(basename "$(dirname "$PWD")") +include="" +while getopts ":p:m:t:r:i:e:s:ogfnch" opt; do + if [[ "${opt}" == "i" ]]; then + include=${OPTARG} + fi +done +user=$(sed -n 's/^ name: *"\([^"]*\)".*/\1/p' mo.yml) +printf '%s\n' "${phase}:${user}:${include}" >> "${FAKE_EVENT_LOG}" +mkdir -p report +printf '%s\n' "${phase}" > report/report.txt +if [[ "${FAKE_FAIL_PHASE:-}" == "${phase}" ]]; then + exit 7 +fi +TESTER + chmod +x "${tester_dir}/run.sh" + + cat > "${group_runner}" <<'GROUP' +#!/usr/bin/env bash +set -euo pipefail +tester_dir=$1 +case_root=$2 +"${tester_dir}/run.sh" -n -g -o -p "${case_root}" -i \ + "${case_root}/before/a.sql,${case_root}/alpha/a.sql,${case_root}/beta/a.sql,${case_root}/after/a.test" +GROUP + chmod +x "${group_runner}" + + cat > "${fake_bin}/mysql" <<'MYSQL' +#!/usr/bin/env bash +set -euo pipefail +sql="${*: -1}" +printf '%s\n' "${sql}" >> "${FAKE_MYSQL_LOG}" +lower=$(printf '%s' "${sql}" | tr '[:upper:]' '[:lower:]') +if [[ "${lower}" == *"create account"* ]]; then + printf '%s\n' "create-account" >> "${FAKE_EVENT_LOG}" +elif [[ "${lower}" == *"drop account"* ]]; then + printf '%s\n' "drop-account" >> "${FAKE_EVENT_LOG}" +elif [[ "${lower}" == *"select 1"* ]]; then + printf '%s\n' "mysql-ready" >> "${FAKE_EVENT_LOG}" +fi +if [[ "${lower}" == *"count(*)"* ]]; then + printf '%s\n' "${FAKE_LEAK_COUNT:-0}" +fi +if [[ -n "${FAKE_MYSQL_FAIL_MATCH:-}" && "${lower}" == *"${FAKE_MYSQL_FAIL_MATCH}"* ]]; then + exit 9 +fi +MYSQL + chmod +x "${fake_bin}/mysql" + + cat > "${policy}" <<'JSON' +{ + "schema_version": 1, + "directories": { + "before": {"phase": "serial-before", "reason": "ordered"}, + "alpha": {"phase": "parallel", "reason": "safe"}, + "beta": {"phase": "parallel", "reason": "safe"}, + "after": {"phase": "serial-after", "reason": "global"} + } +} +JSON +} + +run_fixture() { + PATH="${fake_bin}:${PATH}" \ + FAKE_EVENT_LOG="${event_log}" \ + FAKE_MYSQL_LOG="${mysql_log}" \ + FAKE_FAIL_PHASE="${FAKE_FAIL_PHASE:-}" \ + FAKE_LEAK_COUNT="${FAKE_LEAK_COUNT:-0}" \ + bash "${orchestrator}" \ + --tester-dir "${tester_dir}" \ + --case-root "${case_root}" \ + --group-runner "${group_runner}" \ + --group 0 \ + --policy "${policy}" \ + --planner "${planner}" \ + --output-dir "${output_dir}" \ + --workers 2 \ + --resource-dir "${resource_dir}" +} + +test_successful_phase_order_and_isolation() { + setup_fixture + + run_fixture + + assert_file "${output_dir}/plan.json" + assert_file "${output_dir}/inventory.tsv" + assert_file "${output_dir}/summary.tsv" + assert_file "${output_dir}/phases/worker-0/tester/report/report.txt" + assert_file "${output_dir}/phases/worker-1/tester/report/report.txt" + assert_file "${output_dir}/phases/worker-0/resources/nested/payload.txt" + assert_contains "${output_dir}/phases/worker-0/tester/mo.yml" \ + 'name: "bvtw_g0_w0:admin"' + assert_contains "${output_dir}/phases/worker-1/tester/mo.yml" \ + 'name: "bvtw_g0_w1:admin"' + assert_not_contains "${output_dir}/worker-0.include" ".sql" + assert_not_contains "${output_dir}/worker-1.include" ".test" + assert_order "${event_log}" \ + "serial-before:dump:" \ + "create-account" \ + "worker-" \ + "drop-account" \ + "serial-after:dump:" + assert_contains "${output_dir}/summary.tsv" $'parallel\tworker-0\tpassed' + assert_contains "${output_dir}/summary.tsv" $'parallel\tworker-1\tpassed' +} + +test_worker_failure_waits_for_sibling_and_runs_cleanup_and_after() { + setup_fixture + export FAKE_FAIL_PHASE=worker-0 + + if run_fixture; then + fail "worker failure should fail the orchestrator" + fi + + assert_contains "${event_log}" "worker-0:bvtw_g0_w0:admin:" + assert_contains "${event_log}" "worker-1:bvtw_g0_w1:admin:" + assert_contains "${event_log}" "drop-account" + assert_contains "${event_log}" "serial-after:dump:" + assert_contains "${output_dir}/summary.tsv" $'parallel\tworker-0\tfailed' + assert_contains "${output_dir}/summary.tsv" $'parallel\tworker-1\tpassed' + unset FAKE_FAIL_PHASE +} + +test_serial_before_failure_prevents_tenant_workers() { + setup_fixture + export FAKE_FAIL_PHASE=serial-before + + if run_fixture; then + fail "serial-before failure should fail the orchestrator" + fi + + assert_contains "${event_log}" "serial-before:dump:" + assert_not_contains "${event_log}" "create-account" + assert_not_contains "${event_log}" "worker-0:" + assert_not_contains "${event_log}" "worker-1:" + unset FAKE_FAIL_PHASE +} + +test_empty_parallel_phase_skips_accounts() { + setup_fixture + cat > "${policy}" <<'JSON' +{ + "schema_version": 1, + "directories": { + "before": {"phase": "serial-before", "reason": "ordered"}, + "alpha": {"phase": "serial-after", "reason": "global"}, + "beta": {"phase": "serial-after", "reason": "global"}, + "after": {"phase": "serial-after", "reason": "global"} + } +} +JSON + + run_fixture + + assert_not_contains "${event_log}" "create-account" + assert_not_contains "${event_log}" "worker-0:" + assert_not_contains "${event_log}" "worker-1:" + assert_contains "${event_log}" "serial-after:dump:" +} + +test_leaked_account_fails_before_serial_after() { + setup_fixture + export FAKE_LEAK_COUNT=1 + + if run_fixture; then + fail "leaked account should fail the orchestrator" + fi + + assert_contains "${event_log}" "drop-account" + assert_not_contains "${event_log}" "serial-after:dump:" + unset FAKE_LEAK_COUNT +} + +test_successful_phase_order_and_isolation +echo "ok - successful phase order and isolation" +test_worker_failure_waits_for_sibling_and_runs_cleanup_and_after +echo "ok - worker failure aggregation and cleanup" +test_serial_before_failure_prevents_tenant_workers +echo "ok - serial-before failure barrier" +test_empty_parallel_phase_skips_accounts +echo "ok - empty parallel phase" +test_leaked_account_fails_before_serial_after +echo "ok - leaked account detection" From 506acbaf8563c8a1500421be77cfc36f0f7e5229 Mon Sep 17 00:00:00 2001 From: "Ariznawl@163.com" Date: Wed, 29 Jul 2026 10:36:49 +0800 Subject: [PATCH 06/16] ci: add opt-in tenant-parallel BVT execution --- .github/workflows/e2e-compose-parallel.yaml | 61 ++++++- .../workflows/e2e-standalone-parallel.yaml | 64 +++++++- scripts/test_bvt_workflow_contract.py | 155 ++++++++++++++++++ 3 files changed, 266 insertions(+), 14 deletions(-) create mode 100644 scripts/test_bvt_workflow_contract.py diff --git a/.github/workflows/e2e-compose-parallel.yaml b/.github/workflows/e2e-compose-parallel.yaml index 25aa917..6a3036e 100644 --- a/.github/workflows/e2e-compose-parallel.yaml +++ b/.github/workflows/e2e-compose-parallel.yaml @@ -13,6 +13,21 @@ on: required: false type: string default: "" + tenant_parallel_enabled: + description: "Run directory-safe BVT cases in tenant workers" + required: false + type: boolean + default: false + tenant_parallel_workers: + description: "Tenant worker count (1-4)" + required: false + type: number + default: 2 + ci_ref: + description: "matrixorigin/CI ref containing tenant BVT scripts" + required: false + type: string + default: "main" secrets: TOKEN_ACTION: description: "Token for checkout (e.g. pull from fork/private)" @@ -289,6 +304,14 @@ jobs: path: ./mo-tester ref: main + - name: Checkout tenant-parallel BVT scripts + if: ${{ inputs.tenant_parallel_enabled }} + uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + with: + repository: matrixorigin/CI + ref: ${{ inputs.ci_ref }} + path: .ci/tenant-parallel + - name: Update BVT SQL Timeout run: | cd $GITHUB_WORKSPACE/mo-tester @@ -341,10 +364,22 @@ jobs: "https://raw.githubusercontent.com/matrixorigin/matrixone/7cb78aa5f88e3b98163f6b3bffb5f71586c0cbbf/optools/run_bvt_group.sh" \ --output "${bvt_runner}" fi - bash "${bvt_runner}" \ - "$GITHUB_WORKSPACE/mo-tester" \ - "$GITHUB_WORKSPACE/test/distributed/cases" \ - "${bvt_group}" 2>&1 | tee "${RUNNER_TEMP}/bvt-compose.log" + if [ '${{ inputs.tenant_parallel_enabled }}' = 'true' ]; then + bash "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/run_bvt_tenant_parallel.sh" \ + --tester-dir "$GITHUB_WORKSPACE/mo-tester" \ + --case-root "$GITHUB_WORKSPACE/test/distributed/cases" \ + --group-runner "${bvt_runner}" \ + --group "${bvt_group}" \ + --workers '${{ inputs.tenant_parallel_workers }}' \ + --policy "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/bvt_tenant_policy.json" \ + --planner "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/bvt_tenant_plan.py" \ + --output-dir "${RUNNER_TEMP}/bvt-tenant-compose" + else + bash "${bvt_runner}" \ + "$GITHUB_WORKSPACE/mo-tester" \ + "$GITHUB_WORKSPACE/test/distributed/cases" \ + "${bvt_group}" + fi 2>&1 | tee "${RUNNER_TEMP}/bvt-compose.log" - name: Summarize Compose BVT result if: ${{ always() && !cancelled() }} @@ -359,12 +394,24 @@ jobs: fi { echo '### Compose + Proxy BVT result' - grep '^Run BVT group' "${report}" | sed 's/^/- /' || true + grep -E '^(Run BVT group|BVT directory plan:)' "${report}" | sed 's/^/- /' || true echo echo '#### Top 10 slow BVT scripts' echo } | tee -a "$GITHUB_STEP_SUMMARY" + tenant_summary="${RUNNER_TEMP}/bvt-tenant-compose/summary.tsv" + if [ -s "${tenant_summary}" ]; then + { + echo '#### Tenant BVT phase results' + echo + echo '```text' + cat "${tenant_summary}" + echo '```' + echo + } | tee -a "$GITHUB_STEP_SUMMARY" + fi + sed -nE 's#.*script file\[(.*)\] has been executed, and cost: ([0-9.]+)s.*#\2s\t\1#p' "${report}" \ | sort -nr -k1,1 | awk 'NR <= 10 { print }' > "${timing}" if [ ! -s "${timing}" ]; then @@ -382,7 +429,9 @@ jobs: uses: actions/upload-artifact@v7 with: name: bvt-execution-compose - path: ${{ runner.temp }}/bvt-compose.log + path: | + ${{ runner.temp }}/bvt-compose.log + ${{ runner.temp }}/bvt-tenant-compose if-no-files-found: warn retention-days: 7 diff --git a/.github/workflows/e2e-standalone-parallel.yaml b/.github/workflows/e2e-standalone-parallel.yaml index e2aa1bd..88bd6bd 100644 --- a/.github/workflows/e2e-standalone-parallel.yaml +++ b/.github/workflows/e2e-standalone-parallel.yaml @@ -13,6 +13,21 @@ on: required: false type: string default: "" + tenant_parallel_enabled: + description: "Run directory-safe BVT cases in tenant workers" + required: false + type: boolean + default: false + tenant_parallel_workers: + description: "Tenant worker count (1-4)" + required: false + type: number + default: 2 + ci_ref: + description: "matrixorigin/CI ref containing tenant BVT scripts" + required: false + type: string + default: "main" secrets: TOKEN_ACTION: description: "Token for checkout (e.g. pull from fork/private)" @@ -302,6 +317,13 @@ jobs: repository: matrixorigin/mo-tester path: ./mo-tester ref: main + - name: Checkout tenant-parallel BVT scripts + if: ${{ inputs.tenant_parallel_enabled }} + uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + with: + repository: matrixorigin/CI + ref: ${{ inputs.ci_ref }} + path: .ci/tenant-parallel - name: Update BVT SQL Timeout run: | cd $GITHUB_WORKSPACE/mo-tester @@ -353,12 +375,24 @@ jobs: "https://raw.githubusercontent.com/matrixorigin/matrixone/7cb78aa5f88e3b98163f6b3bffb5f71586c0cbbf/optools/run_bvt_group.sh" \ --output "${bvt_runner}" fi - bash "${bvt_runner}" \ - "$GITHUB_WORKSPACE/mo-tester" \ - "$GITHUB_WORKSPACE/head/test/distributed/cases" \ - "${bvt_group}" \ - "$GITHUB_WORKSPACE/head/test/distributed/resources" \ - 2>&1 | tee "${RUNNER_TEMP}/bvt-pessimistic.log" + if [ '${{ inputs.tenant_parallel_enabled }}' = 'true' ]; then + bash "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/run_bvt_tenant_parallel.sh" \ + --tester-dir "$GITHUB_WORKSPACE/mo-tester" \ + --case-root "$GITHUB_WORKSPACE/head/test/distributed/cases" \ + --group-runner "${bvt_runner}" \ + --group "${bvt_group}" \ + --workers '${{ inputs.tenant_parallel_workers }}' \ + --policy "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/bvt_tenant_policy.json" \ + --planner "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/bvt_tenant_plan.py" \ + --output-dir "${RUNNER_TEMP}/bvt-tenant-pessimistic" \ + --resource-dir "$GITHUB_WORKSPACE/head/test/distributed/resources" + else + bash "${bvt_runner}" \ + "$GITHUB_WORKSPACE/mo-tester" \ + "$GITHUB_WORKSPACE/head/test/distributed/cases" \ + "${bvt_group}" \ + "$GITHUB_WORKSPACE/head/test/distributed/resources" + fi 2>&1 | tee "${RUNNER_TEMP}/bvt-pessimistic.log" - name: Summarize Launch + Pessimistic BVT result if: ${{ always() && !cancelled() }} @@ -373,12 +407,24 @@ jobs: fi { echo '### Launch + Pessimistic BVT result' - grep '^Run BVT group' "${report}" | sed 's/^/- /' || true + grep -E '^(Run BVT group|BVT directory plan:)' "${report}" | sed 's/^/- /' || true echo echo '#### Top 10 slow BVT scripts' echo } | tee -a "$GITHUB_STEP_SUMMARY" + tenant_summary="${RUNNER_TEMP}/bvt-tenant-pessimistic/summary.tsv" + if [ -s "${tenant_summary}" ]; then + { + echo '#### Tenant BVT phase results' + echo + echo '```text' + cat "${tenant_summary}" + echo '```' + echo + } | tee -a "$GITHUB_STEP_SUMMARY" + fi + sed -nE 's#.*script file\[(.*)\] has been executed, and cost: ([0-9.]+)s.*#\2s\t\1#p' "${report}" \ | sort -nr -k1,1 | awk 'NR <= 10 { print }' > "${timing}" if [ ! -s "${timing}" ]; then @@ -396,7 +442,9 @@ jobs: uses: actions/upload-artifact@v7 with: name: bvt-execution-pessimistic - path: ${{ runner.temp }}/bvt-pessimistic.log + path: | + ${{ runner.temp }}/bvt-pessimistic.log + ${{ runner.temp }}/bvt-tenant-pessimistic if-no-files-found: warn retention-days: 7 diff --git a/scripts/test_bvt_workflow_contract.py b/scripts/test_bvt_workflow_contract.py new file mode 100644 index 0000000..11f98f0 --- /dev/null +++ b/scripts/test_bvt_workflow_contract.py @@ -0,0 +1,155 @@ +import json +import subprocess +import unittest +from pathlib import Path + + +REPOSITORY_ROOT = Path(__file__).resolve().parents[1] + + +def load_yaml(path): + ruby = r""" +require "json" +require "yaml" +puts JSON.generate(YAML.load_file(ARGV.fetch(0))) +""" + result = subprocess.run( + ["ruby", "-e", ruby, str(path)], + check=True, + capture_output=True, + text=True, + ) + return json.loads(result.stdout) + + +class WorkflowContract: + def __init__(self, test_case, filename, job_name): + self.test_case = test_case + self.document = load_yaml(REPOSITORY_ROOT / ".github" / "workflows" / filename) + workflow_on = self.document.get("on") or self.document.get("true") + self.inputs = workflow_on["workflow_call"]["inputs"] + self.steps = self.document["jobs"][job_name]["steps"] + + def step(self, name): + matches = [step for step in self.steps if step.get("name") == name] + self.test_case.assertEqual( + len(matches), + 1, + f"expected one step named {name!r}, found {len(matches)}", + ) + return matches[0] + + def assert_inputs(self): + expected = { + "tenant_parallel_enabled": { + "description": "Run directory-safe BVT cases in tenant workers", + "required": False, + "type": "boolean", + "default": False, + }, + "tenant_parallel_workers": { + "description": "Tenant worker count (1-4)", + "required": False, + "type": "number", + "default": 2, + }, + "ci_ref": { + "description": "matrixorigin/CI ref containing tenant BVT scripts", + "required": False, + "type": "string", + "default": "main", + }, + } + for name, value in expected.items(): + self.test_case.assertIn(name, self.inputs) + self.test_case.assertEqual(self.inputs[name], value) + + def assert_ci_checkout(self): + step = self.step("Checkout tenant-parallel BVT scripts") + self.test_case.assertEqual(step["if"], "${{ inputs.tenant_parallel_enabled }}") + self.test_case.assertEqual( + step["uses"], + "actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10", + ) + self.test_case.assertEqual(step["with"]["repository"], "matrixorigin/CI") + self.test_case.assertEqual(step["with"]["ref"], "${{ inputs.ci_ref }}") + self.test_case.assertEqual(step["with"]["path"], ".ci/tenant-parallel") + + +class TestComposeWorkflow(unittest.TestCase): + def setUp(self): + self.workflow = WorkflowContract( + self, + "e2e-compose-parallel.yaml", + "multi-CN-bvt-docker-compose-proxy", + ) + + def test_inputs_and_ci_checkout(self): + self.workflow.assert_inputs() + self.workflow.assert_ci_checkout() + + def test_start_step_has_opt_in_and_fallback_branches(self): + run = self.workflow.step("Start BVT Test")["run"] + + self.assertIn( + "if [ '${{ inputs.tenant_parallel_enabled }}' = 'true' ]; then", + run, + ) + self.assertIn( + 'bash "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/run_bvt_tenant_parallel.sh"', + run, + ) + self.assertIn('--case-root "$GITHUB_WORKSPACE/test/distributed/cases"', run) + self.assertNotIn("--resource-dir", run) + self.assertIn("--workers '${{ inputs.tenant_parallel_workers }}'", run) + self.assertIn('else\n bash "${bvt_runner}"', run) + + def test_artifact_contains_tenant_output(self): + path = self.workflow.step("Upload Compose BVT execution log")["with"]["path"] + + self.assertIn("${{ runner.temp }}/bvt-compose.log", path) + self.assertIn("${{ runner.temp }}/bvt-tenant-compose", path) + + +class TestStandaloneWorkflow(unittest.TestCase): + def setUp(self): + self.workflow = WorkflowContract( + self, + "e2e-standalone-parallel.yaml", + "pessimistic-bvt-linux-x86", + ) + + def test_inputs_and_ci_checkout(self): + self.workflow.assert_inputs() + self.workflow.assert_ci_checkout() + + def test_start_step_has_opt_in_resource_and_fallback_branches(self): + run = self.workflow.step("Start BVT Test")["run"] + + self.assertIn( + "if [ '${{ inputs.tenant_parallel_enabled }}' = 'true' ]; then", + run, + ) + self.assertIn( + 'bash "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/run_bvt_tenant_parallel.sh"', + run, + ) + self.assertIn('--case-root "$GITHUB_WORKSPACE/head/test/distributed/cases"', run) + self.assertIn( + '--resource-dir "$GITHUB_WORKSPACE/head/test/distributed/resources"', + run, + ) + self.assertIn("--workers '${{ inputs.tenant_parallel_workers }}'", run) + self.assertIn('else\n bash "${bvt_runner}"', run) + + def test_artifact_contains_tenant_output(self): + path = self.workflow.step("Upload Launch + Pessimistic BVT execution log")[ + "with" + ]["path"] + + self.assertIn("${{ runner.temp }}/bvt-pessimistic.log", path) + self.assertIn("${{ runner.temp }}/bvt-tenant-pessimistic", path) + + +if __name__ == "__main__": + unittest.main() From 21e242b87a9c3ddb0b6e124605e22981680e7574 Mon Sep 17 00:00:00 2001 From: "Ariznawl@163.com" Date: Wed, 29 Jul 2026 10:40:07 +0800 Subject: [PATCH 07/16] fix: bound BVT includes to case directories --- docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md | 8 ++++---- .../specs/2026-07-28-tenant-parallel-bvt-design.md | 7 +++++++ scripts/bvt_tenant_plan.py | 5 ++++- scripts/test_bvt_tenant_plan.py | 6 +++--- 4 files changed, 18 insertions(+), 8 deletions(-) diff --git a/docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md b/docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md index 67da0cb..8bef8cc 100644 --- a/docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md +++ b/docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md @@ -200,14 +200,14 @@ Expected: all tests pass. - [ ] **Step 6: Dry-run the planner against MatrixOne main** -Capture both group selections with a fake `mo-tester/run.sh`, plan both groups, and verify the union contains 72 directories and 1,133 scripts. +Capture both group selections with a fake `mo-tester/run.sh`, plan both groups, and verify the union contains 72 directories. At the implementation-time MatrixOne main commit `129bd689b5c415fbb448eb7b413ee84b245fb938`, the union contains 1,137 scripts. Expected aggregate: ```text serial-before: 5 directories, 29 scripts -parallel: 29 directories, 215 scripts -serial-after: 38 directories, 889 scripts +parallel: 29 directories, 216 scripts +serial-after: 38 directories, 892 scripts ``` - [ ] **Step 7: Commit the planner** @@ -481,7 +481,7 @@ Use `origin/main:optools/run_bvt_group.sh` from `/Users/ariznawl/weilu/matrixone ```text 72 selected directories -1133 selected scripts +1137 selected scripts at MatrixOne 129bd689b5c415fbb448eb7b413ee84b245fb938 5 serial-before directories 29 parallel directories 38 serial-after directories diff --git a/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md b/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md index c94964e..4100b67 100644 --- a/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md +++ b/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md @@ -68,6 +68,13 @@ The directory is the smallest scheduling and policy unit. The planner never divi This changes the expected optimization ceiling: only 215 of 1,133 scripts are initially parallel candidates. The simpler commands and safer maintenance are preferred over file-level parallel coverage; the actual wall-clock benefit must be established by shadow runs. +An implementation-time rescan at MatrixOne commit +`129bd689b5c415fbb448eb7b413ee84b245fb938` found four newly added +scripts without any new top-level directory. The current total is 1,137: +29 serial-before, 216 parallel-candidate, and 892 serial-after scripts. +The directory counts and policy remain 5, 29, and 38 respectively. Runtime +planning derives script counts from the checked-out MatrixOne revision. + ## Classification policy The policy is stored as data in `scripts/bvt_tenant_policy.json`. It lists each top-level directory, its phase, and its reason. The planner emits `plan.json` and `inventory.tsv` with one record per directory. diff --git a/scripts/bvt_tenant_plan.py b/scripts/bvt_tenant_plan.py index 5167c6b..59e08bb 100644 --- a/scripts/bvt_tenant_plan.py +++ b/scripts/bvt_tenant_plan.py @@ -160,7 +160,10 @@ def build_plan( def _write_include(path: Path, directories: list[str]) -> None: - path.write_text(",".join(directories) + ("\n" if directories else "")) + bounded_directories = [f"{directory.rstrip('/')}/" for directory in directories] + path.write_text( + ",".join(bounded_directories) + ("\n" if bounded_directories else "") + ) def write_plan(plan: dict, output_dir: Path) -> None: diff --git a/scripts/test_bvt_tenant_plan.py b/scripts/test_bvt_tenant_plan.py index 5286259..e1c71bd 100644 --- a/scripts/test_bvt_tenant_plan.py +++ b/scripts/test_bvt_tenant_plan.py @@ -160,16 +160,16 @@ def test_write_plan_emits_directory_only_include_files(self): self.assertEqual( (output_dir / "serial-before.include").read_text().strip(), - str((self.case_root / "before").resolve()), + f"{(self.case_root / 'before').resolve()}/", ) self.assertEqual( (output_dir / "worker-0.include").read_text().strip(), - str((self.case_root / "alpha").resolve()), + f"{(self.case_root / 'alpha').resolve()}/", ) self.assertEqual((output_dir / "worker-1.include").read_text(), "") self.assertEqual( (output_dir / "serial-after.include").read_text().strip(), - str((self.case_root / "after").resolve()), + f"{(self.case_root / 'after').resolve()}/", ) for include_file in output_dir.glob("*.include"): content = include_file.read_text() From b418c4f89aade19b95c483cf56041549373d9acd Mon Sep 17 00:00:00 2001 From: "Ariznawl@163.com" Date: Wed, 29 Jul 2026 14:46:48 +0800 Subject: [PATCH 08/16] fix: harden tenant-parallel BVT execution --- .../plans/2026-07-29-tenant-parallel-bvt.md | 2 + .../2026-07-28-tenant-parallel-bvt-design.md | 12 +- scripts/bvt_tenant_plan.py | 65 ++++++++ scripts/run_bvt_tenant_parallel.sh | 62 +++++++- scripts/test_bvt_tenant_plan.py | 26 ++++ scripts/test_run_bvt_tenant_parallel.sh | 144 ++++++++++++++++-- 6 files changed, 290 insertions(+), 21 deletions(-) diff --git a/docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md b/docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md index 8bef8cc..a1ca2e9 100644 --- a/docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md +++ b/docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md @@ -13,6 +13,8 @@ - The smallest scheduling unit is one immediate child directory of `test/distributed/cases`; no file-level policy or worker assignment is allowed. - The selected directories are exactly the current MatrixOne group intersected with the policy; no selected script may be omitted or assigned twice. - Unknown top-level directories run in `serial-after` and are reported as unreviewed. +- A runtime serial-rule match downgrades the entire allowlisted parallel directory to + `serial-after`; the planner never splits a directory by file. - The default worker count is `2`; accepted values are `1` through `4`. - Each workflow job continues to use one runner and one MatrixOne deployment. - Worker accounts are removed before `serial-after`. diff --git a/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md b/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md index 4100b67..33c8798 100644 --- a/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md +++ b/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md @@ -79,7 +79,12 @@ planning derives script counts from the checked-out MatrixOne revision. The policy is stored as data in `scripts/bvt_tenant_policy.json`. It lists each top-level directory, its phase, and its reason. The planner emits `plan.json` and `inventory.tsv` with one record per directory. -The policy is explicit rather than reclassifying individual files at runtime. A newly added directory defaults to `serial-after` and is reported as unreviewed, so it is exercised but cannot enter a tenant worker without a policy review. +The policy is explicit and never assigns individual files. A newly added directory +defaults to `serial-after` and is reported as unreviewed. Before assigning an +allowlisted parallel directory, the planner rescans every script below it using the +serial content rules. A match downgrades the entire directory to `serial-after`, so a +new or modified case cannot silently inherit parallel status after introducing known +global or cross-account behavior. ### Serial-before directories @@ -303,11 +308,14 @@ After the gates pass, callers enable tenant parallelism. The disabled path remai - rejection of file-level policy entries; - directory union/disjoint validation; - unknown-directory serial fallback; +- runtime whole-directory downgrade when a parallel candidate matches a serial rule; - preservation of all scripts below each selected directory; - deterministic whole-directory worker balancing; - malformed policy and path rejection. -Shell integration tests use fake `mysql` and `mo-tester` commands to verify phase order, all-worker wait behavior, exit aggregation, artifacts, and cleanup. +Shell integration tests use fake `mysql` and `mo-tester` commands to verify phase +order, all-worker wait behavior, signal cancellation, exit aggregation, redacted +artifacts, and cleanup. Repository validation runs: diff --git a/scripts/bvt_tenant_plan.py b/scripts/bvt_tenant_plan.py index 59e08bb..4ff43d9 100644 --- a/scripts/bvt_tenant_plan.py +++ b/scripts/bvt_tenant_plan.py @@ -3,11 +3,56 @@ import argparse import csv import json +import re from dataclasses import dataclass from pathlib import Path VALID_PHASES = {"serial-before", "parallel", "serial-after"} +SERIAL_CONTENT_RULES = ( + ("CREATE ACCOUNT", re.compile(r"\bCREATE\s+ACCOUNT\b", re.IGNORECASE)), + ("DROP ACCOUNT", re.compile(r"\bDROP\s+ACCOUNT\b", re.IGNORECASE)), + ("ALTER ACCOUNT", re.compile(r"\bALTER\s+ACCOUNT\b", re.IGNORECASE)), + ("RESTORE ACCOUNT", re.compile(r"\bRESTORE\s+ACCOUNT\b", re.IGNORECASE)), + ("SHOW ACCOUNTS", re.compile(r"\bSHOW\s+ACCOUNTS\b", re.IGNORECASE)), + ( + "@session explicit credentials", + re.compile( + r"^\s*--\s*@session:[^\n]*(?:&user=|&password=)", + re.IGNORECASE | re.MULTILINE, + ), + ), + ("mo_ctl", re.compile(r"\bmo_ctl\s*\(", re.IGNORECASE)), + ( + "mo_feature_registry", + re.compile(r"\bmo_feature_registry_[A-Za-z0-9_]*", re.IGNORECASE), + ), + ("SET GLOBAL", re.compile(r"\bSET\s+GLOBAL\b", re.IGNORECASE)), + ( + "@system command", + re.compile(r"^\s*--\s*@system\b", re.IGNORECASE | re.MULTILINE), + ), + ( + "system metrics or debug", + re.compile(r"\b(?:system_metrics|mo_debug)\b", re.IGNORECASE), + ), + ( + "account_id = 0", + re.compile(r"\baccount_id\s*=\s*0\b", re.IGNORECASE), + ), + ( + "current account identity", + re.compile(r"\bcurrent_account_(?:id|name)\s*\(", re.IGNORECASE), + ), + ( + "mo_catalog.mo_account", + re.compile(r"\bmo_catalog\s*\.\s*mo_account\b", re.IGNORECASE), + ), + ( + "KILL CONNECTION or QUERY", + re.compile(r"\bKILL\s+(?:CONNECTION|QUERY)\b", re.IGNORECASE), + ), +) @dataclass(frozen=True) @@ -58,6 +103,16 @@ def _scripts_below(directory: Path) -> set[Path]: } +def _serial_content_match(directory: Path, scripts: set[Path]) -> str | None: + for path in sorted(scripts): + content = path.read_text(errors="replace") + for label, pattern in SERIAL_CONTENT_RULES: + if pattern.search(content): + relative = path.relative_to(directory).as_posix() + return f"{label} in {relative}" + return None + + def _selected_by_directory(case_root: Path, selected_file: Path) -> dict[str, set[Path]]: selected = {} for line_number, raw_line in enumerate(selected_file.read_text().splitlines(), 1): @@ -113,6 +168,16 @@ def build_plan( phase="serial-after", reason="unreviewed directory defaults to serial-after", ) + elif entry.phase == "parallel": + serial_match = _serial_content_match(directory, discovered) + if serial_match is not None: + entry = PolicyEntry( + phase="serial-after", + reason=( + f"runtime content scan matched {serial_match}; " + "downgraded from parallel" + ), + ) units.append( { "name": name, diff --git a/scripts/run_bvt_tenant_parallel.sh b/scripts/run_bvt_tenant_parallel.sh index 9870eaf..d926aed 100644 --- a/scripts/run_bvt_tenant_parallel.sh +++ b/scripts/run_bvt_tenant_parallel.sh @@ -162,6 +162,8 @@ mysql_exec() { } declare -a created_accounts=() +declare -a worker_pids=() +worker_pid_count=0 cleanup_accounts() { (( ${#created_accounts[@]} > 0 )) || return 0 @@ -198,12 +200,58 @@ cleanup_accounts() { created_accounts=() } +# Invoked through the EXIT trap's cleanup call graph. +# shellcheck disable=SC2329 +redact_phase_configs() { + python3 - "${output_dir}" <<'PY' +import re +import sys +from pathlib import Path + +output_dir = Path(sys.argv[1]) +secret_field = re.compile(r"^(\s*(?:password|syspass)\s*:\s*).*$", re.IGNORECASE) +for path in output_dir.glob("phases/*/tester/mo.yml"): + lines = path.read_text().splitlines() + redacted = [ + secret_field.sub(r'\1"***"', line) + for line in lines + ] + path.write_text("\n".join(redacted) + "\n") +PY +} + +# shellcheck disable=SC2329 +stop_worker_processes() { + local index pid + index=0 + while (( index < worker_pid_count )); do + pid=${worker_pids[index]} + if kill -0 "${pid}" 2>/dev/null; then + kill -TERM -- "-${pid}" 2>/dev/null || + kill -TERM "${pid}" 2>/dev/null || + true + fi + ((index+=1)) + done + index=0 + while (( index < worker_pid_count )); do + pid=${worker_pids[index]} + wait "${pid}" 2>/dev/null || true + ((index+=1)) + done + worker_pids=() + worker_pid_count=0 +} + +# shellcheck disable=SC2329 cleanup_on_exit() { local status=$? - trap - EXIT + trap - EXIT INT TERM + stop_worker_processes if (( ${#created_accounts[@]} > 0 )); then cleanup_accounts || status=1 fi + redact_phase_configs || status=1 exit "${status}" } @@ -325,7 +373,7 @@ run_phase() { fi ( set -o pipefail - cd "${prepared_tester}" + cd "${prepared_tester}" || exit 1 ./run.sh "${tester_args[@]}" 2>&1 | tee "${prepared_log}" ) } @@ -379,8 +427,10 @@ if (( account_setup_failed != 0 )); then exit 1 fi -declare -a worker_pids=() declare -a launched_workers=() +if (( ${#active_workers[@]} > 0 )); then + set -m +fi for worker_index in "${active_workers[@]}"; do account="bvtw_g${group}_w${worker_index}:admin" run_phase \ @@ -388,12 +438,14 @@ for worker_index in "${active_workers[@]}"; do "${output_dir}/worker-${worker_index}.include" \ "${account}" & worker_pids+=("$!") + ((worker_pid_count+=1)) launched_workers+=("${worker_index}") done +set +m final_status=0 pid_index=0 -while (( pid_index < ${#worker_pids[@]} )); do +while (( pid_index < worker_pid_count )); do worker_index=${launched_workers[pid_index]} worker_include="${output_dir}/worker-${worker_index}.include" worker_log="${output_dir}/phases/worker-${worker_index}/worker-${worker_index}.log" @@ -407,6 +459,8 @@ while (( pid_index < ${#worker_pids[@]} )); do fi ((pid_index+=1)) done +worker_pids=() +worker_pid_count=0 cleanup_ok=1 if (( ${#created_accounts[@]} > 0 )); then diff --git a/scripts/test_bvt_tenant_plan.py b/scripts/test_bvt_tenant_plan.py index e1c71bd..8b9533e 100644 --- a/scripts/test_bvt_tenant_plan.py +++ b/scripts/test_bvt_tenant_plan.py @@ -103,6 +103,32 @@ def test_unknown_directory_defaults_to_unreviewed_serial_after(self): ], ) + def test_parallel_directory_with_serial_content_is_downgraded_whole(self): + self.case("alpha/safe.sql", "select 1;") + self.case("alpha/global.sql", "-- new case\nCrEaTe AcCoUnT risky ADMIN_NAME 'a';") + selected = self.selected("alpha/safe.sql", "alpha/global.sql") + + plan = build_plan( + self.case_root, + selected, + {"alpha": PolicyEntry("parallel", "previously reviewed")}, + workers=2, + ) + + self.assertEqual(plan["workers"][0]["directories"], []) + self.assertEqual(plan["workers"][1]["directories"], []) + self.assertEqual( + plan["serial_after"], + [str((self.case_root / "alpha").resolve())], + ) + self.assertEqual(plan["directories"][0]["phase"], "serial-after") + self.assertEqual( + plan["directories"][0]["reason"], + "runtime content scan matched CREATE ACCOUNT in global.sql; " + "downgraded from parallel", + ) + self.assertTrue(plan["directories"][0]["reviewed"]) + def test_worker_count_must_be_between_one_and_four(self): self.case("alpha/a.sql") selected = self.selected("alpha/a.sql") diff --git a/scripts/test_run_bvt_tenant_parallel.sh b/scripts/test_run_bvt_tenant_parallel.sh index 2ce32b6..00c524a 100644 --- a/scripts/test_run_bvt_tenant_parallel.sh +++ b/scripts/test_run_bvt_tenant_parallel.sh @@ -50,6 +50,20 @@ assert_order() { done } +wait_for_event() { + local path=$1 + local expected=$2 + local attempt=0 + while (( attempt < 100 )); do + if grep -F -- "${expected}" "${path}" >/dev/null; then + return 0 + fi + sleep 0.05 + ((attempt+=1)) + done + return 1 +} + setup_fixture() { fixture_root=$(mktemp -d) case_root="${fixture_root}/cases" @@ -58,6 +72,7 @@ setup_fixture() { fake_bin="${fixture_root}/bin" event_log="${fixture_root}/events.log" mysql_log="${fixture_root}/mysql.log" + worker_pid_file="${fixture_root}/worker.pid" output_dir="${fixture_root}/output" policy="${fixture_root}/policy.json" group_runner="${fixture_root}/run_bvt_group.sh" @@ -113,6 +128,14 @@ printf '%s\n' "${phase}" > report/report.txt if [[ "${FAKE_FAIL_PHASE:-}" == "${phase}" ]]; then exit 7 fi +if [[ "${FAKE_BLOCK_PHASE:-}" == "${phase}" ]]; then + printf '%s\n' "${BASHPID}" > "${FAKE_WORKER_PID_FILE}" + printf '%s\n' "${phase}-blocked" >> "${FAKE_EVENT_LOG}" + trap 'printf "%s\n" "${phase}-stopped" >> "${FAKE_EVENT_LOG}"; exit 143' INT TERM + while true; do + sleep 1 + done +fi TESTER chmod +x "${tester_dir}/run.sh" @@ -135,6 +158,10 @@ lower=$(printf '%s' "${sql}" | tr '[:upper:]' '[:lower:]') if [[ "${lower}" == *"create account"* ]]; then printf '%s\n' "create-account" >> "${FAKE_EVENT_LOG}" elif [[ "${lower}" == *"drop account"* ]]; then + if [[ -s "${FAKE_WORKER_PID_FILE}" ]] && + kill -0 "$(<"${FAKE_WORKER_PID_FILE}")" 2>/dev/null; then + printf '%s\n' "drop-while-worker-alive" >> "${FAKE_EVENT_LOG}" + fi printf '%s\n' "drop-account" >> "${FAKE_EVENT_LOG}" elif [[ "${lower}" == *"select 1"* ]]; then printf '%s\n' "mysql-ready" >> "${FAKE_EVENT_LOG}" @@ -162,21 +189,32 @@ JSON } run_fixture() { - PATH="${fake_bin}:${PATH}" \ - FAKE_EVENT_LOG="${event_log}" \ - FAKE_MYSQL_LOG="${mysql_log}" \ - FAKE_FAIL_PHASE="${FAKE_FAIL_PHASE:-}" \ - FAKE_LEAK_COUNT="${FAKE_LEAK_COUNT:-0}" \ - bash "${orchestrator}" \ - --tester-dir "${tester_dir}" \ - --case-root "${case_root}" \ - --group-runner "${group_runner}" \ - --group 0 \ - --policy "${policy}" \ - --planner "${planner}" \ - --output-dir "${output_dir}" \ - --workers 2 \ - --resource-dir "${resource_dir}" + local -a command=( + env + "PATH=${fake_bin}:${PATH}" + "FAKE_EVENT_LOG=${event_log}" + "FAKE_MYSQL_LOG=${mysql_log}" + "FAKE_FAIL_PHASE=${FAKE_FAIL_PHASE:-}" + "FAKE_BLOCK_PHASE=${FAKE_BLOCK_PHASE:-}" + "FAKE_WORKER_PID_FILE=${worker_pid_file}" + "FAKE_LEAK_COUNT=${FAKE_LEAK_COUNT:-0}" + "FAKE_MYSQL_FAIL_MATCH=${FAKE_MYSQL_FAIL_MATCH:-}" + bash "${orchestrator}" + --tester-dir "${tester_dir}" + --case-root "${case_root}" + --group-runner "${group_runner}" + --group 0 + --policy "${policy}" + --planner "${planner}" + --output-dir "${output_dir}" + --workers 2 + --tenant-password "tenant-secret" + --resource-dir "${resource_dir}" + ) + if [[ "${RUN_FIXTURE_IN_PLACE:-0}" == "1" ]]; then + exec "${command[@]}" + fi + "${command[@]}" } test_successful_phase_order_and_isolation() { @@ -194,6 +232,12 @@ test_successful_phase_order_and_isolation() { 'name: "bvtw_g0_w0:admin"' assert_contains "${output_dir}/phases/worker-1/tester/mo.yml" \ 'name: "bvtw_g0_w1:admin"' + assert_not_contains "${output_dir}/phases/worker-0/tester/mo.yml" \ + 'password: "111"' + assert_not_contains "${output_dir}/phases/worker-0/tester/mo.yml" \ + 'password: "tenant-secret"' + assert_contains "${output_dir}/phases/worker-0/tester/mo.yml" \ + 'password: "***"' assert_not_contains "${output_dir}/worker-0.include" ".sql" assert_not_contains "${output_dir}/worker-1.include" ".test" assert_order "${event_log}" \ @@ -273,6 +317,70 @@ test_leaked_account_fails_before_serial_after() { unset FAKE_LEAK_COUNT } +test_signal_stops_workers_before_cleanup() { + setup_fixture + export FAKE_BLOCK_PHASE=worker-0 + + RUN_FIXTURE_IN_PLACE=1 run_fixture & + local orchestrator_pid=$! + if ! wait_for_event "${event_log}" "worker-0-blocked"; then + kill -TERM "${orchestrator_pid}" 2>/dev/null || true + wait "${orchestrator_pid}" 2>/dev/null || true + fail "worker did not enter blocking phase" + fi + + kill -TERM "${orchestrator_pid}" + local status=0 + wait "${orchestrator_pid}" || status=$? + + local worker_pid + worker_pid=$(<"${worker_pid_file}") + local worker_was_alive=0 + if kill -0 "${worker_pid}" 2>/dev/null; then + worker_was_alive=1 + kill -TERM "${worker_pid}" 2>/dev/null || true + fi + + unset FAKE_BLOCK_PHASE + (( status == 130 )) || fail "expected signal exit 130, got ${status}" + (( worker_was_alive == 0 )) || fail "worker remained alive after orchestrator exit" + assert_not_contains "${event_log}" "drop-while-worker-alive" + assert_order "${event_log}" \ + "worker-0-blocked" \ + "drop-account" +} + +test_account_creation_failure_prevents_workers() { + setup_fixture + export FAKE_MYSQL_FAIL_MATCH="create account" + + if run_fixture; then + fail "account creation failure should fail the orchestrator" + fi + + unset FAKE_MYSQL_FAIL_MATCH + assert_contains "${output_dir}/summary.tsv" \ + $'parallel\taccount-setup\tfailed' + assert_not_contains "${event_log}" "worker-0:" + assert_not_contains "${event_log}" "worker-1:" + assert_not_contains "${event_log}" "serial-after:dump:" +} + +test_unreachable_matrixone_skips_serial_after() { + setup_fixture + export FAKE_MYSQL_FAIL_MATCH="select 1;" + + if run_fixture; then + fail "MatrixOne readiness failure should fail the orchestrator" + fi + + unset FAKE_MYSQL_FAIL_MATCH + assert_contains "${event_log}" "drop-account" + assert_not_contains "${event_log}" "serial-after:dump:" + assert_contains "${output_dir}/summary.tsv" \ + $'serial\tserial-after\tskipped' +} + test_successful_phase_order_and_isolation echo "ok - successful phase order and isolation" test_worker_failure_waits_for_sibling_and_runs_cleanup_and_after @@ -283,3 +391,9 @@ test_empty_parallel_phase_skips_accounts echo "ok - empty parallel phase" test_leaked_account_fails_before_serial_after echo "ok - leaked account detection" +test_signal_stops_workers_before_cleanup +echo "ok - signal stops workers before cleanup" +test_account_creation_failure_prevents_workers +echo "ok - account creation failure barrier" +test_unreachable_matrixone_skips_serial_after +echo "ok - unreachable MatrixOne skips serial-after" From 26fdd58eb0fbf7596bda3a42a6a2335ebe90eaca Mon Sep 17 00:00:00 2001 From: "Ariznawl@163.com" Date: Wed, 29 Jul 2026 15:59:04 +0800 Subject: [PATCH 09/16] refactor: define BVT tenant execution by directory --- scripts/bvt_tenant_directories.sh | 172 ++++++++++++++++++++++++ scripts/test_run_bvt_tenant_parallel.sh | 58 ++++++++ 2 files changed, 230 insertions(+) create mode 100644 scripts/bvt_tenant_directories.sh diff --git a/scripts/bvt_tenant_directories.sh b/scripts/bvt_tenant_directories.sh new file mode 100644 index 0000000..4c2e617 --- /dev/null +++ b/scripts/bvt_tenant_directories.sh @@ -0,0 +1,172 @@ +#!/usr/bin/env bash + +# Top-level MatrixOne BVT case directories are the scheduling unit. This file is +# sourced by run_bvt_tenant_parallel.sh after bvt_group has been validated. + +if ! [[ "${bvt_group:-}" =~ ^[01]$ ]]; then + echo "bvt_tenant_directories.sh: bvt_group must be 0 or 1" >&2 + return 2 2>/dev/null || exit 2 +fi + +bvt_group_0_directories=( + array auto_increment benchmark dataXtest ddl disttae fake_pk function + git4data hint join keyword load_data mo_cloud optimizer pg_cast plugin + prepare procedure query_result sample save_query_result sequence snapshot + sql_inject stage system system_variable temporary tenant tenxcloud_xx + time_window union util vector view +) + +bvt_group_1_directories=( + analyze charset_collation comment cte database distinct dml dtype expression + feature_limit foreign_key fulltext geo iceberg log metadata operator + pessimistic_transaction pitr plan_cache publication_subscription qexec + recursive_cte replace_statement result_count security set sql_source_type + statement_query_type subquery table task udf window zz_accesscontrol + zz_statement_query_type +) + +bvt_serial_before_all=( + log + result_count + sql_source_type + statement_query_type + zz_statement_query_type +) + +bvt_parallel_group_0_worker_0=( + view +) + +bvt_parallel_group_0_worker_1=( + auto_increment + sequence + procedure + keyword + sample + pg_cast + plugin + time_window + union + fake_pk + dataXtest +) + +bvt_parallel_group_1_worker_0=( + dtype + expression + comment + recursive_cte + qexec + replace_statement +) + +bvt_parallel_group_1_worker_1=( + window + fulltext + operator + geo + charset_collation + distinct + udf + cte + plan_cache +) + +bvt_serial_after_all=( + analyze + array + benchmark + database + ddl + disttae + dml + feature_limit + foreign_key + function + git4data + hint + iceberg + join + load_data + metadata + mo_cloud + optimizer + pessimistic_transaction + pitr + prepare + publication_subscription + query_result + save_query_result + security + set + snapshot + sql_inject + stage + subquery + system + system_variable + table + task + temporary + tenant + tenxcloud_xx + util + vector + zz_accesscontrol +) + +bvt_excluded=( + optimistic +) + +bvt_directory_in_array() { + local expected=$1 + shift + local value + for value in "$@"; do + if [[ "${value}" == "${expected}" ]]; then + return 0 + fi + done + return 1 +} + +bvt_group_for_directory() { + local name=$1 + if bvt_directory_in_array "${name}" "${bvt_group_0_directories[@]}"; then + printf '0\n' + elif bvt_directory_in_array "${name}" "${bvt_group_1_directories[@]}"; then + printf '1\n' + else + printf '%s' "${name}" | + cksum | + awk '{ print $1 % 2 }' + fi +} + +bvt_serial_before=() +bvt_serial_after=() +bvt_worker_0=() +bvt_worker_1=() + +for bvt_directory_name in "${bvt_serial_before_all[@]}"; do + if [[ "$(bvt_group_for_directory "${bvt_directory_name}")" == "${bvt_group}" ]]; then + bvt_serial_before+=("${bvt_directory_name}") + fi +done + +for bvt_directory_name in "${bvt_serial_after_all[@]}"; do + if [[ "$(bvt_group_for_directory "${bvt_directory_name}")" == "${bvt_group}" ]]; then + bvt_serial_after+=("${bvt_directory_name}") + fi +done + +if [[ "${bvt_group}" == "0" ]]; then + bvt_worker_0=("${bvt_parallel_group_0_worker_0[@]}") + bvt_worker_1=("${bvt_parallel_group_0_worker_1[@]}") +else + bvt_worker_0=("${bvt_parallel_group_1_worker_0[@]}") + bvt_worker_1=("${bvt_parallel_group_1_worker_1[@]}") +fi + +unset bvt_directory_name diff --git a/scripts/test_run_bvt_tenant_parallel.sh b/scripts/test_run_bvt_tenant_parallel.sh index 00c524a..17e4b96 100644 --- a/scripts/test_run_bvt_tenant_parallel.sh +++ b/scripts/test_run_bvt_tenant_parallel.sh @@ -5,6 +5,7 @@ set -euo pipefail script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) orchestrator="${script_dir}/run_bvt_tenant_parallel.sh" planner="${script_dir}/bvt_tenant_plan.py" +directory_config="${script_dir}/bvt_tenant_directories.sh" fail() { echo "not ok - $*" >&2 @@ -64,6 +65,61 @@ wait_for_event() { return 1 } +array_contains() { + local expected=$1 + shift + local value + for value in "$@"; do + [[ "${value}" == "${expected}" ]] && return 0 + done + return 1 +} + +test_directory_contract() { + [[ -f "${directory_config}" ]] || + fail "expected directory contract: ${directory_config}" + + local bvt_group=0 + # shellcheck source=/dev/null + source "${directory_config}" + array_contains view "${bvt_worker_0[@]}" || + fail "group 0 worker 0 must contain view" + array_contains auto_increment "${bvt_worker_1[@]}" || + fail "group 0 worker 1 must contain auto_increment" + array_contains benchmark "${bvt_serial_after[@]}" || + fail "group 0 sys-after must contain benchmark" + ! array_contains analyze "${bvt_worker_0[@]}" || + fail "analyze must not run in an ordinary tenant" + ! array_contains benchmark "${bvt_worker_1[@]}" || + fail "benchmark must not run in an ordinary tenant" + + local -a all_selected=( + "${bvt_serial_before[@]}" + "${bvt_worker_0[@]}" + "${bvt_worker_1[@]}" + "${bvt_serial_after[@]}" + ) + local duplicate + duplicate=$( + printf '%s\n' "${all_selected[@]}" | + sort | + uniq -d | + head -n 1 + ) + [[ -z "${duplicate}" ]] || + fail "directory appears in more than one phase: ${duplicate}" + + bvt_group=1 + # shellcheck source=/dev/null + source "${directory_config}" + array_contains analyze "${bvt_serial_after[@]}" || + fail "group 1 sys-after must contain analyze" + ! array_contains analyze "${bvt_worker_0[@]}" || + fail "analyze must not run in an ordinary tenant" + ! array_contains analyze "${bvt_worker_1[@]}" || + fail "analyze must not run in an ordinary tenant" +} + setup_fixture() { fixture_root=$(mktemp -d) case_root="${fixture_root}/cases" @@ -381,6 +437,8 @@ test_unreachable_matrixone_skips_serial_after() { $'serial\tserial-after\tskipped' } +test_directory_contract +echo "ok - static directory contract" test_successful_phase_order_and_isolation echo "ok - successful phase order and isolation" test_worker_failure_waits_for_sibling_and_runs_cleanup_and_after From dc3f425f30d10d29b976a1e98d326e36513f6083 Mon Sep 17 00:00:00 2001 From: "Ariznawl@163.com" Date: Wed, 29 Jul 2026 16:03:45 +0800 Subject: [PATCH 10/16] feat: run BVT directories in two tenant processes --- scripts/bvt_tenant_plan.py | 299 ------------------------ scripts/bvt_tenant_policy.json | 79 ------- scripts/run_bvt_tenant_parallel.sh | 284 +++++++++++++--------- scripts/test_bvt_tenant_plan.py | 262 --------------------- scripts/test_run_bvt_tenant_parallel.sh | 272 +++++++++++---------- 5 files changed, 322 insertions(+), 874 deletions(-) delete mode 100644 scripts/bvt_tenant_plan.py delete mode 100644 scripts/bvt_tenant_policy.json delete mode 100644 scripts/test_bvt_tenant_plan.py diff --git a/scripts/bvt_tenant_plan.py b/scripts/bvt_tenant_plan.py deleted file mode 100644 index 4ff43d9..0000000 --- a/scripts/bvt_tenant_plan.py +++ /dev/null @@ -1,299 +0,0 @@ -#!/usr/bin/env python3 - -import argparse -import csv -import json -import re -from dataclasses import dataclass -from pathlib import Path - - -VALID_PHASES = {"serial-before", "parallel", "serial-after"} -SERIAL_CONTENT_RULES = ( - ("CREATE ACCOUNT", re.compile(r"\bCREATE\s+ACCOUNT\b", re.IGNORECASE)), - ("DROP ACCOUNT", re.compile(r"\bDROP\s+ACCOUNT\b", re.IGNORECASE)), - ("ALTER ACCOUNT", re.compile(r"\bALTER\s+ACCOUNT\b", re.IGNORECASE)), - ("RESTORE ACCOUNT", re.compile(r"\bRESTORE\s+ACCOUNT\b", re.IGNORECASE)), - ("SHOW ACCOUNTS", re.compile(r"\bSHOW\s+ACCOUNTS\b", re.IGNORECASE)), - ( - "@session explicit credentials", - re.compile( - r"^\s*--\s*@session:[^\n]*(?:&user=|&password=)", - re.IGNORECASE | re.MULTILINE, - ), - ), - ("mo_ctl", re.compile(r"\bmo_ctl\s*\(", re.IGNORECASE)), - ( - "mo_feature_registry", - re.compile(r"\bmo_feature_registry_[A-Za-z0-9_]*", re.IGNORECASE), - ), - ("SET GLOBAL", re.compile(r"\bSET\s+GLOBAL\b", re.IGNORECASE)), - ( - "@system command", - re.compile(r"^\s*--\s*@system\b", re.IGNORECASE | re.MULTILINE), - ), - ( - "system metrics or debug", - re.compile(r"\b(?:system_metrics|mo_debug)\b", re.IGNORECASE), - ), - ( - "account_id = 0", - re.compile(r"\baccount_id\s*=\s*0\b", re.IGNORECASE), - ), - ( - "current account identity", - re.compile(r"\bcurrent_account_(?:id|name)\s*\(", re.IGNORECASE), - ), - ( - "mo_catalog.mo_account", - re.compile(r"\bmo_catalog\s*\.\s*mo_account\b", re.IGNORECASE), - ), - ( - "KILL CONNECTION or QUERY", - re.compile(r"\bKILL\s+(?:CONNECTION|QUERY)\b", re.IGNORECASE), - ), -) - - -@dataclass(frozen=True) -class PolicyEntry: - phase: str - reason: str - - -def _reject_duplicate_keys(pairs): - result = {} - for key, value in pairs: - if key in result: - raise ValueError(f"duplicate JSON key: {key}") - result[key] = value - return result - - -def load_policy(path: Path) -> dict[str, PolicyEntry]: - with path.open() as policy_file: - raw = json.load(policy_file, object_pairs_hook=_reject_duplicate_keys) - if raw.get("schema_version") != 1: - raise ValueError("policy schema_version must be 1") - directories = raw.get("directories") - if not isinstance(directories, dict): - raise ValueError("policy directories must be an object") - - policy = {} - for name, value in directories.items(): - if not name or "/" in name or "\\" in name: - raise ValueError(f"invalid top-level directory name: {name}") - if not isinstance(value, dict): - raise ValueError(f"policy entry for {name} must be an object") - phase = value.get("phase") - reason = value.get("reason") - if phase not in VALID_PHASES: - raise ValueError(f"invalid phase for {name}: {phase}") - if not isinstance(reason, str) or not reason.strip(): - raise ValueError(f"policy reason for {name} must be non-empty") - policy[name] = PolicyEntry(phase=phase, reason=reason.strip()) - return policy - - -def _scripts_below(directory: Path) -> set[Path]: - return { - path.resolve() - for path in directory.rglob("*") - if path.is_file() and path.suffix in {".sql", ".test"} - } - - -def _serial_content_match(directory: Path, scripts: set[Path]) -> str | None: - for path in sorted(scripts): - content = path.read_text(errors="replace") - for label, pattern in SERIAL_CONTENT_RULES: - if pattern.search(content): - relative = path.relative_to(directory).as_posix() - return f"{label} in {relative}" - return None - - -def _selected_by_directory(case_root: Path, selected_file: Path) -> dict[str, set[Path]]: - selected = {} - for line_number, raw_line in enumerate(selected_file.read_text().splitlines(), 1): - if not raw_line.strip(): - continue - path = Path(raw_line.strip()).resolve() - try: - relative = path.relative_to(case_root) - except ValueError as error: - raise ValueError( - f"selected file outside case root at line {line_number}: {path}" - ) from error - if len(relative.parts) < 2: - raise ValueError(f"selected path is not below a top-level directory: {path}") - if not path.is_file() or path.suffix not in {".sql", ".test"}: - raise ValueError(f"selected path is not a .sql or .test file: {path}") - selected.setdefault(relative.parts[0], set()).add(path) - if not selected: - raise ValueError("selected file list is empty") - return selected - - -def build_plan( - case_root: Path, - selected_file: Path, - policy: dict[str, PolicyEntry], - workers: int, -) -> dict: - if workers < 1 or workers > 4: - raise ValueError("workers must be between 1 and 4") - case_root = case_root.resolve() - if not case_root.is_dir(): - raise ValueError(f"case root is not a directory: {case_root}") - selected_by_directory = _selected_by_directory(case_root, selected_file) - - units = [] - for name in sorted(selected_by_directory): - directory = (case_root / name).resolve() - if directory.parent != case_root or not directory.is_dir(): - raise ValueError(f"selected top-level directory is invalid: {name}") - discovered = _scripts_below(directory) - selected = selected_by_directory[name] - if selected != discovered: - missing = sorted(str(path) for path in discovered - selected) - extra = sorted(str(path) for path in selected - discovered) - detail = f"missing={missing[:3]}, extra={extra[:3]}" - raise ValueError(f"partial directory selection for {name}: {detail}") - - entry = policy.get(name) - reviewed = entry is not None - if entry is None: - entry = PolicyEntry( - phase="serial-after", - reason="unreviewed directory defaults to serial-after", - ) - elif entry.phase == "parallel": - serial_match = _serial_content_match(directory, discovered) - if serial_match is not None: - entry = PolicyEntry( - phase="serial-after", - reason=( - f"runtime content scan matched {serial_match}; " - "downgraded from parallel" - ), - ) - units.append( - { - "name": name, - "path": str(directory), - "phase": entry.phase, - "reason": entry.reason, - "reviewed": reviewed, - "script_count": len(discovered), - "weight_bytes": sum(path.stat().st_size for path in discovered), - "worker": None, - } - ) - - worker_units = [ - {"index": index, "weight_bytes": 0, "directories": []} - for index in range(workers) - ] - parallel_units = sorted( - (unit for unit in units if unit["phase"] == "parallel"), - key=lambda unit: (-unit["weight_bytes"], unit["name"]), - ) - for unit in parallel_units: - target = min( - worker_units, - key=lambda worker: (worker["weight_bytes"], worker["index"]), - ) - target["directories"].append(unit["path"]) - target["weight_bytes"] += unit["weight_bytes"] - unit["worker"] = target["index"] - - return { - "schema_version": 1, - "case_root": str(case_root), - "selected_script_count": sum(unit["script_count"] for unit in units), - "selected_directory_count": len(units), - "serial_before": [ - unit["path"] for unit in units if unit["phase"] == "serial-before" - ], - "workers": worker_units, - "serial_after": [ - unit["path"] for unit in units if unit["phase"] == "serial-after" - ], - "directories": units, - } - - -def _write_include(path: Path, directories: list[str]) -> None: - bounded_directories = [f"{directory.rstrip('/')}/" for directory in directories] - path.write_text( - ",".join(bounded_directories) + ("\n" if bounded_directories else "") - ) - - -def write_plan(plan: dict, output_dir: Path) -> None: - output_dir.mkdir(parents=True, exist_ok=True) - (output_dir / "plan.json").write_text( - json.dumps(plan, indent=2, sort_keys=True) + "\n" - ) - _write_include(output_dir / "serial-before.include", plan["serial_before"]) - for worker in plan["workers"]: - _write_include( - output_dir / f"worker-{worker['index']}.include", - worker["directories"], - ) - _write_include(output_dir / "serial-after.include", plan["serial_after"]) - - with (output_dir / "inventory.tsv").open("w", newline="") as inventory_file: - writer = csv.writer(inventory_file, delimiter="\t", lineterminator="\n") - writer.writerow( - [ - "directory", - "phase", - "worker", - "reviewed", - "script_count", - "weight_bytes", - "reason", - ] - ) - for unit in plan["directories"]: - writer.writerow( - [ - unit["name"], - unit["phase"], - "" if unit["worker"] is None else unit["worker"], - str(unit["reviewed"]).lower(), - unit["script_count"], - unit["weight_bytes"], - unit["reason"], - ] - ) - - -def main() -> None: - parser = argparse.ArgumentParser( - description="Plan tenant-parallel BVT execution by top-level case directory." - ) - parser.add_argument("--case-root", required=True, type=Path) - parser.add_argument("--selected-files", required=True, type=Path) - parser.add_argument("--policy", required=True, type=Path) - parser.add_argument("--workers", required=True, type=int) - parser.add_argument("--output-dir", required=True, type=Path) - args = parser.parse_args() - - policy = load_policy(args.policy) - plan = build_plan(args.case_root, args.selected_files, policy, args.workers) - write_plan(plan, args.output_dir) - print( - "BVT directory plan: " - f"{plan['selected_directory_count']} directories, " - f"{plan['selected_script_count']} scripts, " - f"{len(plan['serial_before'])} serial-before, " - f"{sum(len(worker['directories']) for worker in plan['workers'])} parallel, " - f"{len(plan['serial_after'])} serial-after" - ) - - -if __name__ == "__main__": - main() diff --git a/scripts/bvt_tenant_policy.json b/scripts/bvt_tenant_policy.json deleted file mode 100644 index 0b2bdf3..0000000 --- a/scripts/bvt_tenant_policy.json +++ /dev/null @@ -1,79 +0,0 @@ -{ - "schema_version": 1, - "directories": { - "log": {"phase": "serial-before", "reason": "statement and log metadata ordering"}, - "result_count": {"phase": "serial-before", "reason": "statement result metadata ordering"}, - "sql_source_type": {"phase": "serial-before", "reason": "statement source metadata ordering"}, - "statement_query_type": {"phase": "serial-before", "reason": "statement query metadata producer"}, - "zz_statement_query_type": {"phase": "serial-before", "reason": "statement query metadata verifier"}, - - "analyze": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "auto_increment": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "benchmark": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker; keep the entire directory on one worker"}, - "charset_collation": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "comment": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "cte": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "dataXtest": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "distinct": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "dtype": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "expression": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "fake_pk": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "fulltext": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "geo": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "keyword": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "operator": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "pg_cast": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "plan_cache": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "plugin": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "procedure": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "qexec": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "recursive_cte": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "replace_statement": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "sample": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "sequence": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "time_window": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "udf": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "union": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "view": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "window": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - - "array": {"phase": "serial-after", "reason": "contains mo_ctl"}, - "database": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, - "ddl": {"phase": "serial-after", "reason": "contains account DDL, mo_ctl, or SET GLOBAL"}, - "disttae": {"phase": "serial-after", "reason": "contains account DDL or mo_ctl"}, - "dml": {"phase": "serial-after", "reason": "contains account DDL, mo_ctl, or global state"}, - "feature_limit": {"phase": "serial-after", "reason": "global feature registry and account state"}, - "foreign_key": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, - "function": {"phase": "serial-after", "reason": "contains mo_ctl, account DDL, or account identity functions"}, - "git4data": {"phase": "serial-after", "reason": "branch, account, debug, and global state"}, - "hint": {"phase": "serial-after", "reason": "contains account DDL, SET GLOBAL, or system metrics"}, - "iceberg": {"phase": "serial-after", "reason": "contains explicit account sessions and SET GLOBAL"}, - "join": {"phase": "serial-after", "reason": "contains mo_ctl or account DDL"}, - "load_data": {"phase": "serial-after", "reason": "contains account DDL or mo_catalog.mo_account"}, - "metadata": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, - "mo_cloud": {"phase": "serial-after", "reason": "external environment and system metrics"}, - "optimizer": {"phase": "serial-after", "reason": "contains mo_ctl or current account assumptions"}, - "pessimistic_transaction": {"phase": "serial-after", "reason": "account, restore, debug, and transaction-global state"}, - "pitr": {"phase": "serial-after", "reason": "account recovery and debug state"}, - "prepare": {"phase": "serial-after", "reason": "contains account DDL or SET GLOBAL"}, - "publication_subscription": {"phase": "serial-after", "reason": "cross-account publication state"}, - "query_result": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, - "save_query_result": {"phase": "serial-after", "reason": "contains account enumeration and DDL"}, - "security": {"phase": "serial-after", "reason": "explicit account sessions and SET GLOBAL"}, - "set": {"phase": "serial-after", "reason": "contains account DDL or SET GLOBAL"}, - "snapshot": {"phase": "serial-after", "reason": "account snapshot and restore state"}, - "sql_inject": {"phase": "serial-after", "reason": "cluster-global failpoint state"}, - "stage": {"phase": "serial-after", "reason": "contains account DDL or mo_ctl"}, - "subquery": {"phase": "serial-after", "reason": "contains mo_ctl"}, - "system": {"phase": "serial-after", "reason": "system account behavior"}, - "system_variable": {"phase": "serial-after", "reason": "global system variables and account sessions"}, - "table": {"phase": "serial-after", "reason": "cluster table, account, and system metrics state"}, - "task": {"phase": "serial-after", "reason": "background task and account state"}, - "temporary": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, - "tenant": {"phase": "serial-after", "reason": "tenant and cross-account behavior"}, - "tenxcloud_xx": {"phase": "serial-after", "reason": "external environment"}, - "util": {"phase": "serial-after", "reason": "contains mo_ctl"}, - "vector": {"phase": "serial-after", "reason": "contains mo_ctl or SET GLOBAL"}, - "zz_accesscontrol": {"phase": "serial-after", "reason": "account, password, and global access-control state"} - } -} diff --git a/scripts/run_bvt_tenant_parallel.sh b/scripts/run_bvt_tenant_parallel.sh index d926aed..dfe5b65 100644 --- a/scripts/run_bvt_tenant_parallel.sh +++ b/scripts/run_bvt_tenant_parallel.sh @@ -8,12 +8,9 @@ Usage: run_bvt_tenant_parallel.sh \ --tester-dir PATH \ --case-root PATH \ - --group-runner PATH \ --group 0|1 \ - --policy PATH \ - --planner PATH \ + --directories PATH \ --output-dir PATH \ - --workers 1..4 \ [--resource-dir PATH] \ [--mysql-host HOST] \ [--mysql-port PORT] \ @@ -42,12 +39,9 @@ absolute_file() { tester_dir="" case_root="" -group_runner="" group="" -policy="" -planner="" +directories="" output_dir="" -workers="" resource_dir="" mysql_host="127.0.0.1" mysql_port="6001" @@ -57,7 +51,7 @@ tenant_password="111" while (( $# > 0 )); do case "$1" in - --tester-dir|--case-root|--group-runner|--group|--policy|--planner|--output-dir|--workers|--resource-dir|--mysql-host|--mysql-port|--mysql-user|--mysql-password|--tenant-password) + --tester-dir|--case-root|--group|--directories|--output-dir|--resource-dir|--mysql-host|--mysql-port|--mysql-user|--mysql-password|--tenant-password) (( $# >= 2 )) || die "missing value for $1" option=$1 value=$2 @@ -65,12 +59,9 @@ while (( $# > 0 )); do case "${option}" in --tester-dir) tester_dir=${value} ;; --case-root) case_root=${value} ;; - --group-runner) group_runner=${value} ;; --group) group=${value} ;; - --policy) policy=${value} ;; - --planner) planner=${value} ;; + --directories) directories=${value} ;; --output-dir) output_dir=${value} ;; - --workers) workers=${value} ;; --resource-dir) resource_dir=${value} ;; --mysql-host) mysql_host=${value} ;; --mysql-port) mysql_port=${value} ;; @@ -91,14 +82,10 @@ done [[ -n "${tester_dir}" ]] || die "--tester-dir is required" [[ -n "${case_root}" ]] || die "--case-root is required" -[[ -n "${group_runner}" ]] || die "--group-runner is required" [[ -n "${group}" ]] || die "--group is required" -[[ -n "${policy}" ]] || die "--policy is required" -[[ -n "${planner}" ]] || die "--planner is required" +[[ -n "${directories}" ]] || die "--directories is required" [[ -n "${output_dir}" ]] || die "--output-dir is required" -[[ -n "${workers}" ]] || die "--workers is required" [[ "${group}" =~ ^[01]$ ]] || die "--group must be 0 or 1" -[[ "${workers}" =~ ^[1-4]$ ]] || die "--workers must be between 1 and 4" [[ "${mysql_port}" =~ ^[0-9]+$ ]] || die "--mysql-port must be numeric" [[ "${tenant_password}" =~ ^[A-Za-z0-9._-]+$ ]] || die "--tenant-password may contain only letters, digits, dot, underscore, and hyphen" @@ -107,15 +94,10 @@ tester_dir=$(absolute_directory "${tester_dir}") || die "tester directory does not exist" case_root=$(absolute_directory "${case_root}") || die "case root does not exist" -group_runner=$(absolute_file "${group_runner}") || - die "group runner does not exist" -policy=$(absolute_file "${policy}") || - die "policy does not exist" -planner=$(absolute_file "${planner}") || - die "planner does not exist" -[[ -f "${group_runner}" ]] || die "group runner does not exist: ${group_runner}" -[[ -f "${policy}" ]] || die "policy does not exist: ${policy}" -[[ -f "${planner}" ]] || die "planner does not exist: ${planner}" +directories=$(absolute_file "${directories}") || + die "directory configuration does not exist" +[[ -f "${directories}" ]] || + die "directory configuration does not exist: ${directories}" if [[ -z "${resource_dir}" && -d "$(dirname "${case_root}")/resources" ]]; then resource_dir="$(dirname "${case_root}")/resources" @@ -136,6 +118,154 @@ fi output_dir=$(absolute_directory "${output_dir}") || die "cannot resolve output directory" +bvt_group=${group} +# shellcheck source=/dev/null +source "${directories}" || die "failed to load directory configuration" + +validate_directory_name() { + local name=$1 + [[ "${name}" =~ ^[A-Za-z0-9_]+$ ]] || + die "invalid case directory name: ${name}" +} + +has_test_scripts() { + local path=$1 + find "${path}" -type f \( -name '*.sql' -o -name '*.test' \) -print -quit | + grep -q . +} + +configured_directory() { + local name=$1 + bvt_directory_in_array "${name}" \ + "${bvt_serial_before_all[@]}" \ + "${bvt_parallel_group_0_worker_0[@]}" \ + "${bvt_parallel_group_0_worker_1[@]}" \ + "${bvt_parallel_group_1_worker_0[@]}" \ + "${bvt_parallel_group_1_worker_1[@]}" \ + "${bvt_serial_after_all[@]}" \ + "${bvt_excluded[@]}" +} + +filter_existing_directories() { + local name + for name in "$@"; do + validate_directory_name "${name}" + if [[ -d "${case_root}/${name}" ]] && + has_test_scripts "${case_root}/${name}"; then + printf '%s\n' "${name}" + fi + done +} + +serial_before_names=() +worker_0_names=() +worker_1_names=() +serial_after_names=() + +while IFS= read -r directory_name; do + [[ -n "${directory_name}" ]] && + serial_before_names+=("${directory_name}") +done < <(filter_existing_directories "${bvt_serial_before[@]}") + +while IFS= read -r directory_name; do + [[ -n "${directory_name}" ]] && + worker_0_names+=("${directory_name}") +done < <(filter_existing_directories "${bvt_worker_0[@]}") + +while IFS= read -r directory_name; do + [[ -n "${directory_name}" ]] && + worker_1_names+=("${directory_name}") +done < <(filter_existing_directories "${bvt_worker_1[@]}") + +while IFS= read -r directory_name; do + [[ -n "${directory_name}" ]] && + serial_after_names+=("${directory_name}") +done < <(filter_existing_directories "${bvt_serial_after[@]}") + +inventory_file="${output_dir}/inventory.tsv" +printf 'directory\tgroup\tphase\treviewed\n' > "${inventory_file}" + +for directory_name in \ + "${serial_before_names[@]}" \ + "${worker_0_names[@]}" \ + "${worker_1_names[@]}" \ + "${serial_after_names[@]}"; do + case " ${serial_before_names[*]} " in + *" ${directory_name} "*) phase="serial-before" ;; + *) + case " ${worker_0_names[*]} " in + *" ${directory_name} "*) phase="worker-0" ;; + *) + case " ${worker_1_names[*]} " in + *" ${directory_name} "*) phase="worker-1" ;; + *) phase="serial-after" ;; + esac + ;; + esac + ;; + esac + printf '%s\t%s\t%s\ttrue\n' \ + "${directory_name}" "${group}" "${phase}" >> "${inventory_file}" +done + +while IFS= read -r discovered_path; do + directory_name=$(basename "${discovered_path}") + validate_directory_name "${directory_name}" + has_test_scripts "${discovered_path}" || continue + if configured_directory "${directory_name}"; then + continue + fi + if [[ "$(bvt_group_for_directory "${directory_name}")" == "${group}" ]]; then + serial_after_names+=("${directory_name}") + printf '%s\t%s\tserial-after\tfalse\n' \ + "${directory_name}" "${group}" >> "${inventory_file}" + echo "Unreviewed BVT directory runs as sys-after: ${directory_name}" + fi +done < <( + find "${case_root}" -mindepth 1 -maxdepth 1 -type d -print | + LC_ALL=C sort +) + +write_include_file() { + local path=$1 + shift + local include_value="" + local name + for name in "$@"; do + if [[ -n "${include_value}" ]]; then + include_value+="," + fi + include_value+="${case_root}/${name}/" + done + printf '%s\n' "${include_value}" > "${path}" +} + +serial_before_include="${output_dir}/serial-before.include" +worker_0_include="${output_dir}/worker-0.include" +worker_1_include="${output_dir}/worker-1.include" +serial_after_include="${output_dir}/serial-after.include" +write_include_file "${serial_before_include}" "${serial_before_names[@]}" +write_include_file "${worker_0_include}" "${worker_0_names[@]}" +write_include_file "${worker_1_include}" "${worker_1_names[@]}" +write_include_file "${serial_after_include}" "${serial_after_names[@]}" + +selected_directory_count=$( + ( + printf '%s\n' \ + "${serial_before_names[@]}" \ + "${worker_0_names[@]}" \ + "${worker_1_names[@]}" \ + "${serial_after_names[@]}" + ) | + sed '/^$/d' | + wc -l | + tr -d '[:space:]' +) +(( selected_directory_count > 0 )) || + die "BVT group ${group} has no test directories" + +echo "BVT group ${group}: ${selected_directory_count} directories; fixed 2 tenant workers" + summary_file="${output_dir}/summary.tsv" printf 'phase\tname\tstatus\tinclude_file\tlog_file\n' > "${summary_file}" @@ -161,8 +291,8 @@ mysql_exec() { --execute "${sql}" } -declare -a created_accounts=() -declare -a worker_pids=() +created_accounts=() +worker_pids=() worker_pid_count=0 cleanup_accounts() { @@ -179,9 +309,7 @@ cleanup_accounts() { local quoted="" for account in "${created_accounts[@]}"; do - if [[ -n "${quoted}" ]]; then - quoted+="," - fi + [[ -z "${quoted}" ]] || quoted+="," quoted+="'${account}'" done local leak_count @@ -200,8 +328,6 @@ cleanup_accounts() { created_accounts=() } -# Invoked through the EXIT trap's cleanup call graph. -# shellcheck disable=SC2329 redact_phase_configs() { python3 - "${output_dir}" <<'PY' import re @@ -212,18 +338,14 @@ output_dir = Path(sys.argv[1]) secret_field = re.compile(r"^(\s*(?:password|syspass)\s*:\s*).*$", re.IGNORECASE) for path in output_dir.glob("phases/*/tester/mo.yml"): lines = path.read_text().splitlines() - redacted = [ - secret_field.sub(r'\1"***"', line) - for line in lines - ] + redacted = [secret_field.sub(r'\1"***"', line) for line in lines] path.write_text("\n".join(redacted) + "\n") PY } -# shellcheck disable=SC2329 stop_worker_processes() { - local index pid - index=0 + local index=0 + local pid while (( index < worker_pid_count )); do pid=${worker_pids[index]} if kill -0 "${pid}" 2>/dev/null; then @@ -235,15 +357,13 @@ stop_worker_processes() { done index=0 while (( index < worker_pid_count )); do - pid=${worker_pids[index]} - wait "${pid}" 2>/dev/null || true + wait "${worker_pids[index]}" 2>/dev/null || true ((index+=1)) done worker_pids=() worker_pid_count=0 } -# shellcheck disable=SC2329 cleanup_on_exit() { local status=$? trap - EXIT INT TERM @@ -258,49 +378,6 @@ cleanup_on_exit() { trap cleanup_on_exit EXIT trap 'exit 130' INT TERM -capture_tester="${output_dir}/capture-tester" -mkdir -p "${capture_tester}" -cat > "${capture_tester}/run.sh" <<'CAPTURE' -#!/usr/bin/env bash -set -euo pipefail -include="" -while getopts ":p:m:t:r:i:e:s:ogfnch" opt; do - if [[ "${opt}" == "i" ]]; then - include=${OPTARG} - fi -done -[[ -n "${include}" ]] -printf '%s\n' "${include}" | tr ',' '\n' > "${BVT_CAPTURE_FILE}" -CAPTURE -chmod +x "${capture_tester}/run.sh" - -selected_files="${output_dir}/selected-files.txt" -capture_log="${output_dir}/group-capture.log" -group_command=( - bash "${group_runner}" - "${capture_tester}" - "${case_root}" - "${group}" -) -if [[ -n "${resource_dir}" ]]; then - group_command+=("${resource_dir}") -fi -if ! BVT_CAPTURE_FILE="${selected_files}" \ - "${group_command[@]}" > "${capture_log}" 2>&1; then - cat "${capture_log}" >&2 - die "failed to capture BVT group ${group}" -fi -[[ -s "${selected_files}" ]] || die "captured BVT group is empty" - -if ! python3 "${planner}" \ - --case-root "${case_root}" \ - --selected-files "${selected_files}" \ - --policy "${policy}" \ - --workers "${workers}" \ - --output-dir "${output_dir}"; then - die "failed to build BVT directory plan" -fi - prepare_phase() { local phase_name=$1 local account_user=$2 @@ -378,10 +455,8 @@ run_phase() { ) } -serial_before_include="${output_dir}/serial-before.include" -serial_after_include="${output_dir}/serial-after.include" - -if [[ -s "${serial_before_include}" ]]; then +if [[ -s "${serial_before_include}" ]] && + [[ -n "$(<"${serial_before_include}")" ]]; then if run_phase "serial-before" "${serial_before_include}" ""; then record_summary \ "serial" "serial-before" "passed" \ @@ -398,19 +473,17 @@ else record_summary "serial" "serial-before" "skipped" "${serial_before_include}" "" fi -declare -a active_workers=() -worker_index=0 -while (( worker_index < workers )); do +active_workers=() +for worker_index in 0 1; do worker_include="${output_dir}/worker-${worker_index}.include" - if [[ -s "${worker_include}" ]]; then + if [[ -n "$(<"${worker_include}")" ]]; then active_workers+=("${worker_index}") else - record_summary "parallel" "worker-${worker_index}" "skipped" "${worker_include}" "" + record_summary \ + "parallel" "worker-${worker_index}" "skipped" "${worker_include}" "" fi - ((worker_index+=1)) done -account_setup_failed=0 for worker_index in "${active_workers[@]}"; do account="bvtw_g${group}_w${worker_index}" if mysql_exec \ @@ -418,25 +491,20 @@ for worker_index in "${active_workers[@]}"; do created_accounts+=("${account}") else echo "failed to create worker account ${account}" >&2 - account_setup_failed=1 - break + record_summary "parallel" "account-setup" "failed" "" "" + exit 1 fi done -if (( account_setup_failed != 0 )); then - record_summary "parallel" "account-setup" "failed" "" "" - exit 1 -fi -declare -a launched_workers=() +launched_workers=() if (( ${#active_workers[@]} > 0 )); then set -m fi for worker_index in "${active_workers[@]}"; do - account="bvtw_g${group}_w${worker_index}:admin" run_phase \ "worker-${worker_index}" \ "${output_dir}/worker-${worker_index}.include" \ - "${account}" & + "bvtw_g${group}_w${worker_index}:admin" & worker_pids+=("$!") ((worker_pid_count+=1)) launched_workers+=("${worker_index}") @@ -479,7 +547,7 @@ elif ! mysql_exec "SELECT 1;" >/dev/null; then echo "MatrixOne is unreachable; serial-after cannot run" >&2 record_summary "serial" "serial-after" "skipped" "${serial_after_include}" "" final_status=1 -elif [[ -s "${serial_after_include}" ]]; then +elif [[ -n "$(<"${serial_after_include}")" ]]; then if run_phase "serial-after" "${serial_after_include}" ""; then record_summary \ "serial" "serial-after" "passed" \ diff --git a/scripts/test_bvt_tenant_plan.py b/scripts/test_bvt_tenant_plan.py deleted file mode 100644 index 8b9533e..0000000 --- a/scripts/test_bvt_tenant_plan.py +++ /dev/null @@ -1,262 +0,0 @@ -import json -import tempfile -import unittest -from pathlib import Path - -from scripts.bvt_tenant_plan import PolicyEntry, build_plan, load_policy, write_plan - - -class PlannerFixture(unittest.TestCase): - def setUp(self): - self.temp_dir = tempfile.TemporaryDirectory() - self.root = Path(self.temp_dir.name) - self.case_root = self.root / "cases" - self.case_root.mkdir() - - def tearDown(self): - self.temp_dir.cleanup() - - def case(self, relative_path, content="select 1;"): - path = self.case_root / relative_path - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(content) - return path.resolve() - - def selected(self, *relative_paths): - path = self.root / "selected-files.txt" - path.write_text( - "".join(f"{(self.case_root / relative).resolve()}\n" for relative in relative_paths) - ) - return path - - -class TestBuildPlan(PlannerFixture): - def test_assigns_whole_directories_and_preserves_selection(self): - self.case("alpha/a.sql", "x") - self.case("alpha/b.sql", "xx") - self.case("beta/a.sql", "xxx") - self.case("global/a.sql", "xxxx") - selected = self.selected("alpha/a.sql", "alpha/b.sql", "beta/a.sql", "global/a.sql") - policy = { - "alpha": PolicyEntry("parallel", "safe"), - "beta": PolicyEntry("parallel", "safe"), - "global": PolicyEntry("serial-after", "global state"), - } - - plan = build_plan(self.case_root, selected, policy, workers=2) - - self.assertEqual(plan["serial_after"], [str((self.case_root / "global").resolve())]) - self.assertEqual( - sorted(path for worker in plan["workers"] for path in worker["directories"]), - sorted( - [ - str((self.case_root / "alpha").resolve()), - str((self.case_root / "beta").resolve()), - ] - ), - ) - self.assertEqual(plan["selected_script_count"], 4) - self.assertEqual(plan["selected_directory_count"], 3) - - def test_rejects_partial_top_level_directory_selection(self): - self.case("alpha/a.sql") - self.case("alpha/b.sql") - selected = self.selected("alpha/a.sql") - - with self.assertRaisesRegex(ValueError, "partial directory selection.*alpha"): - build_plan( - self.case_root, - selected, - {"alpha": PolicyEntry("parallel", "safe")}, - workers=2, - ) - - def test_rejects_selected_file_outside_case_root(self): - outside = self.root / "outside.sql" - outside.write_text("select 1;") - selected = self.root / "selected-files.txt" - selected.write_text(f"{outside.resolve()}\n") - - with self.assertRaisesRegex(ValueError, "outside case root"): - build_plan(self.case_root, selected, {}, workers=2) - - def test_unknown_directory_defaults_to_unreviewed_serial_after(self): - self.case("new_suite/a.sql") - selected = self.selected("new_suite/a.sql") - - plan = build_plan(self.case_root, selected, {}, workers=2) - - self.assertEqual(plan["serial_after"], [str((self.case_root / "new_suite").resolve())]) - self.assertEqual( - plan["directories"], - [ - { - "name": "new_suite", - "path": str((self.case_root / "new_suite").resolve()), - "phase": "serial-after", - "reason": "unreviewed directory defaults to serial-after", - "reviewed": False, - "script_count": 1, - "weight_bytes": len("select 1;"), - "worker": None, - } - ], - ) - - def test_parallel_directory_with_serial_content_is_downgraded_whole(self): - self.case("alpha/safe.sql", "select 1;") - self.case("alpha/global.sql", "-- new case\nCrEaTe AcCoUnT risky ADMIN_NAME 'a';") - selected = self.selected("alpha/safe.sql", "alpha/global.sql") - - plan = build_plan( - self.case_root, - selected, - {"alpha": PolicyEntry("parallel", "previously reviewed")}, - workers=2, - ) - - self.assertEqual(plan["workers"][0]["directories"], []) - self.assertEqual(plan["workers"][1]["directories"], []) - self.assertEqual( - plan["serial_after"], - [str((self.case_root / "alpha").resolve())], - ) - self.assertEqual(plan["directories"][0]["phase"], "serial-after") - self.assertEqual( - plan["directories"][0]["reason"], - "runtime content scan matched CREATE ACCOUNT in global.sql; " - "downgraded from parallel", - ) - self.assertTrue(plan["directories"][0]["reviewed"]) - - def test_worker_count_must_be_between_one_and_four(self): - self.case("alpha/a.sql") - selected = self.selected("alpha/a.sql") - policy = {"alpha": PolicyEntry("parallel", "safe")} - - for workers in (0, 5): - with self.subTest(workers=workers): - with self.assertRaisesRegex(ValueError, "workers must be between 1 and 4"): - build_plan(self.case_root, selected, policy, workers=workers) - - def test_longest_first_balancing_is_deterministic(self): - self.case("alpha/a.sql", "a" * 10) - self.case("beta/a.sql", "b" * 7) - self.case("gamma/a.sql", "g" * 6) - selected = self.selected("alpha/a.sql", "beta/a.sql", "gamma/a.sql") - policy = { - name: PolicyEntry("parallel", "safe") for name in ("alpha", "beta", "gamma") - } - - plan = build_plan(self.case_root, selected, policy, workers=2) - - self.assertEqual( - plan["workers"], - [ - { - "index": 0, - "weight_bytes": 10, - "directories": [str((self.case_root / "alpha").resolve())], - }, - { - "index": 1, - "weight_bytes": 13, - "directories": [ - str((self.case_root / "beta").resolve()), - str((self.case_root / "gamma").resolve()), - ], - }, - ], - ) - - def test_write_plan_emits_directory_only_include_files(self): - self.case("before/a.sql") - self.case("alpha/a.sql") - self.case("after/a.test") - selected = self.selected("before/a.sql", "alpha/a.sql", "after/a.test") - policy = { - "before": PolicyEntry("serial-before", "ordered"), - "alpha": PolicyEntry("parallel", "safe"), - "after": PolicyEntry("serial-after", "global"), - } - output_dir = self.root / "plan" - - plan = build_plan(self.case_root, selected, policy, workers=2) - write_plan(plan, output_dir) - - self.assertEqual( - (output_dir / "serial-before.include").read_text().strip(), - f"{(self.case_root / 'before').resolve()}/", - ) - self.assertEqual( - (output_dir / "worker-0.include").read_text().strip(), - f"{(self.case_root / 'alpha').resolve()}/", - ) - self.assertEqual((output_dir / "worker-1.include").read_text(), "") - self.assertEqual( - (output_dir / "serial-after.include").read_text().strip(), - f"{(self.case_root / 'after').resolve()}/", - ) - for include_file in output_dir.glob("*.include"): - content = include_file.read_text() - self.assertNotIn(".sql", content) - self.assertNotIn(".test", content) - written_plan = json.loads((output_dir / "plan.json").read_text()) - self.assertEqual(written_plan, plan) - inventory = (output_dir / "inventory.tsv").read_text().splitlines() - self.assertEqual( - inventory[0], - "directory\tphase\tworker\treviewed\tscript_count\tweight_bytes\treason", - ) - self.assertEqual(len(inventory), 4) - - -class TestLoadPolicy(PlannerFixture): - def test_rejects_duplicate_directory_keys(self): - policy_path = self.root / "policy.json" - policy_path.write_text( - """{ - "schema_version": 1, - "directories": { - "alpha": {"phase": "parallel", "reason": "one"}, - "alpha": {"phase": "serial-after", "reason": "two"} - } -} -""" - ) - - with self.assertRaisesRegex(ValueError, "duplicate JSON key: alpha"): - load_policy(policy_path) - - def test_rejects_invalid_phase(self): - policy_path = self.root / "policy.json" - policy_path.write_text( - json.dumps( - { - "schema_version": 1, - "directories": { - "alpha": {"phase": "sometimes", "reason": "invalid"} - }, - } - ) - ) - - with self.assertRaisesRegex(ValueError, "invalid phase.*sometimes"): - load_policy(policy_path) - - def test_repository_policy_has_expected_directory_counts(self): - policy = load_policy(Path(__file__).with_name("bvt_tenant_policy.json")) - counts = { - phase: sum(entry.phase == phase for entry in policy.values()) - for phase in ("serial-before", "parallel", "serial-after") - } - - self.assertEqual(len(policy), 72) - self.assertEqual( - counts, - {"serial-before": 5, "parallel": 29, "serial-after": 38}, - ) - - -if __name__ == "__main__": - unittest.main() diff --git a/scripts/test_run_bvt_tenant_parallel.sh b/scripts/test_run_bvt_tenant_parallel.sh index 17e4b96..726c9a5 100644 --- a/scripts/test_run_bvt_tenant_parallel.sh +++ b/scripts/test_run_bvt_tenant_parallel.sh @@ -4,7 +4,6 @@ set -euo pipefail script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) orchestrator="${script_dir}/run_bvt_tenant_parallel.sh" -planner="${script_dir}/bvt_tenant_plan.py" directory_config="${script_dir}/bvt_tenant_directories.sh" fail() { @@ -76,9 +75,6 @@ array_contains() { } test_directory_contract() { - [[ -f "${directory_config}" ]] || - fail "expected directory contract: ${directory_config}" - local bvt_group=0 # shellcheck source=/dev/null source "${directory_config}" @@ -88,26 +84,21 @@ test_directory_contract() { fail "group 0 worker 1 must contain auto_increment" array_contains benchmark "${bvt_serial_after[@]}" || fail "group 0 sys-after must contain benchmark" - ! array_contains analyze "${bvt_worker_0[@]}" || - fail "analyze must not run in an ordinary tenant" + ! array_contains benchmark "${bvt_worker_0[@]}" || + fail "benchmark must not run in an ordinary tenant" ! array_contains benchmark "${bvt_worker_1[@]}" || fail "benchmark must not run in an ordinary tenant" - local -a all_selected=( + local -a group_0_selected=( "${bvt_serial_before[@]}" "${bvt_worker_0[@]}" "${bvt_worker_1[@]}" "${bvt_serial_after[@]}" ) local duplicate - duplicate=$( - printf '%s\n' "${all_selected[@]}" | - sort | - uniq -d | - head -n 1 - ) + duplicate=$(printf '%s\n' "${group_0_selected[@]}" | sort | uniq -d | head -n 1) [[ -z "${duplicate}" ]] || - fail "directory appears in more than one phase: ${duplicate}" + fail "group 0 directory appears in more than one phase: ${duplicate}" bvt_group=1 # shellcheck source=/dev/null @@ -118,6 +109,16 @@ test_directory_contract() { fail "analyze must not run in an ordinary tenant" ! array_contains analyze "${bvt_worker_1[@]}" || fail "analyze must not run in an ordinary tenant" + + local -a group_1_selected=( + "${bvt_serial_before[@]}" + "${bvt_worker_0[@]}" + "${bvt_worker_1[@]}" + "${bvt_serial_after[@]}" + ) + duplicate=$(printf '%s\n' "${group_1_selected[@]}" | sort | uniq -d | head -n 1) + [[ -z "${duplicate}" ]] || + fail "group 1 directory appears in more than one phase: ${duplicate}" } setup_fixture() { @@ -130,23 +131,43 @@ setup_fixture() { mysql_log="${fixture_root}/mysql.log" worker_pid_file="${fixture_root}/worker.pid" output_dir="${fixture_root}/output" - policy="${fixture_root}/policy.json" - group_runner="${fixture_root}/run_bvt_group.sh" + + unknown_directory="unknown_bvt_case" + local unknown_group + unknown_group=$( + printf '%s' "${unknown_directory}" | + cksum | + awk '{ print $1 % 2 }' + ) + if [[ "${unknown_group}" != "1" ]]; then + unknown_directory="unknown_bvt_suite" + fi + unknown_group=$( + printf '%s' "${unknown_directory}" | + cksum | + awk '{ print $1 % 2 }' + ) + [[ "${unknown_group}" == "1" ]] || + fail "test fixture needs an unknown directory assigned to group 1" mkdir -p \ - "${case_root}/before" \ - "${case_root}/alpha" \ - "${case_root}/beta" \ - "${case_root}/after" \ + "${case_root}/log" \ + "${case_root}/dtype" \ + "${case_root}/window" \ + "${case_root}/analyze" \ + "${case_root}/${unknown_directory}" \ + "${case_root}/optimistic" \ "${tester_dir}/lib" \ "${tester_dir}/log" \ "${tester_dir}/report" \ "${resource_dir}/nested" \ "${fake_bin}" - printf 'select 1;\n' > "${case_root}/before/a.sql" - printf 'select 2;\n' > "${case_root}/alpha/a.sql" - printf 'select 3;\n' > "${case_root}/beta/a.sql" - printf 'select 4;\n' > "${case_root}/after/a.test" + printf 'select 1;\n' > "${case_root}/log/a.sql" + printf 'select 2;\n' > "${case_root}/dtype/a.sql" + printf 'select 3;\n' > "${case_root}/window/a.sql" + printf 'select 4;\n' > "${case_root}/analyze/a.test" + printf 'select 5;\n' > "${case_root}/${unknown_directory}/a.sql" + printf 'select 6;\n' > "${case_root}/optimistic/a.sql" printf 'resource\n' > "${resource_dir}/nested/payload.txt" : > "${tester_dir}/lib/fake.jar" : > "${event_log}" @@ -172,13 +193,46 @@ setup_fixture() { set -euo pipefail phase=$(basename "$(dirname "$PWD")") include="" -while getopts ":p:m:t:r:i:e:s:ogfnch" opt; do - if [[ "${opt}" == "i" ]]; then - include=${OPTARG} - fi +case_path="" +declare -a received=() +while (( $# > 0 )); do + received+=("$1") + case "$1" in + -i) + received+=("$2") + include=$2 + shift 2 + ;; + -p) + received+=("$2") + case_path=$2 + shift 2 + ;; + -s) + received+=("$2") + shift 2 + ;; + *) + shift + ;; + esac done user=$(sed -n 's/^ name: *"\([^"]*\)".*/\1/p' mo.yml) -printf '%s\n' "${phase}:${user}:${include}" >> "${FAKE_EVENT_LOG}" +printf '%s\n' "${phase}:start:${user}:${case_path}:${include}" >> "${FAKE_EVENT_LOG}" +printf '%s\n' "${received[*]}" > received-arguments.txt + +if [[ "${FAKE_REQUIRE_CONCURRENCY:-0}" == "1" && "${phase}" == "worker-0" ]]; then + attempt=0 + while (( attempt < 100 )); do + if grep -F "worker-1:start:" "${FAKE_EVENT_LOG}" >/dev/null; then + break + fi + sleep 0.02 + ((attempt+=1)) + done + (( attempt < 100 )) || exit 19 +fi + mkdir -p report printf '%s\n' "${phase}" > report/report.txt if [[ "${FAKE_FAIL_PHASE:-}" == "${phase}" ]]; then @@ -186,25 +240,16 @@ if [[ "${FAKE_FAIL_PHASE:-}" == "${phase}" ]]; then fi if [[ "${FAKE_BLOCK_PHASE:-}" == "${phase}" ]]; then printf '%s\n' "${BASHPID}" > "${FAKE_WORKER_PID_FILE}" - printf '%s\n' "${phase}-blocked" >> "${FAKE_EVENT_LOG}" - trap 'printf "%s\n" "${phase}-stopped" >> "${FAKE_EVENT_LOG}"; exit 143' INT TERM + printf '%s\n' "${phase}:blocked" >> "${FAKE_EVENT_LOG}" + trap 'printf "%s\n" "${phase}:stopped" >> "${FAKE_EVENT_LOG}"; exit 143' INT TERM while true; do sleep 1 done fi +printf '%s\n' "${phase}:done" >> "${FAKE_EVENT_LOG}" TESTER chmod +x "${tester_dir}/run.sh" - cat > "${group_runner}" <<'GROUP' -#!/usr/bin/env bash -set -euo pipefail -tester_dir=$1 -case_root=$2 -"${tester_dir}/run.sh" -n -g -o -p "${case_root}" -i \ - "${case_root}/before/a.sql,${case_root}/alpha/a.sql,${case_root}/beta/a.sql,${case_root}/after/a.test" -GROUP - chmod +x "${group_runner}" - cat > "${fake_bin}/mysql" <<'MYSQL' #!/usr/bin/env bash set -euo pipefail @@ -230,18 +275,6 @@ if [[ -n "${FAKE_MYSQL_FAIL_MATCH:-}" && "${lower}" == *"${FAKE_MYSQL_FAIL_MATCH fi MYSQL chmod +x "${fake_bin}/mysql" - - cat > "${policy}" <<'JSON' -{ - "schema_version": 1, - "directories": { - "before": {"phase": "serial-before", "reason": "ordered"}, - "alpha": {"phase": "parallel", "reason": "safe"}, - "beta": {"phase": "parallel", "reason": "safe"}, - "after": {"phase": "serial-after", "reason": "global"} - } -} -JSON } run_fixture() { @@ -252,18 +285,16 @@ run_fixture() { "FAKE_MYSQL_LOG=${mysql_log}" "FAKE_FAIL_PHASE=${FAKE_FAIL_PHASE:-}" "FAKE_BLOCK_PHASE=${FAKE_BLOCK_PHASE:-}" + "FAKE_REQUIRE_CONCURRENCY=${FAKE_REQUIRE_CONCURRENCY:-0}" "FAKE_WORKER_PID_FILE=${worker_pid_file}" "FAKE_LEAK_COUNT=${FAKE_LEAK_COUNT:-0}" "FAKE_MYSQL_FAIL_MATCH=${FAKE_MYSQL_FAIL_MATCH:-}" bash "${orchestrator}" --tester-dir "${tester_dir}" --case-root "${case_root}" - --group-runner "${group_runner}" - --group 0 - --policy "${policy}" - --planner "${planner}" + --group 1 + --directories "${directory_config}" --output-dir "${output_dir}" - --workers 2 --tenant-password "tenant-secret" --resource-dir "${resource_dir}" ) @@ -273,37 +304,54 @@ run_fixture() { "${command[@]}" } -test_successful_phase_order_and_isolation() { +test_successful_direct_commands_and_isolation() { setup_fixture + export FAKE_REQUIRE_CONCURRENCY=1 run_fixture - assert_file "${output_dir}/plan.json" - assert_file "${output_dir}/inventory.tsv" assert_file "${output_dir}/summary.tsv" + assert_file "${output_dir}/serial-before.include" + assert_file "${output_dir}/worker-0.include" + assert_file "${output_dir}/worker-1.include" + assert_file "${output_dir}/serial-after.include" assert_file "${output_dir}/phases/worker-0/tester/report/report.txt" assert_file "${output_dir}/phases/worker-1/tester/report/report.txt" assert_file "${output_dir}/phases/worker-0/resources/nested/payload.txt" + assert_contains "${output_dir}/worker-0.include" "${case_root}/dtype/" + assert_contains "${output_dir}/worker-1.include" "${case_root}/window/" + assert_contains "${output_dir}/serial-after.include" "${case_root}/analyze/" + assert_contains "${output_dir}/serial-after.include" \ + "${case_root}/${unknown_directory}/" + assert_not_contains "${output_dir}/serial-after.include" "/optimistic/" + assert_not_contains "${output_dir}/worker-0.include" ".sql" + assert_not_contains "${output_dir}/worker-1.include" ".test" + local resolved_case_root + resolved_case_root=$(cd "${case_root}" && pwd -P) + assert_contains \ + "${output_dir}/phases/worker-0/tester/received-arguments.txt" \ + "-n -g -o -p ${resolved_case_root} -i ${resolved_case_root}/dtype/" assert_contains "${output_dir}/phases/worker-0/tester/mo.yml" \ - 'name: "bvtw_g0_w0:admin"' + 'name: "bvtw_g1_w0:admin"' assert_contains "${output_dir}/phases/worker-1/tester/mo.yml" \ - 'name: "bvtw_g0_w1:admin"' + 'name: "bvtw_g1_w1:admin"' assert_not_contains "${output_dir}/phases/worker-0/tester/mo.yml" \ - 'password: "111"' + "tenant-secret" assert_not_contains "${output_dir}/phases/worker-0/tester/mo.yml" \ - 'password: "tenant-secret"' + 'syspass: "111"' assert_contains "${output_dir}/phases/worker-0/tester/mo.yml" \ 'password: "***"' - assert_not_contains "${output_dir}/worker-0.include" ".sql" - assert_not_contains "${output_dir}/worker-1.include" ".test" + assert_contains "${output_dir}/phases/worker-0/tester/mo.yml" \ + 'syspass: "***"' assert_order "${event_log}" \ - "serial-before:dump:" \ + "serial-before:start:dump:" \ "create-account" \ "worker-" \ "drop-account" \ - "serial-after:dump:" + "serial-after:start:dump:" assert_contains "${output_dir}/summary.tsv" $'parallel\tworker-0\tpassed' assert_contains "${output_dir}/summary.tsv" $'parallel\tworker-1\tpassed' + unset FAKE_REQUIRE_CONCURRENCY } test_worker_failure_waits_for_sibling_and_runs_cleanup_and_after() { @@ -314,16 +362,16 @@ test_worker_failure_waits_for_sibling_and_runs_cleanup_and_after() { fail "worker failure should fail the orchestrator" fi - assert_contains "${event_log}" "worker-0:bvtw_g0_w0:admin:" - assert_contains "${event_log}" "worker-1:bvtw_g0_w1:admin:" + assert_contains "${event_log}" "worker-0:start:bvtw_g1_w0:admin:" + assert_contains "${event_log}" "worker-1:done" assert_contains "${event_log}" "drop-account" - assert_contains "${event_log}" "serial-after:dump:" + assert_contains "${event_log}" "serial-after:start:dump:" assert_contains "${output_dir}/summary.tsv" $'parallel\tworker-0\tfailed' assert_contains "${output_dir}/summary.tsv" $'parallel\tworker-1\tpassed' unset FAKE_FAIL_PHASE } -test_serial_before_failure_prevents_tenant_workers() { +test_serial_before_failure_prevents_accounts_and_workers() { setup_fixture export FAKE_FAIL_PHASE=serial-before @@ -331,35 +379,13 @@ test_serial_before_failure_prevents_tenant_workers() { fail "serial-before failure should fail the orchestrator" fi - assert_contains "${event_log}" "serial-before:dump:" + assert_contains "${event_log}" "serial-before:start:dump:" assert_not_contains "${event_log}" "create-account" - assert_not_contains "${event_log}" "worker-0:" - assert_not_contains "${event_log}" "worker-1:" + assert_not_contains "${event_log}" "worker-0:start:" + assert_not_contains "${event_log}" "worker-1:start:" unset FAKE_FAIL_PHASE } -test_empty_parallel_phase_skips_accounts() { - setup_fixture - cat > "${policy}" <<'JSON' -{ - "schema_version": 1, - "directories": { - "before": {"phase": "serial-before", "reason": "ordered"}, - "alpha": {"phase": "serial-after", "reason": "global"}, - "beta": {"phase": "serial-after", "reason": "global"}, - "after": {"phase": "serial-after", "reason": "global"} - } -} -JSON - - run_fixture - - assert_not_contains "${event_log}" "create-account" - assert_not_contains "${event_log}" "worker-0:" - assert_not_contains "${event_log}" "worker-1:" - assert_contains "${event_log}" "serial-after:dump:" -} - test_leaked_account_fails_before_serial_after() { setup_fixture export FAKE_LEAK_COUNT=1 @@ -369,7 +395,7 @@ test_leaked_account_fails_before_serial_after() { fi assert_contains "${event_log}" "drop-account" - assert_not_contains "${event_log}" "serial-after:dump:" + assert_not_contains "${event_log}" "serial-after:start:" unset FAKE_LEAK_COUNT } @@ -379,7 +405,7 @@ test_signal_stops_workers_before_cleanup() { RUN_FIXTURE_IN_PLACE=1 run_fixture & local orchestrator_pid=$! - if ! wait_for_event "${event_log}" "worker-0-blocked"; then + if ! wait_for_event "${event_log}" "worker-0:blocked"; then kill -TERM "${orchestrator_pid}" 2>/dev/null || true wait "${orchestrator_pid}" 2>/dev/null || true fail "worker did not enter blocking phase" @@ -399,11 +425,9 @@ test_signal_stops_workers_before_cleanup() { unset FAKE_BLOCK_PHASE (( status == 130 )) || fail "expected signal exit 130, got ${status}" - (( worker_was_alive == 0 )) || fail "worker remained alive after orchestrator exit" + (( worker_was_alive == 0 )) || fail "worker remained alive after exit" assert_not_contains "${event_log}" "drop-while-worker-alive" - assert_order "${event_log}" \ - "worker-0-blocked" \ - "drop-account" + assert_order "${event_log}" "worker-0:blocked" "drop-account" } test_account_creation_failure_prevents_workers() { @@ -417,9 +441,9 @@ test_account_creation_failure_prevents_workers() { unset FAKE_MYSQL_FAIL_MATCH assert_contains "${output_dir}/summary.tsv" \ $'parallel\taccount-setup\tfailed' - assert_not_contains "${event_log}" "worker-0:" - assert_not_contains "${event_log}" "worker-1:" - assert_not_contains "${event_log}" "serial-after:dump:" + assert_not_contains "${event_log}" "worker-0:start:" + assert_not_contains "${event_log}" "worker-1:start:" + assert_not_contains "${event_log}" "serial-after:start:" } test_unreachable_matrixone_skips_serial_after() { @@ -432,26 +456,22 @@ test_unreachable_matrixone_skips_serial_after() { unset FAKE_MYSQL_FAIL_MATCH assert_contains "${event_log}" "drop-account" - assert_not_contains "${event_log}" "serial-after:dump:" + assert_not_contains "${event_log}" "serial-after:start:" assert_contains "${output_dir}/summary.tsv" \ $'serial\tserial-after\tskipped' } -test_directory_contract -echo "ok - static directory contract" -test_successful_phase_order_and_isolation -echo "ok - successful phase order and isolation" -test_worker_failure_waits_for_sibling_and_runs_cleanup_and_after -echo "ok - worker failure aggregation and cleanup" -test_serial_before_failure_prevents_tenant_workers -echo "ok - serial-before failure barrier" -test_empty_parallel_phase_skips_accounts -echo "ok - empty parallel phase" -test_leaked_account_fails_before_serial_after -echo "ok - leaked account detection" -test_signal_stops_workers_before_cleanup -echo "ok - signal stops workers before cleanup" -test_account_creation_failure_prevents_workers -echo "ok - account creation failure barrier" -test_unreachable_matrixone_skips_serial_after -echo "ok - unreachable MatrixOne skips serial-after" +run_test() { + local name=$1 + "$2" + echo "ok - ${name}" +} + +run_test "static directory contract" test_directory_contract +run_test "direct commands and isolation" test_successful_direct_commands_and_isolation +run_test "worker failure aggregation" test_worker_failure_waits_for_sibling_and_runs_cleanup_and_after +run_test "serial-before barrier" test_serial_before_failure_prevents_accounts_and_workers +run_test "leaked account detection" test_leaked_account_fails_before_serial_after +run_test "signal cleanup ordering" test_signal_stops_workers_before_cleanup +run_test "account creation barrier" test_account_creation_failure_prevents_workers +run_test "unreachable MatrixOne handling" test_unreachable_matrixone_skips_serial_after From c2d155a0f692610aaea43be567e4199b52b78563 Mon Sep 17 00:00:00 2001 From: "Ariznawl@163.com" Date: Wed, 29 Jul 2026 16:04:45 +0800 Subject: [PATCH 11/16] ci: use fixed two-tenant BVT execution --- .github/workflows/e2e-compose-parallel.yaml | 10 +------ .../workflows/e2e-standalone-parallel.yaml | 10 +------ scripts/test_bvt_workflow_contract.py | 27 +++++++++++++------ 3 files changed, 21 insertions(+), 26 deletions(-) diff --git a/.github/workflows/e2e-compose-parallel.yaml b/.github/workflows/e2e-compose-parallel.yaml index 6a3036e..dcae3bd 100644 --- a/.github/workflows/e2e-compose-parallel.yaml +++ b/.github/workflows/e2e-compose-parallel.yaml @@ -18,11 +18,6 @@ on: required: false type: boolean default: false - tenant_parallel_workers: - description: "Tenant worker count (1-4)" - required: false - type: number - default: 2 ci_ref: description: "matrixorigin/CI ref containing tenant BVT scripts" required: false @@ -368,11 +363,8 @@ jobs: bash "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/run_bvt_tenant_parallel.sh" \ --tester-dir "$GITHUB_WORKSPACE/mo-tester" \ --case-root "$GITHUB_WORKSPACE/test/distributed/cases" \ - --group-runner "${bvt_runner}" \ --group "${bvt_group}" \ - --workers '${{ inputs.tenant_parallel_workers }}' \ - --policy "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/bvt_tenant_policy.json" \ - --planner "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/bvt_tenant_plan.py" \ + --directories "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/bvt_tenant_directories.sh" \ --output-dir "${RUNNER_TEMP}/bvt-tenant-compose" else bash "${bvt_runner}" \ diff --git a/.github/workflows/e2e-standalone-parallel.yaml b/.github/workflows/e2e-standalone-parallel.yaml index 88bd6bd..0ca218e 100644 --- a/.github/workflows/e2e-standalone-parallel.yaml +++ b/.github/workflows/e2e-standalone-parallel.yaml @@ -18,11 +18,6 @@ on: required: false type: boolean default: false - tenant_parallel_workers: - description: "Tenant worker count (1-4)" - required: false - type: number - default: 2 ci_ref: description: "matrixorigin/CI ref containing tenant BVT scripts" required: false @@ -379,11 +374,8 @@ jobs: bash "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/run_bvt_tenant_parallel.sh" \ --tester-dir "$GITHUB_WORKSPACE/mo-tester" \ --case-root "$GITHUB_WORKSPACE/head/test/distributed/cases" \ - --group-runner "${bvt_runner}" \ --group "${bvt_group}" \ - --workers '${{ inputs.tenant_parallel_workers }}' \ - --policy "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/bvt_tenant_policy.json" \ - --planner "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/bvt_tenant_plan.py" \ + --directories "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/bvt_tenant_directories.sh" \ --output-dir "${RUNNER_TEMP}/bvt-tenant-pessimistic" \ --resource-dir "$GITHUB_WORKSPACE/head/test/distributed/resources" else diff --git a/scripts/test_bvt_workflow_contract.py b/scripts/test_bvt_workflow_contract.py index 11f98f0..a725e02 100644 --- a/scripts/test_bvt_workflow_contract.py +++ b/scripts/test_bvt_workflow_contract.py @@ -47,12 +47,6 @@ def assert_inputs(self): "type": "boolean", "default": False, }, - "tenant_parallel_workers": { - "description": "Tenant worker count (1-4)", - "required": False, - "type": "number", - "default": 2, - }, "ci_ref": { "description": "matrixorigin/CI ref containing tenant BVT scripts", "required": False, @@ -63,6 +57,7 @@ def assert_inputs(self): for name, value in expected.items(): self.test_case.assertIn(name, self.inputs) self.test_case.assertEqual(self.inputs[name], value) + self.test_case.assertNotIn("tenant_parallel_workers", self.inputs) def assert_ci_checkout(self): step = self.step("Checkout tenant-parallel BVT scripts") @@ -101,7 +96,15 @@ def test_start_step_has_opt_in_and_fallback_branches(self): ) self.assertIn('--case-root "$GITHUB_WORKSPACE/test/distributed/cases"', run) self.assertNotIn("--resource-dir", run) - self.assertIn("--workers '${{ inputs.tenant_parallel_workers }}'", run) + self.assertIn( + '--directories "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/' + 'bvt_tenant_directories.sh"', + run, + ) + self.assertNotIn("--workers", run) + self.assertNotIn("--planner", run) + self.assertNotIn("--policy", run) + self.assertNotIn("--group-runner", run) self.assertIn('else\n bash "${bvt_runner}"', run) def test_artifact_contains_tenant_output(self): @@ -139,7 +142,15 @@ def test_start_step_has_opt_in_resource_and_fallback_branches(self): '--resource-dir "$GITHUB_WORKSPACE/head/test/distributed/resources"', run, ) - self.assertIn("--workers '${{ inputs.tenant_parallel_workers }}'", run) + self.assertIn( + '--directories "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/' + 'bvt_tenant_directories.sh"', + run, + ) + self.assertNotIn("--workers", run) + self.assertNotIn("--planner", run) + self.assertNotIn("--policy", run) + self.assertNotIn("--group-runner", run) self.assertIn('else\n bash "${bvt_runner}"', run) def test_artifact_contains_tenant_output(self): From 48d0c5d7e78c51ea281df1d6fddd4ac6573d5ac5 Mon Sep 17 00:00:00 2001 From: "Ariznawl@163.com" Date: Wed, 29 Jul 2026 16:13:32 +0800 Subject: [PATCH 12/16] docs: simplify tenant-parallel BVT rollout --- .../plans/2026-07-29-tenant-parallel-bvt.md | 584 +++++++----------- .../2026-07-28-tenant-parallel-bvt-design.md | 414 ++++--------- 2 files changed, 320 insertions(+), 678 deletions(-) diff --git a/docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md b/docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md index a1ca2e9..7526de6 100644 --- a/docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md +++ b/docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md @@ -1,278 +1,164 @@ -# Tenant-Parallel BVT Implementation Plan +# Direct Tenant-Parallel BVT Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. -**Goal:** Run the directory-safe portion of each MatrixOne PR BVT group concurrently in two tenant accounts on one deployment, while preserving sys-account serial phases and the existing fallback. +**Goal:** Run the directory-safe part of each PR BVT group through two direct, +concurrent `mo-tester` commands using ordinary tenants, with all privileged and +global cases serialized as `sys`. -**Architecture:** A Python planner converts the MatrixOne group runner's selected script list into whole top-level directory units using an explicit JSON policy. A shell orchestrator creates isolated `mo-tester` copies and tenant accounts, runs serial-before, tenant workers, cleanup, and serial-after in order, and records one artifact directory. The two reusable workflows checkout the matching CI revision and select the old or new path through an opt-in input. +**Architecture:** A sourced Bash file owns the complete top-level directory +classification and fixed two-worker split. A Bash orchestrator prepares +isolated tester/resource copies, invokes `run.sh -p CASE_ROOT -i DIRS` for each +phase, manages two accounts, and aggregates failures. The two reusable +workflows select this path behind an opt-in boolean and otherwise retain their +existing group runner. -**Tech Stack:** Python 3 standard library, Bash, MySQL CLI, `mo-tester`, GitHub Actions YAML. +**Tech Stack:** Bash, MySQL CLI, `mo-tester`, Python standard-library workflow +contract tests, GitHub Actions YAML. ## Global Constraints -- The smallest scheduling unit is one immediate child directory of `test/distributed/cases`; no file-level policy or worker assignment is allowed. -- The selected directories are exactly the current MatrixOne group intersected with the policy; no selected script may be omitted or assigned twice. -- Unknown top-level directories run in `serial-after` and are reported as unreviewed. -- A runtime serial-rule match downgrades the entire allowlisted parallel directory to - `serial-after`; the planner never splits a directory by file. -- The default worker count is `2`; accepted values are `1` through `4`. -- Each workflow job continues to use one runner and one MatrixOne deployment. -- Worker accounts are removed before `serial-after`. -- `tenant_parallel_enabled` defaults to `false`, preserving the existing `run_bvt_group.sh` path. -- Test code uses only the Python standard library and fake local executables; repository tests must not require a live MatrixOne deployment. +- One workflow job uses exactly one runner and one MatrixOne deployment. +- The scheduling unit is one immediate child directory of + `test/distributed/cases`; no test-file list is maintained. +- The parallel worker count is fixed at exactly `2`. +- Account-management, tenant, snapshot/PITR, publication, explicit-credential, + cluster-global, task, failpoint, and system behavior runs as `sys`. +- Unknown directories run in `sys-after`; `optimistic` remains excluded. +- Include values are comma-separated absolute directory paths ending in `/`. +- Each concurrent invocation has isolated tester, report, log, and resource + directories. +- Workers are waited for before accounts are dropped, including failure and + signal paths. +- `tenant_parallel_enabled` defaults to `false`; the legacy command is + unchanged when disabled. --- -### Task 1: Directory Policy and Planner +### Task 1: Replace dynamic planning with a static directory contract **Files:** -- Create: `scripts/bvt_tenant_policy.json` -- Create: `scripts/bvt_tenant_plan.py` -- Create: `scripts/test_bvt_tenant_plan.py` +- Create: `scripts/bvt_tenant_directories.sh` +- Delete: `scripts/bvt_tenant_plan.py` +- Delete: `scripts/bvt_tenant_policy.json` +- Delete: `scripts/test_bvt_tenant_plan.py` +- Modify: `scripts/test_run_bvt_tenant_parallel.sh` **Interfaces:** -- Consumes: MatrixOne case root, newline-delimited selected script paths captured from `run_bvt_group.sh`, worker count, and JSON policy. -- Produces: `plan.json`, `inventory.tsv`, `serial-before.include`, `worker-N.include`, and `serial-after.include`. -- Python API: - - `load_policy(path: pathlib.Path) -> dict[str, PolicyEntry]` - - `build_plan(case_root: pathlib.Path, selected_file: pathlib.Path, policy: dict[str, PolicyEntry], workers: int) -> dict` - - `write_plan(plan: dict, output_dir: pathlib.Path) -> None` - -- [ ] **Step 1: Write failing policy-validation and directory-planning tests** - -Create `scripts/test_bvt_tenant_plan.py` with `unittest`. Fixtures must use literal expected directories and script counts: - -```python -def test_build_plan_assigns_whole_directories_and_preserves_selection(self): - self.case("alpha/a.sql", "x") - self.case("alpha/b.sql", "xx") - self.case("beta/a.sql", "xxx") - self.case("global/a.sql", "xxxx") - selected = self.selected("alpha/a.sql", "alpha/b.sql", "beta/a.sql", "global/a.sql") - policy = { - "alpha": PolicyEntry("parallel", "safe"), - "beta": PolicyEntry("parallel", "safe"), - "global": PolicyEntry("serial-after", "global state"), - } - - plan = build_plan(self.case_root, selected, policy, workers=2) - - self.assertEqual(plan["serial_after"], [str(self.case_root / "global")]) - self.assertEqual( - sorted(path for worker in plan["workers"] for path in worker["directories"]), - sorted([str(self.case_root / "alpha"), str(self.case_root / "beta")]), - ) - self.assertEqual(plan["selected_script_count"], 4) -``` - -Also cover: +- Consumes: global variable `bvt_group` set to `0` or `1`, plus a case root + supplied to helper functions by the orchestrator. +- Produces arrays `bvt_serial_before`, `bvt_worker_0`, `bvt_worker_1`, + `bvt_serial_after`, and `bvt_excluded`. +- Produces function `bvt_group_for_directory NAME` that prints `0` or `1` using + the explicit MatrixOne group lists and `cksum % 2` for unknown names. -- duplicate directory entries rejected by the JSON object loader; -- invalid phase rejected; -- `workers=0` and `workers=5` rejected; -- a selected file outside the case root rejected; -- a partial selection from one top-level directory rejected; -- an unknown directory assigned to `serial-after` with `reviewed=false`; -- serial-before, worker, and serial-after sets are disjoint; -- longest-first directory weighting is deterministic; -- emitted include files contain absolute directory paths, never `.sql` or `.test` paths. +- [ ] **Step 1: Write the failing directory-contract test** -- [ ] **Step 2: Run the planner tests and verify RED** - -Run: +Add a shell test that sources the configuration for groups 0 and 1 and asserts: ```bash -python3 -m unittest scripts/test_bvt_tenant_plan.py -v +assert_array_contains bvt_worker_0 view +assert_array_contains bvt_worker_1 auto_increment +assert_array_contains bvt_serial_after analyze +assert_array_contains bvt_serial_after benchmark +assert_array_not_contains bvt_worker_0 analyze +assert_array_not_contains bvt_worker_1 benchmark +assert_all_unique_selected_directories ``` -Expected: `ModuleNotFoundError` for `scripts.bvt_tenant_plan`. - -- [ ] **Step 3: Add the explicit 72-directory policy** - -Create schema version 1 with one object per directory: - -```json -{ - "schema_version": 1, - "directories": { - "log": {"phase": "serial-before", "reason": "statement and log metadata ordering"}, - "result_count": {"phase": "serial-before", "reason": "statement result metadata ordering"}, - "sql_source_type": {"phase": "serial-before", "reason": "statement source metadata ordering"}, - "statement_query_type": {"phase": "serial-before", "reason": "statement query metadata producer"}, - "zz_statement_query_type": {"phase": "serial-before", "reason": "statement query metadata verifier"}, - - "analyze": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "auto_increment": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "benchmark": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker; keep the entire directory on one worker"}, - "charset_collation": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "comment": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "cte": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "dataXtest": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "distinct": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "dtype": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "expression": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "fake_pk": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "fulltext": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "geo": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "keyword": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "operator": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "pg_cast": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "plan_cache": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "plugin": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "procedure": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "qexec": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "recursive_cte": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "replace_statement": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "sample": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "sequence": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "time_window": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "udf": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "union": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "view": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - "window": {"phase": "parallel", "reason": "directory scan found no tenant isolation blocker"}, - - "array": {"phase": "serial-after", "reason": "contains mo_ctl"}, - "database": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, - "ddl": {"phase": "serial-after", "reason": "contains account DDL, mo_ctl, or SET GLOBAL"}, - "disttae": {"phase": "serial-after", "reason": "contains account DDL or mo_ctl"}, - "dml": {"phase": "serial-after", "reason": "contains account DDL, mo_ctl, or global state"}, - "feature_limit": {"phase": "serial-after", "reason": "global feature registry and account state"}, - "foreign_key": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, - "function": {"phase": "serial-after", "reason": "contains mo_ctl, account DDL, or account identity functions"}, - "git4data": {"phase": "serial-after", "reason": "branch, account, debug, and global state"}, - "hint": {"phase": "serial-after", "reason": "contains account DDL, SET GLOBAL, or system metrics"}, - "iceberg": {"phase": "serial-after", "reason": "contains explicit account sessions and SET GLOBAL"}, - "join": {"phase": "serial-after", "reason": "contains mo_ctl or account DDL"}, - "load_data": {"phase": "serial-after", "reason": "contains account DDL or mo_catalog.mo_account"}, - "metadata": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, - "mo_cloud": {"phase": "serial-after", "reason": "external environment and system metrics"}, - "optimizer": {"phase": "serial-after", "reason": "contains mo_ctl or current account assumptions"}, - "pessimistic_transaction": {"phase": "serial-after", "reason": "account, restore, debug, and transaction-global state"}, - "pitr": {"phase": "serial-after", "reason": "account recovery and debug state"}, - "prepare": {"phase": "serial-after", "reason": "contains account DDL or SET GLOBAL"}, - "publication_subscription": {"phase": "serial-after", "reason": "cross-account publication state"}, - "query_result": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, - "save_query_result": {"phase": "serial-after", "reason": "contains account enumeration and DDL"}, - "security": {"phase": "serial-after", "reason": "explicit account sessions and SET GLOBAL"}, - "set": {"phase": "serial-after", "reason": "contains account DDL or SET GLOBAL"}, - "snapshot": {"phase": "serial-after", "reason": "account snapshot and restore state"}, - "sql_inject": {"phase": "serial-after", "reason": "cluster-global failpoint state"}, - "stage": {"phase": "serial-after", "reason": "contains account DDL or mo_ctl"}, - "subquery": {"phase": "serial-after", "reason": "contains mo_ctl"}, - "system": {"phase": "serial-after", "reason": "system account behavior"}, - "system_variable": {"phase": "serial-after", "reason": "global system variables and account sessions"}, - "table": {"phase": "serial-after", "reason": "cluster table, account, and system metrics state"}, - "task": {"phase": "serial-after", "reason": "background task and account state"}, - "temporary": {"phase": "serial-after", "reason": "contains account DDL or explicit account sessions"}, - "tenant": {"phase": "serial-after", "reason": "tenant and cross-account behavior"}, - "tenxcloud_xx": {"phase": "serial-after", "reason": "external environment"}, - "util": {"phase": "serial-after", "reason": "contains mo_ctl"}, - "vector": {"phase": "serial-after", "reason": "contains mo_ctl or SET GLOBAL"}, - "zz_accesscontrol": {"phase": "serial-after", "reason": "account, password, and global access-control state"} - } -} -``` +Also construct a case root with an unknown top-level directory and assert the +orchestrator writes it to `serial-after.include`, never a worker include. -- [ ] **Step 4: Implement the minimal planner** +- [ ] **Step 2: Run the test to verify RED** -Implement duplicate-key detection with `json.load(..., object_pairs_hook=...)`, normalize every selected path with `Path.resolve()`, map it to `relative.parts[0]`, and reject partial directory selection by comparing selected `.sql`/`.test` files with a recursive discovery of that directory. - -Balance only `parallel` directories: +Run: -```python -units = sorted(parallel_units, key=lambda unit: (-unit["weight_bytes"], unit["name"])) -worker_units = [{"index": index, "weight_bytes": 0, "directories": []} for index in range(workers)] -for unit in units: - target = min(worker_units, key=lambda worker: (worker["weight_bytes"], worker["index"])) - target["directories"].append(unit["path"]) - target["weight_bytes"] += unit["weight_bytes"] +```bash +bash scripts/test_run_bvt_tenant_parallel.sh ``` -Write include files as a single comma-separated line of absolute directory paths. Empty phases produce empty files. +Expected: fail because `scripts/bvt_tenant_directories.sh` does not exist and +the old orchestrator still requires `--planner`, `--policy`, and `--workers`. -- [ ] **Step 5: Run planner tests and verify GREEN** +- [ ] **Step 3: Add the directory arrays and group fallback** -Run: +Define literal arrays: ```bash -python3 -m unittest scripts/test_bvt_tenant_plan.py -v +bvt_serial_before_all=( + log result_count sql_source_type statement_query_type zz_statement_query_type +) +bvt_parallel_group_0_worker_0=(view) +bvt_parallel_group_0_worker_1=( + auto_increment sequence procedure keyword sample pg_cast plugin + time_window union fake_pk dataXtest +) +bvt_parallel_group_1_worker_0=( + dtype expression comment recursive_cte qexec replace_statement +) +bvt_parallel_group_1_worker_1=( + window fulltext operator geo charset_collation distinct udf cte plan_cache +) ``` -Expected: all tests pass. - -- [ ] **Step 6: Dry-run the planner against MatrixOne main** +Add the complete 40-directory sys-after list, the two explicit +`run_bvt_group.sh` directory lists, and `optimistic`. Build phase arrays by +filtering every literal directory through `bvt_group_for_directory`. -Capture both group selections with a fake `mo-tester/run.sh`, plan both groups, and verify the union contains 72 directories. At the implementation-time MatrixOne main commit `129bd689b5c415fbb448eb7b413ee84b245fb938`, the union contains 1,137 scripts. +- [ ] **Step 4: Run the directory-contract test to verify GREEN** -Expected aggregate: +Run: -```text -serial-before: 5 directories, 29 scripts -parallel: 29 directories, 216 scripts -serial-after: 38 directories, 892 scripts +```bash +bash scripts/test_run_bvt_tenant_parallel.sh directory_contract ``` -- [ ] **Step 7: Commit the planner** +Expected: the directory contract scenario passes. + +- [ ] **Step 5: Commit** ```bash -git add scripts/bvt_tenant_policy.json scripts/bvt_tenant_plan.py scripts/test_bvt_tenant_plan.py -git commit -m "feat: plan BVT execution by case directory" +git add scripts/bvt_tenant_directories.sh scripts/test_run_bvt_tenant_parallel.sh +git rm scripts/bvt_tenant_plan.py scripts/bvt_tenant_policy.json scripts/test_bvt_tenant_plan.py +git commit -m "refactor: define BVT tenant execution by directory" ``` --- -### Task 2: Tenant-Parallel Shell Orchestrator +### Task 2: Rewrite the direct mo-tester orchestrator **Files:** -- Create: `scripts/run_bvt_tenant_parallel.sh` -- Create: `scripts/test_run_bvt_tenant_parallel.sh` +- Modify: `scripts/run_bvt_tenant_parallel.sh` +- Modify: `scripts/test_run_bvt_tenant_parallel.sh` **Interfaces:** -- Consumes: - - `--tester-dir PATH` - - `--case-root PATH` - - `--group-runner PATH` - - `--group 0|1` - - `--policy PATH` - - `--planner PATH` - - `--output-dir PATH` - - `--workers 1..4` - - optional `--resource-dir PATH` - - optional MySQL flags defaulting to `127.0.0.1:6001`, `dump/111` -- Produces: planner artifacts, per-phase logs and reports, `summary.tsv`, and a nonzero aggregate exit status. - -- [ ] **Step 1: Write the failing shell integration test** - -Use a temporary fixture containing: - -- a fake group runner that invokes its tester with a literal four-file include list; -- a fake original `mo-tester` tree with `run.sh`, `mo.yml`, `run.yml`, `lib/`, `log/`, and `report/`; -- a fake `mysql` executable that appends SQL to `${FAKE_MYSQL_LOG}`; -- a fake planner implementing the real planner CLI output contract. - -Assert observable behavior: +- Consumes `--tester-dir`, `--case-root`, `--group 0|1`, `--directories`, + `--output-dir`, optional `--resource-dir`, MySQL connection flags, and + `--tenant-password`. +- Runs a phase using: ```bash -assert_order serial-before create-account worker-0 worker-1 drop-account serial-after -assert_contains "$FAKE_MYSQL_LOG" "create account" -assert_contains "$FAKE_MYSQL_LOG" "drop account" -assert_file "$output_dir/plan.json" -assert_file "$output_dir/summary.tsv" -assert_not_contains "$output_dir/worker-0.include" ".sql" +./run.sh -n -g -o -p "${case_root}" -i "${include_value}" ``` -Add separate cases proving: +- Produces `serial-before.include`, `worker-0.include`, `worker-1.include`, + `serial-after.include`, per-phase tester/report/log trees, and `summary.tsv`. + +- [ ] **Step 1: Rewrite the fake tester assertions and verify RED** -- sibling workers finish even if worker 0 fails; -- worker failure makes the final script fail; -- serial-after still runs after a worker assertion failure; -- serial-before failure prevents account creation and workers; -- account cleanup occurs on worker failure; -- an empty parallel phase skips account creation; -- generated worker `mo.yml` uses `bvtw_g_w:admin`; -- resource directories are copied per phase when `--resource-dir` is supplied. +Require exact directory-based invocation, fixed worker users, two concurrently +started workers, isolated resources, config redaction, and this event order: + +```text +serial-before +create-account (twice) +worker-0 and worker-1 +drop-account (twice) +serial-after +``` -- [ ] **Step 2: Run the shell test and verify RED** +Add failure scenarios for serial-before, one worker, cleanup, leak detection, +and TERM while one worker is blocked. Run: @@ -280,110 +166,95 @@ Run: bash scripts/test_run_bvt_tenant_parallel.sh ``` -Expected: fail because `scripts/run_bvt_tenant_parallel.sh` does not exist. - -- [ ] **Step 3: Implement argument validation and group capture** - -Use `set -uo pipefail`, resolve all required paths, validate workers and group, create only children below the explicit output directory, and install a trap that drops only names recorded in an in-memory `created_accounts` array. +Expected: failures show the old planner CLI and capture logic do not satisfy the +new interface. -The capture tester parses the group runner's `-i` argument and writes one selected script per line to `${output_dir}/selected-files.txt`. +- [ ] **Step 2: Implement validation and include generation** -- [ ] **Step 4: Implement isolated tester preparation** +Source only the explicit `--directories` file after validating it is a regular +file. Validate all configured names with: -For every phase copy only: - -```text -run.sh -run.yml -mo.yml -kafka.yml -log4j.properties -pprof.sh -lib/ +```bash +[[ "${name}" =~ ^[A-Za-z0-9_]+$ ]] || + die "invalid case directory name: ${name}" ``` -Create fresh `log/` and `report/`. Copy the MatrixOne resource directory separately for every phase because `mo-tester` removes configured output paths during startup and cleanup. - -For workers, replace only the YAML `user.name` and `user.password` values; retain `sysuser` and `syspass`. +Discover immediate case-root directories, ignore `optimistic`, append unknown +directories belonging to the requested group to sys-after, and write absolute +paths with trailing `/` joined by commas. -- [ ] **Step 5: Implement phase execution and status aggregation** +- [ ] **Step 3: Implement isolated phase execution** -Run serial-before synchronously. Create accounts with: - -```sql -create account if not exists `bvtw_g0_w0` admin_name 'admin' identified by '111'; -``` +Copy the tester runtime files and `lib/` into +`output/phases//tester`, create fresh `log/` and `report/`, and copy the +resource tree to `output/phases//resources` when supplied. Change only +the worker `user.name` and `user.password` fields before running the worker. -Run every tenant worker in a background subshell with `set -o pipefail` and `tee`. Wait for all PIDs without exiting on the first failure. Drop every created account, then run serial-after when MySQL is reachable. +- [ ] **Step 4: Implement account lifecycle and failure aggregation** -Write `summary.tsv` columns: +Create: -```text -phase name status include_file log_file +```sql +CREATE ACCOUNT IF NOT EXISTS `bvtw_g0_w0` + ADMIN_NAME 'admin' IDENTIFIED BY '111'; ``` -Exit nonzero when a required phase failed, cleanup failed, or serial-after was skipped because MatrixOne was unreachable. +for both workers, launch each worker in its own process group, wait for both, +drop and leak-check both accounts, then run sys-after when MySQL answers +`SELECT 1`. Record every status in `summary.tsv`; return nonzero if any required +step failed. -- [ ] **Step 6: Run shell tests and verify GREEN** - -Run: - -```bash -bash scripts/test_run_bvt_tenant_parallel.sh -``` +- [ ] **Step 5: Implement signal cleanup and artifact redaction** -Expected: all cases print `ok` and the script exits 0. +On `INT`, `TERM`, or `EXIT`, terminate and wait for live worker process groups +before dropping recorded accounts. Rewrite both `password:` and `syspass:` +values in every generated `mo.yml` to `"***"`. -- [ ] **Step 7: Run planner and shell tests together** +- [ ] **Step 6: Run shell verification** Run: ```bash -python3 -m unittest scripts/test_bvt_tenant_plan.py -v -bash scripts/test_run_bvt_tenant_parallel.sh bash -n scripts/run_bvt_tenant_parallel.sh +bash -n scripts/bvt_tenant_directories.sh +bash scripts/test_run_bvt_tenant_parallel.sh +shellcheck scripts/run_bvt_tenant_parallel.sh scripts/bvt_tenant_directories.sh scripts/test_run_bvt_tenant_parallel.sh ``` -Expected: all pass. +Expected: syntax checks, every shell scenario, and ShellCheck pass. -- [ ] **Step 8: Commit the orchestrator** +- [ ] **Step 7: Commit** ```bash git add scripts/run_bvt_tenant_parallel.sh scripts/test_run_bvt_tenant_parallel.sh -git commit -m "feat: orchestrate tenant-parallel BVT phases" +git commit -m "feat: run BVT directories in two tenant processes" ``` --- -### Task 3: Reusable Workflow Integration +### Task 3: Simplify reusable workflow integration **Files:** - Modify: `.github/workflows/e2e-compose-parallel.yaml` - Modify: `.github/workflows/e2e-standalone-parallel.yaml` -- Create: `scripts/test_bvt_workflow_contract.py` +- Modify: `scripts/test_bvt_workflow_contract.py` **Interfaces:** -- New workflow inputs: - - `tenant_parallel_enabled`: boolean, default `false` - - `tenant_parallel_workers`: number, default `2` - - `ci_ref`: string, default `main` -- Existing caller contracts and fallback commands remain unchanged. - -- [ ] **Step 1: Write a failing executable workflow-contract test** +- Workflow inputs: `tenant_parallel_enabled` boolean defaulting to `false`, and + `ci_ref` string defaulting to `main`. +- The enabled branch passes `--directories` and no planner, policy, group + runner, or worker-count arguments. -Load both YAML files as text and use a small structural extractor to assert: +- [ ] **Step 1: Update workflow tests and verify RED** -- all three inputs exist with exact defaults; -- the active BVT job checks out `matrixorigin/CI` at `${{ inputs.ci_ref }}` into `.ci/tenant-parallel`; -- the old `run_bvt_group.sh` command is guarded by disabled tenant parallelism; -- the orchestrator command is guarded by enabled tenant parallelism; -- compose passes its case root without a resource directory; -- standalone passes `head/test/distributed/resources`; -- artifact upload includes the orchestrator output directory. +Assert `tenant_parallel_workers` is absent and each enabled command contains: -The test must fail on the current workflows because the new inputs and guarded orchestrator call do not exist. +```text +--directories "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/bvt_tenant_directories.sh" +``` -- [ ] **Step 2: Run the workflow test and verify RED** +Assert `--planner`, `--policy`, `--group-runner`, and `--workers` are absent, +while the `else` branch still invokes the original BVT runner. Run: @@ -391,129 +262,98 @@ Run: python3 -m unittest scripts/test_bvt_workflow_contract.py -v ``` -Expected: failures naming missing `tenant_parallel_enabled`. - -- [ ] **Step 3: Add opt-in inputs and CI checkout** - -Add the three inputs to both `workflow_call.inputs`. In each active job, checkout: - -```yaml -- name: Checkout tenant-parallel BVT scripts - if: ${{ inputs.tenant_parallel_enabled }} - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 - with: - repository: matrixorigin/CI - ref: ${{ inputs.ci_ref }} - path: .ci/tenant-parallel -``` - -- [ ] **Step 4: Guard fallback and tenant-parallel execution** - -Keep the current group assignment and manifest generation. Replace only the final test invocation branch: +Expected: tests fail against the old workflow arguments. -```bash -if [ '${{ inputs.tenant_parallel_enabled }}' = 'true' ]; then - bash "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/run_bvt_tenant_parallel.sh" \ - --tester-dir "$GITHUB_WORKSPACE/mo-tester" \ - --case-root "$GITHUB_WORKSPACE/test/distributed/cases" \ - --group-runner "${bvt_runner}" \ - --group "${bvt_group}" \ - --workers '${{ inputs.tenant_parallel_workers }}' \ - --policy "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/bvt_tenant_policy.json" \ - --planner "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/bvt_tenant_plan.py" \ - --output-dir "${RUNNER_TEMP}/bvt-tenant-compose" -else - bash "${bvt_runner}" \ - "$GITHUB_WORKSPACE/mo-tester" \ - "$GITHUB_WORKSPACE/test/distributed/cases" \ - "${bvt_group}" -fi -``` +- [ ] **Step 2: Modify both workflows** -The standalone branch uses `head/test/distributed/cases`, adds `--resource-dir "$GITHUB_WORKSPACE/head/test/distributed/resources"`, and writes to `bvt-tenant-pessimistic`. +Remove the worker-count input. Keep the feature default disabled. Replace the +old orchestrator arguments with `--directories`, retaining the correct +compose/standalone case root, tester root, output directory, group, and +standalone resource directory. -- [ ] **Step 5: Extend artifact paths and summaries** - -Upload the existing combined log plus the corresponding tenant output directory. Add plan phase counts and `summary.tsv` to `$GITHUB_STEP_SUMMARY` when present. - -- [ ] **Step 6: Run contract and script tests** +- [ ] **Step 3: Verify workflow syntax and tests** Run: ```bash -python3 -m unittest scripts/test_bvt_tenant_plan.py scripts/test_bvt_workflow_contract.py -v -bash scripts/test_run_bvt_tenant_parallel.sh -bash -n scripts/run_bvt_tenant_parallel.sh -ruby -e 'require "yaml"; ARGV.each { |path| YAML.load_file(path, aliases: true) }' \ +python3 -m unittest scripts/test_bvt_workflow_contract.py -v +ruby -e 'require "yaml"; ARGV.each { |path| YAML.load_file(path) }' \ .github/workflows/e2e-compose-parallel.yaml \ .github/workflows/e2e-standalone-parallel.yaml ``` -Expected: every command exits 0. +Expected: all tests pass and Ruby parses both YAML documents. -- [ ] **Step 7: Commit workflow integration** +- [ ] **Step 4: Commit** ```bash -git add .github/workflows/e2e-compose-parallel.yaml .github/workflows/e2e-standalone-parallel.yaml scripts/test_bvt_workflow_contract.py -git commit -m "ci: add opt-in tenant-parallel BVT execution" +git add .github/workflows/e2e-compose-parallel.yaml \ + .github/workflows/e2e-standalone-parallel.yaml \ + scripts/test_bvt_workflow_contract.py +git commit -m "ci: use fixed two-tenant BVT execution" ``` --- -### Task 4: Full Verification and Trial Handoff +### Task 4: Audit, trial instructions, and PR update **Files:** -- Modify if verification finds a tested defect: only files introduced or named above. +- Modify: `docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md` +- Modify: `docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md` **Interfaces:** -- Produces a local CI commit SHA suitable for pinning from a MatrixOne trial branch. +- Produces a reviewable directory inventory and an opt-in branch trial command. + +- [ ] **Step 1: Verify the directory inventory against MatrixOne main** + +Export MatrixOne main, enumerate immediate case directories, and verify: -- [ ] **Step 1: Run the complete local verification** +```text +sys-before: 5 directories / 29 scripts +tenant-parallel: 27 directories / 175 scripts +sys-after: 40 directories / 935 scripts +excluded optimistic: 1 directory / 22 scripts +``` + +Verify `analyze` and `benchmark` occur only in sys-after and every non-excluded +directory belongs to exactly one phase. + +- [ ] **Step 2: Run the full local suite** + +Run: ```bash -python3 -m unittest scripts/test_bvt_tenant_plan.py scripts/test_bvt_workflow_contract.py -v +python3 -m unittest scripts/test_bvt_workflow_contract.py -v bash scripts/test_run_bvt_tenant_parallel.sh -bash -n scripts/run_bvt_tenant_parallel.sh +bash -n scripts/run_bvt_tenant_parallel.sh scripts/bvt_tenant_directories.sh +shellcheck scripts/run_bvt_tenant_parallel.sh scripts/bvt_tenant_directories.sh scripts/test_run_bvt_tenant_parallel.sh git diff --check ``` -- [ ] **Step 2: Run a real MatrixOne-main planning dry-run** +Expected: all checks pass. -Use `origin/main:optools/run_bvt_group.sh` from `/Users/ariznawl/weilu/matrixone` and the real case tree exported from that ref. Verify both groups together produce: +- [ ] **Step 3: Commit documentation** -```text -72 selected directories -1137 selected scripts at MatrixOne 129bd689b5c415fbb448eb7b413ee84b245fb938 -5 serial-before directories -29 parallel directories -38 serial-after directories +```bash +git add docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md \ + docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md +git commit -m "docs: simplify tenant-parallel BVT rollout" ``` -- [ ] **Step 3: Inspect the final diff** +- [ ] **Step 4: Push and update Draft PR #409** -Confirm: +Push `codex/tenant-parallel-bvt`, update the PR body with the direct-command +architecture, and keep the PR in Draft while the opt-in trial is running. -- no MatrixOne or `mo-tester` source file changed; -- workflow defaults preserve serial behavior; -- no password appears in `plan.json`, `inventory.tsv`, or summary output; -- cleanup targets only `bvtw_g_w`; -- no generated test artifact is staged. +- [ ] **Step 5: Trigger the shadow trial** -- [ ] **Step 4: Prepare the MatrixOne trial configuration** - -The caller pins both reusable workflows to the final CI SHA and passes: +From a MatrixOne branch, call the reusable workflow with: ```yaml -with: - tenant_parallel_enabled: true - tenant_parallel_workers: 2 - ci_ref: codex/tenant-parallel-bvt +tenant_parallel_enabled: true +ci_ref: codex/tenant-parallel-bvt ``` -For a non-experimental rollout, replace the branch in both the reusable-workflow `uses:` reference and `ci_ref` with the same value printed by `git rev-parse HEAD`. - -Run the Compose + Proxy and Launch + Pessimistic jobs once. Compare BVT result coverage, leaked accounts, MatrixOne crashes/restarts, and wall-clock time with the serial baseline. - -- [ ] **Step 5: Record final status** - -Report the final local branch, commit SHA, verification commands, exact trial inputs, and the remaining push-permission limitation. +Compare its duration, failed cases, `summary.tsv`, and worker reports against an +unchanged run from the same MatrixOne commit. Do not merge or enable the new +path by default until coverage and cleanup are confirmed. diff --git a/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md b/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md index 33c8798..3c42c38 100644 --- a/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md +++ b/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md @@ -1,335 +1,137 @@ -# Tenant-Parallel BVT Design +# Direct Tenant-Parallel BVT Design -## Status +## Decision -Approved direction: run one MatrixOne deployment on one GitHub Actions runner, then run multiple `mo-tester` processes against different MatrixOne tenants on that deployment. +Keep one GitHub Actions runner and one MatrixOne deployment per existing BVT +group. Inside that runner, invoke `mo-tester/run.sh` directly: -This design targets the active PR coverage workflows: +1. run ordering-sensitive directories as `sys`; +2. create two ordinary accounts; +3. run two isolated `mo-tester` processes concurrently, one account per process; +4. wait for both processes and drop both accounts; +5. run the remaining directories as `sys`. -- `.github/workflows/e2e-compose-parallel.yaml` -- `.github/workflows/e2e-standalone-parallel.yaml` +This replaces the Python planner and JSON policy. Directory lists are explicit +Bash arrays, and each `mo-tester -i` argument contains directory paths rather +than individual test files. -The first rollout keeps the existing complementary BVT group assignment across the two workflows. Each workflow parallelizes only its assigned group inside its own runner. Checkin Regression adoption is a later rollout after the PR workflow is stable. +The new path remains opt-in through `tenant_parallel_enabled: false`. When it +is disabled, each workflow runs its existing `run_bvt_group.sh` command +unchanged. -The CI repository baseline for this design is commit: +## Why CI-level processes -`940ee29eea1ee59eed5d3bec69cae7586b9a1162` +`mo-tester` accepts one case root through `-p` and comma-separated include +substrings through `-i`. An include ending in `/` safely selects a whole +directory. Its dormant in-process parallel code is not suitable because it is +disabled, fixed to two threads, and uses a hard-coded account. Separate +processes also require separate `mo-tester` and resource directories because +each invocation cleans databases and writes fixed report paths. -## Goals +## Directory rule -1. Reduce BVT wall-clock time without adding one runner per tenant. -2. Preserve the current union/disjoint guarantees of BVT groups 0 and 1. -3. Keep global-state and cross-account tests serial. -4. Produce an explainable per-directory classification and a merged report. -5. Make the new path opt-in until shadow validation proves it stable. -6. Leave the existing serial invocation available as an immediate fallback. +The immediate child directory of `test/distributed/cases` is the scheduling +unit. If any case in a directory needs account-management privileges, +cross-account state, cluster-global state, snapshots/PITR, publication state, +explicit credentials, system tables, failpoints, tasks, or external shared +state, the whole directory runs serially as `sys`. -## Non-goals +Unknown future directories are assigned to the same complementary BVT group as +`run_bvt_group.sh` and run in `sys-after`. `optimistic` remains excluded, +matching the current group runner. -- Enabling the dormant in-process parallel mode in `mo-tester`. -- Running one MatrixOne deployment per tenant. -- Moving or renaming MatrixOne BVT case directories. -- Converting every serial test into a parallel test in the first rollout. -- Changing Checkin Regression in the first rollout. +The classification was rescanned against MatrixOne main +`f6dab28046d70412cec132f0068840896852101c`: -## Considered approaches +| Phase | Directories | Scripts | +|---|---:|---:| +| sys-before | 5 | 29 | +| ordinary-tenant parallel | 27 | 175 | +| sys-after | 40 | 935 | +| excluded `optimistic` | 1 | 22 | -### CI-managed processes on one runner — selected +Runtime improvement must be measured in the opt-in trial because only about +15% of selected scripts are initially safe enough for ordinary tenants. -The CI workflow starts one MatrixOne deployment and creates two test tenants. It launches one `mo-tester` process per tenant, using isolated working and resource directories. +## Fixed classification -This keeps the implementation in `matrixorigin/CI`, does not depend on the disabled `mo-tester` parallel switch, and matches the requested same-cluster tenant isolation model. - -### In-process scheduling inside `mo-tester` - -This would provide tighter report integration but requires coordinated changes in `matrixorigin/mo-tester`. Its existing implementation is fixed to one extra tenant, lacks before/after barriers, and is disabled. It is not selected for the first rollout. - -### One GitHub Actions runner per group - -This is the current complementary-group model. It isolates failures well, but it creates separate MatrixOne deployments rather than using tenants on one deployment. It remains the outer deployment model but is not used for per-tenant parallelism. - -## Current inventory - -The baseline scan is anchored to MatrixOne commit: - -`d17b5a1f8e83cee4999181b9509af6126517985c` - -The full `test/distributed/cases` tree contains 1,155 `.sql` and `.test` scripts in 73 top-level directories. The current PR coverage selection excludes the `optimistic` directory, leaving 1,133 scripts in 72 top-level directories. - -The conservative initial classification is: - -| Phase | Directories | Scripts | Meaning | -|---|---:|---:|---| -| `serial-before` | 5 | 29 | Ordered observability producers and verifiers that must run before tenant-generated traffic | -| `parallel-candidate` | 29 | 215 | Every script in the directory passed the conservative scan; requires shadow validation | -| `serial-after` | 38 | 889 | At least one script in the directory has cross-account, cluster-global, explicit-user, recovery, or other high-risk behavior | - -The directory is the smallest scheduling and policy unit. The planner never divides scripts from the same top-level directory between phases or workers. - -This changes the expected optimization ceiling: only 215 of 1,133 scripts are initially parallel candidates. The simpler commands and safer maintenance are preferred over file-level parallel coverage; the actual wall-clock benefit must be established by shadow runs. - -An implementation-time rescan at MatrixOne commit -`129bd689b5c415fbb448eb7b413ee84b245fb938` found four newly added -scripts without any new top-level directory. The current total is 1,137: -29 serial-before, 216 parallel-candidate, and 892 serial-after scripts. -The directory counts and policy remain 5, 29, and 38 respectively. Runtime -planning derives script counts from the checked-out MatrixOne revision. - -## Classification policy - -The policy is stored as data in `scripts/bvt_tenant_policy.json`. It lists each top-level directory, its phase, and its reason. The planner emits `plan.json` and `inventory.tsv` with one record per directory. - -The policy is explicit and never assigns individual files. A newly added directory -defaults to `serial-after` and is reported as unreviewed. Before assigning an -allowlisted parallel directory, the planner rescans every script below it using the -serial content rules. A match downgrades the entire directory to `serial-after`, so a -new or modified case cannot silently inherit parallel status after introducing known -global or cross-account behavior. - -### Serial-before directories - -The following directories run in their existing lexical order as sys before test tenants are created: - -- `log` -- `result_count` -- `sql_source_type` -- `statement_query_type` -- `zz_statement_query_type` - -They inspect statement, log, and result metadata. Running them after tenant workers would expose them to parallel test traffic. - -### Parallel directories - -The following directories are the initial parallel candidates: - -- `analyze` -- `auto_increment` -- `benchmark` -- `charset_collation` -- `comment` -- `cte` -- `dataXtest` -- `distinct` -- `dtype` -- `expression` -- `fake_pk` -- `fulltext` -- `geo` -- `keyword` -- `operator` -- `pg_cast` -- `plan_cache` -- `plugin` -- `procedure` -- `qexec` -- `recursive_cte` -- `replace_statement` -- `sample` -- `sequence` -- `time_window` -- `udf` -- `union` -- `view` -- `window` - -Each directory is assigned to exactly one tenant worker. `benchmark` remains one unit, which preserves the lexical DDL, load, query, and cleanup order below `benchmark/tpch`. - -### Serial-after directories - -The following directories run as sys after tenant workers: - -- `array` -- `database` -- `ddl` -- `disttae` -- `dml` -- `feature_limit` -- `foreign_key` -- `function` -- `git4data` -- `hint` -- `iceberg` -- `join` -- `load_data` -- `metadata` -- `mo_cloud` -- `optimizer` -- `pessimistic_transaction` -- `pitr` -- `prepare` -- `publication_subscription` -- `query_result` -- `save_query_result` -- `security` -- `set` -- `snapshot` -- `sql_inject` -- `stage` -- `subquery` -- `system` -- `system_variable` -- `table` -- `task` -- `temporary` -- `tenant` -- `tenxcloud_xx` -- `util` -- `vector` -- `zz_accesscontrol` - -These directories exercise cross-account state, account recovery, global feature configuration, background tasks, failpoints, external environments, or contain at least one script matching a global-state rule. - -### Serial content rules - -A directory is classified as serial-after during policy review when any script below it contains: - -- `CREATE ACCOUNT`, `DROP ACCOUNT`, or `ALTER ACCOUNT` -- `RESTORE ACCOUNT` -- `SHOW ACCOUNTS` -- an `@session` directive with an explicit user or password -- `mo_ctl(...)` -- `mo_feature_registry_*` -- `SET GLOBAL` -- an `@system` command -- `system_metrics` or `mo_debug` -- an explicit `account_id = 0` assumption -- `current_account_id()` or `current_account_name()` -- `mo_catalog.mo_account` -- `KILL CONNECTION` or `KILL QUERY` - -Matching ignores case. Comment matches are intentionally conservative in the first rollout. The scan explains why a directory is serial; it does not split safe-looking files out of that directory. - -## Selection and planning - -MatrixOne's `optools/run_bvt_group.sh` remains the source of truth for complementary groups 0 and 1. - -The orchestrator captures its `-i` selection by invoking it with a temporary no-op `mo-tester/run.sh`. The captured paths are reduced to their top-level directories and passed to the planner. This avoids copying the group mapping into the CI repository. - -The planner then: - -1. validates every selected directory exists immediately below the case root; -2. verifies that every selected script maps to exactly one selected top-level directory; -3. looks up each directory in the explicit policy; -4. defaults an unknown directory to `serial-after` and reports it as unreviewed; -5. verifies the three phases are disjoint and their directory union equals the selected group; -6. assigns whole parallel directories to workers using longest-first balancing; -7. uses an optional `--timings ` input when available and aggregate directory file size as the deterministic fallback weight; -8. writes per-phase and per-worker directory include lists. - -Every `mo-tester -i` argument is therefore a comma-separated list of directories, not hundreds of individual scripts. - -## Runtime architecture - -Each workflow job continues to consume one runner and start one MatrixOne deployment. +### Sys-before ```text -runner -├── MatrixOne deployment -├── serial-before mo-tester (sys) -├── tenant worker 0 mo-tester -├── tenant worker 1 mo-tester -└── serial-after mo-tester (sys) +log +result_count +sql_source_type +statement_query_type +zz_statement_query_type ``` -The default worker count is two and is configurable from one to four. - -### Worker isolation - -Each worker receives: - -- a unique MatrixOne account, `bvtw_`; -- a unique `mo-tester` working directory; -- a copied 12 MiB MatrixOne resource directory; -- its own `mo.yml`, logs, reports, and pprof directory; -- the shared case tree as read-only input. - -The worker's default JDBC user is `:admin`. The sys credentials remain available to `mo-tester` for its internal sync-commit connection, but classification prevents test scripts with explicit credentials or known global SQL from entering a tenant worker. - -### Phase order - -1. Capture the outer BVT group and build the plan. -2. Run serial-before as sys. -3. Create test tenants. -4. Run all tenant workers, each with a directory include list, and wait for every worker. -5. Collect reports and statuses. -6. Drop all test tenants. -7. Run serial-after as sys if MatrixOne is reachable. -8. Merge reports and return failure when any phase failed or was unexpectedly skipped. - -Test tenants are removed before serial-after so account enumeration and recovery tests see the same account state as the serial baseline. +These directories inspect statement and log metadata, so they run before +parallel test traffic. -## Failure handling +### Tenant-parallel -- A serial-before failure prevents tenant workers from starting. -- Parallel worker assertion failures do not terminate sibling workers; all available reports are collected. -- If MatrixOne remains reachable, serial-after still runs after worker failures to maximize diagnostic coverage. -- If MatrixOne is unreachable, serial-after is recorded as skipped and the job fails. -- A shell trap attempts tenant cleanup on success, failure, timeout, and cancellation. -- Cleanup targets only the exact `bvtw_` accounts created by the current process. -- The final cleanup check queries `mo_catalog.mo_account`; any leaked worker account fails the job. +Group 0: -Artifacts include: - -- `plan.json` -- `inventory.tsv` -- per-phase and per-worker logs -- original `mo-tester` reports -- merged summary and timing table -- cleanup status - -## Workflow integration and rollout - -The reusable workflows gain: - -- `tenant_parallel_enabled`, default `false` -- `tenant_parallel_workers`, default `2` - -When disabled, the existing `run_bvt_group.sh` invocation is unchanged. When enabled, the workflow calls `scripts/run_bvt_tenant_parallel.sh`. - -Validation uses a MatrixOne workflow pinned to the CI branch commit SHA. The serial baseline and tenant-parallel candidate run in separate jobs and separate MatrixOne deployments. - -Rollout gates: - -1. planner and shell tests pass; -2. a curated integration subset proves phase barriers, tenant isolation, failure reporting, and cleanup; -3. full shadow runs complete at least ten times with no new concurrency-caused failure; -4. selected-script union and result coverage match the serial baseline; -5. no worker account or resource output leaks; -6. no new MatrixOne crash, OOM, or restart; -7. the measured wall-clock improvement justifies enabling the feature; the previous 30% target is no longer assumed because directory-level classification leaves only 215 scripts parallel. - -After the gates pass, callers enable tenant parallelism. The disabled path remains available for immediate fallback. - -## Test strategy - -`scripts/test_bvt_tenant_plan.py` covers: - -- directory phase lookup and reason reporting; -- rejection of file-level policy entries; -- directory union/disjoint validation; -- unknown-directory serial fallback; -- runtime whole-directory downgrade when a parallel candidate matches a serial rule; -- preservation of all scripts below each selected directory; -- deterministic whole-directory worker balancing; -- malformed policy and path rejection. - -Shell integration tests use fake `mysql` and `mo-tester` commands to verify phase -order, all-worker wait behavior, signal cancellation, exit aggregation, redacted -artifacts, and cleanup. +```text +worker 0: view +worker 1: auto_increment sequence procedure keyword sample pg_cast plugin + time_window union fake_pk dataXtest +``` -Repository validation runs: +Group 1: -- Python unit tests -- `shellcheck` -- `actionlint` -- planner `--dry-run` against a MatrixOne checkout +```text +worker 0: dtype expression comment recursive_cte qexec replace_statement +worker 1: window fulltext operator geo charset_collation distinct udf cte + plan_cache +``` -The cross-repository validation workflow then runs the curated subset and full shadow comparison. +The split is fixed and balances current directory sizes without runtime +planning. Every directory stays on exactly one worker. -## Security and trust +### Sys-after -The workflows already execute the checked-out MatrixOne `optools/run_bvt_group.sh`. Capturing its selection does not broaden that trust boundary. +```text +analyze array benchmark database ddl disttae dml feature_limit foreign_key +function git4data hint iceberg join load_data metadata mo_cloud optimizer +pessimistic_transaction pitr prepare publication_subscription query_result +save_query_result security set snapshot sql_inject stage subquery system +system_variable table task temporary tenant tenxcloud_xx util vector +zz_accesscontrol +``` -Secrets are not written to `plan.json`, inventory, or logs. Generated `mo.yml` files are included in failure artifacts only after password fields are redacted. +`analyze` contains snapshot DDL and `benchmark` contains account/cluster +snapshot DDL, so both require `sys`. + +## Runtime contract + +- The orchestrator accepts a case root, tester directory, group number, + directory configuration, output directory, and optional resource directory. +- It derives the selected group from the explicit directory arrays and applies + the same `cksum % 2` fallback as `run_bvt_group.sh` for unknown directories. +- Every include value is an absolute directory path ending in `/`. +- Each phase gets an isolated tester copy, report directory, log directory, + `mo.yml`, and optional resource copy. +- Worker users are `bvtw_g_w:admin`; account names are unique to + the group and worker. Account creation and deletion use the sys connection. +- Both workers are always waited for. A worker assertion failure does not skip + sibling completion, account cleanup, or reachable sys-after execution. +- On a signal, worker process groups are terminated and waited for before + account deletion. +- Generated `mo.yml` artifacts redact both `password` and `syspass`. +- The final exit status is nonzero when any required phase, worker, cleanup, or + leak check fails. + +## Trial and rollback + +The first trial invokes the reusable workflows from a branch while setting: + +```yaml +tenant_parallel_enabled: true +ci_ref: codex/tenant-parallel-bvt +``` -Account and filesystem cleanup use explicit generated paths and account names; no recursive cleanup accepts an empty or unresolved root. +The job still uses one runner. Its artifacts contain phase logs, reports, +directory includes, and `summary.tsv`. Compare wall-clock time, failures, and +case coverage with an unchanged run from the same MatrixOne commit. Turning the +boolean off immediately restores the original test path. From f781b36893c37ca7f8a6d3f97f068871400f006c Mon Sep 17 00:00:00 2001 From: "Ariznawl@163.com" Date: Wed, 29 Jul 2026 16:22:37 +0800 Subject: [PATCH 13/16] test: cover both tenant BVT groups --- scripts/run_bvt_tenant_parallel.sh | 2 +- scripts/test_run_bvt_tenant_parallel.sh | 39 +++++++++++++++++++++++-- 2 files changed, 38 insertions(+), 3 deletions(-) diff --git a/scripts/run_bvt_tenant_parallel.sh b/scripts/run_bvt_tenant_parallel.sh index dfe5b65..9a2b177 100644 --- a/scripts/run_bvt_tenant_parallel.sh +++ b/scripts/run_bvt_tenant_parallel.sh @@ -264,7 +264,7 @@ selected_directory_count=$( (( selected_directory_count > 0 )) || die "BVT group ${group} has no test directories" -echo "BVT group ${group}: ${selected_directory_count} directories; fixed 2 tenant workers" +echo "BVT directory plan: group ${group}: ${selected_directory_count} directories; fixed 2 tenant workers" summary_file="${output_dir}/summary.tsv" printf 'phase\tname\tstatus\tinclude_file\tlog_file\n' > "${summary_file}" diff --git a/scripts/test_run_bvt_tenant_parallel.sh b/scripts/test_run_bvt_tenant_parallel.sh index 726c9a5..ed78639 100644 --- a/scripts/test_run_bvt_tenant_parallel.sh +++ b/scripts/test_run_bvt_tenant_parallel.sh @@ -131,6 +131,8 @@ setup_fixture() { mysql_log="${fixture_root}/mysql.log" worker_pid_file="${fixture_root}/worker.pid" output_dir="${fixture_root}/output" + run_log="${fixture_root}/orchestrator.log" + fixture_group=1 unknown_directory="unknown_bvt_case" local unknown_group @@ -292,7 +294,7 @@ run_fixture() { bash "${orchestrator}" --tester-dir "${tester_dir}" --case-root "${case_root}" - --group 1 + --group "${fixture_group}" --directories "${directory_config}" --output-dir "${output_dir}" --tenant-password "tenant-secret" @@ -308,8 +310,9 @@ test_successful_direct_commands_and_isolation() { setup_fixture export FAKE_REQUIRE_CONCURRENCY=1 - run_fixture + run_fixture > "${run_log}" + assert_contains "${run_log}" "BVT directory plan: group 1:" assert_file "${output_dir}/summary.tsv" assert_file "${output_dir}/serial-before.include" assert_file "${output_dir}/worker-0.include" @@ -354,6 +357,37 @@ test_successful_direct_commands_and_isolation() { unset FAKE_REQUIRE_CONCURRENCY } +test_group_0_direct_commands_skip_empty_serial_before() { + setup_fixture + fixture_group=0 + mkdir -p \ + "${case_root}/view" \ + "${case_root}/auto_increment" \ + "${case_root}/benchmark" + printf 'select 7;\n' > "${case_root}/view/a.sql" + printf 'select 8;\n' > "${case_root}/auto_increment/a.sql" + printf 'select 9;\n' > "${case_root}/benchmark/a.sql" + + run_fixture > "${run_log}" + + assert_contains "${run_log}" "BVT directory plan: group 0:" + assert_contains "${output_dir}/summary.tsv" \ + $'serial\tserial-before\tskipped' + assert_contains "${output_dir}/worker-0.include" "${case_root}/view/" + assert_contains "${output_dir}/worker-1.include" \ + "${case_root}/auto_increment/" + assert_contains "${output_dir}/serial-after.include" \ + "${case_root}/benchmark/" + assert_contains "${event_log}" "worker-0:start:bvtw_g0_w0:admin:" + assert_contains "${event_log}" "worker-1:start:bvtw_g0_w1:admin:" + assert_not_contains "${event_log}" "serial-before:start:" + assert_order "${event_log}" \ + "create-account" \ + "worker-" \ + "drop-account" \ + "serial-after:start:dump:" +} + test_worker_failure_waits_for_sibling_and_runs_cleanup_and_after() { setup_fixture export FAKE_FAIL_PHASE=worker-0 @@ -469,6 +503,7 @@ run_test() { run_test "static directory contract" test_directory_contract run_test "direct commands and isolation" test_successful_direct_commands_and_isolation +run_test "group 0 direct commands" test_group_0_direct_commands_skip_empty_serial_before run_test "worker failure aggregation" test_worker_failure_waits_for_sibling_and_runs_cleanup_and_after run_test "serial-before barrier" test_serial_before_failure_prevents_accounts_and_workers run_test "leaked account detection" test_leaked_account_fails_before_serial_after From 2aa7d9b4640c540fc02c73cdecafe6f3da69b4a3 Mon Sep 17 00:00:00 2001 From: "Ariznawl@163.com" Date: Wed, 29 Jul 2026 17:07:07 +0800 Subject: [PATCH 14/16] chore: trim tenant BVT pull request --- .../plans/2026-07-29-tenant-parallel-bvt.md | 359 ------------ .../2026-07-28-tenant-parallel-bvt-design.md | 137 ----- scripts/test_bvt_workflow_contract.py | 166 ------ scripts/test_run_bvt_tenant_parallel.sh | 512 ------------------ 4 files changed, 1174 deletions(-) delete mode 100644 docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md delete mode 100644 docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md delete mode 100644 scripts/test_bvt_workflow_contract.py delete mode 100644 scripts/test_run_bvt_tenant_parallel.sh diff --git a/docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md b/docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md deleted file mode 100644 index 7526de6..0000000 --- a/docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md +++ /dev/null @@ -1,359 +0,0 @@ -# Direct Tenant-Parallel BVT Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Run the directory-safe part of each PR BVT group through two direct, -concurrent `mo-tester` commands using ordinary tenants, with all privileged and -global cases serialized as `sys`. - -**Architecture:** A sourced Bash file owns the complete top-level directory -classification and fixed two-worker split. A Bash orchestrator prepares -isolated tester/resource copies, invokes `run.sh -p CASE_ROOT -i DIRS` for each -phase, manages two accounts, and aggregates failures. The two reusable -workflows select this path behind an opt-in boolean and otherwise retain their -existing group runner. - -**Tech Stack:** Bash, MySQL CLI, `mo-tester`, Python standard-library workflow -contract tests, GitHub Actions YAML. - -## Global Constraints - -- One workflow job uses exactly one runner and one MatrixOne deployment. -- The scheduling unit is one immediate child directory of - `test/distributed/cases`; no test-file list is maintained. -- The parallel worker count is fixed at exactly `2`. -- Account-management, tenant, snapshot/PITR, publication, explicit-credential, - cluster-global, task, failpoint, and system behavior runs as `sys`. -- Unknown directories run in `sys-after`; `optimistic` remains excluded. -- Include values are comma-separated absolute directory paths ending in `/`. -- Each concurrent invocation has isolated tester, report, log, and resource - directories. -- Workers are waited for before accounts are dropped, including failure and - signal paths. -- `tenant_parallel_enabled` defaults to `false`; the legacy command is - unchanged when disabled. - ---- - -### Task 1: Replace dynamic planning with a static directory contract - -**Files:** -- Create: `scripts/bvt_tenant_directories.sh` -- Delete: `scripts/bvt_tenant_plan.py` -- Delete: `scripts/bvt_tenant_policy.json` -- Delete: `scripts/test_bvt_tenant_plan.py` -- Modify: `scripts/test_run_bvt_tenant_parallel.sh` - -**Interfaces:** -- Consumes: global variable `bvt_group` set to `0` or `1`, plus a case root - supplied to helper functions by the orchestrator. -- Produces arrays `bvt_serial_before`, `bvt_worker_0`, `bvt_worker_1`, - `bvt_serial_after`, and `bvt_excluded`. -- Produces function `bvt_group_for_directory NAME` that prints `0` or `1` using - the explicit MatrixOne group lists and `cksum % 2` for unknown names. - -- [ ] **Step 1: Write the failing directory-contract test** - -Add a shell test that sources the configuration for groups 0 and 1 and asserts: - -```bash -assert_array_contains bvt_worker_0 view -assert_array_contains bvt_worker_1 auto_increment -assert_array_contains bvt_serial_after analyze -assert_array_contains bvt_serial_after benchmark -assert_array_not_contains bvt_worker_0 analyze -assert_array_not_contains bvt_worker_1 benchmark -assert_all_unique_selected_directories -``` - -Also construct a case root with an unknown top-level directory and assert the -orchestrator writes it to `serial-after.include`, never a worker include. - -- [ ] **Step 2: Run the test to verify RED** - -Run: - -```bash -bash scripts/test_run_bvt_tenant_parallel.sh -``` - -Expected: fail because `scripts/bvt_tenant_directories.sh` does not exist and -the old orchestrator still requires `--planner`, `--policy`, and `--workers`. - -- [ ] **Step 3: Add the directory arrays and group fallback** - -Define literal arrays: - -```bash -bvt_serial_before_all=( - log result_count sql_source_type statement_query_type zz_statement_query_type -) -bvt_parallel_group_0_worker_0=(view) -bvt_parallel_group_0_worker_1=( - auto_increment sequence procedure keyword sample pg_cast plugin - time_window union fake_pk dataXtest -) -bvt_parallel_group_1_worker_0=( - dtype expression comment recursive_cte qexec replace_statement -) -bvt_parallel_group_1_worker_1=( - window fulltext operator geo charset_collation distinct udf cte plan_cache -) -``` - -Add the complete 40-directory sys-after list, the two explicit -`run_bvt_group.sh` directory lists, and `optimistic`. Build phase arrays by -filtering every literal directory through `bvt_group_for_directory`. - -- [ ] **Step 4: Run the directory-contract test to verify GREEN** - -Run: - -```bash -bash scripts/test_run_bvt_tenant_parallel.sh directory_contract -``` - -Expected: the directory contract scenario passes. - -- [ ] **Step 5: Commit** - -```bash -git add scripts/bvt_tenant_directories.sh scripts/test_run_bvt_tenant_parallel.sh -git rm scripts/bvt_tenant_plan.py scripts/bvt_tenant_policy.json scripts/test_bvt_tenant_plan.py -git commit -m "refactor: define BVT tenant execution by directory" -``` - ---- - -### Task 2: Rewrite the direct mo-tester orchestrator - -**Files:** -- Modify: `scripts/run_bvt_tenant_parallel.sh` -- Modify: `scripts/test_run_bvt_tenant_parallel.sh` - -**Interfaces:** -- Consumes `--tester-dir`, `--case-root`, `--group 0|1`, `--directories`, - `--output-dir`, optional `--resource-dir`, MySQL connection flags, and - `--tenant-password`. -- Runs a phase using: - -```bash -./run.sh -n -g -o -p "${case_root}" -i "${include_value}" -``` - -- Produces `serial-before.include`, `worker-0.include`, `worker-1.include`, - `serial-after.include`, per-phase tester/report/log trees, and `summary.tsv`. - -- [ ] **Step 1: Rewrite the fake tester assertions and verify RED** - -Require exact directory-based invocation, fixed worker users, two concurrently -started workers, isolated resources, config redaction, and this event order: - -```text -serial-before -create-account (twice) -worker-0 and worker-1 -drop-account (twice) -serial-after -``` - -Add failure scenarios for serial-before, one worker, cleanup, leak detection, -and TERM while one worker is blocked. - -Run: - -```bash -bash scripts/test_run_bvt_tenant_parallel.sh -``` - -Expected: failures show the old planner CLI and capture logic do not satisfy the -new interface. - -- [ ] **Step 2: Implement validation and include generation** - -Source only the explicit `--directories` file after validating it is a regular -file. Validate all configured names with: - -```bash -[[ "${name}" =~ ^[A-Za-z0-9_]+$ ]] || - die "invalid case directory name: ${name}" -``` - -Discover immediate case-root directories, ignore `optimistic`, append unknown -directories belonging to the requested group to sys-after, and write absolute -paths with trailing `/` joined by commas. - -- [ ] **Step 3: Implement isolated phase execution** - -Copy the tester runtime files and `lib/` into -`output/phases//tester`, create fresh `log/` and `report/`, and copy the -resource tree to `output/phases//resources` when supplied. Change only -the worker `user.name` and `user.password` fields before running the worker. - -- [ ] **Step 4: Implement account lifecycle and failure aggregation** - -Create: - -```sql -CREATE ACCOUNT IF NOT EXISTS `bvtw_g0_w0` - ADMIN_NAME 'admin' IDENTIFIED BY '111'; -``` - -for both workers, launch each worker in its own process group, wait for both, -drop and leak-check both accounts, then run sys-after when MySQL answers -`SELECT 1`. Record every status in `summary.tsv`; return nonzero if any required -step failed. - -- [ ] **Step 5: Implement signal cleanup and artifact redaction** - -On `INT`, `TERM`, or `EXIT`, terminate and wait for live worker process groups -before dropping recorded accounts. Rewrite both `password:` and `syspass:` -values in every generated `mo.yml` to `"***"`. - -- [ ] **Step 6: Run shell verification** - -Run: - -```bash -bash -n scripts/run_bvt_tenant_parallel.sh -bash -n scripts/bvt_tenant_directories.sh -bash scripts/test_run_bvt_tenant_parallel.sh -shellcheck scripts/run_bvt_tenant_parallel.sh scripts/bvt_tenant_directories.sh scripts/test_run_bvt_tenant_parallel.sh -``` - -Expected: syntax checks, every shell scenario, and ShellCheck pass. - -- [ ] **Step 7: Commit** - -```bash -git add scripts/run_bvt_tenant_parallel.sh scripts/test_run_bvt_tenant_parallel.sh -git commit -m "feat: run BVT directories in two tenant processes" -``` - ---- - -### Task 3: Simplify reusable workflow integration - -**Files:** -- Modify: `.github/workflows/e2e-compose-parallel.yaml` -- Modify: `.github/workflows/e2e-standalone-parallel.yaml` -- Modify: `scripts/test_bvt_workflow_contract.py` - -**Interfaces:** -- Workflow inputs: `tenant_parallel_enabled` boolean defaulting to `false`, and - `ci_ref` string defaulting to `main`. -- The enabled branch passes `--directories` and no planner, policy, group - runner, or worker-count arguments. - -- [ ] **Step 1: Update workflow tests and verify RED** - -Assert `tenant_parallel_workers` is absent and each enabled command contains: - -```text ---directories "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/bvt_tenant_directories.sh" -``` - -Assert `--planner`, `--policy`, `--group-runner`, and `--workers` are absent, -while the `else` branch still invokes the original BVT runner. - -Run: - -```bash -python3 -m unittest scripts/test_bvt_workflow_contract.py -v -``` - -Expected: tests fail against the old workflow arguments. - -- [ ] **Step 2: Modify both workflows** - -Remove the worker-count input. Keep the feature default disabled. Replace the -old orchestrator arguments with `--directories`, retaining the correct -compose/standalone case root, tester root, output directory, group, and -standalone resource directory. - -- [ ] **Step 3: Verify workflow syntax and tests** - -Run: - -```bash -python3 -m unittest scripts/test_bvt_workflow_contract.py -v -ruby -e 'require "yaml"; ARGV.each { |path| YAML.load_file(path) }' \ - .github/workflows/e2e-compose-parallel.yaml \ - .github/workflows/e2e-standalone-parallel.yaml -``` - -Expected: all tests pass and Ruby parses both YAML documents. - -- [ ] **Step 4: Commit** - -```bash -git add .github/workflows/e2e-compose-parallel.yaml \ - .github/workflows/e2e-standalone-parallel.yaml \ - scripts/test_bvt_workflow_contract.py -git commit -m "ci: use fixed two-tenant BVT execution" -``` - ---- - -### Task 4: Audit, trial instructions, and PR update - -**Files:** -- Modify: `docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md` -- Modify: `docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md` - -**Interfaces:** -- Produces a reviewable directory inventory and an opt-in branch trial command. - -- [ ] **Step 1: Verify the directory inventory against MatrixOne main** - -Export MatrixOne main, enumerate immediate case directories, and verify: - -```text -sys-before: 5 directories / 29 scripts -tenant-parallel: 27 directories / 175 scripts -sys-after: 40 directories / 935 scripts -excluded optimistic: 1 directory / 22 scripts -``` - -Verify `analyze` and `benchmark` occur only in sys-after and every non-excluded -directory belongs to exactly one phase. - -- [ ] **Step 2: Run the full local suite** - -Run: - -```bash -python3 -m unittest scripts/test_bvt_workflow_contract.py -v -bash scripts/test_run_bvt_tenant_parallel.sh -bash -n scripts/run_bvt_tenant_parallel.sh scripts/bvt_tenant_directories.sh -shellcheck scripts/run_bvt_tenant_parallel.sh scripts/bvt_tenant_directories.sh scripts/test_run_bvt_tenant_parallel.sh -git diff --check -``` - -Expected: all checks pass. - -- [ ] **Step 3: Commit documentation** - -```bash -git add docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md \ - docs/superpowers/plans/2026-07-29-tenant-parallel-bvt.md -git commit -m "docs: simplify tenant-parallel BVT rollout" -``` - -- [ ] **Step 4: Push and update Draft PR #409** - -Push `codex/tenant-parallel-bvt`, update the PR body with the direct-command -architecture, and keep the PR in Draft while the opt-in trial is running. - -- [ ] **Step 5: Trigger the shadow trial** - -From a MatrixOne branch, call the reusable workflow with: - -```yaml -tenant_parallel_enabled: true -ci_ref: codex/tenant-parallel-bvt -``` - -Compare its duration, failed cases, `summary.tsv`, and worker reports against an -unchanged run from the same MatrixOne commit. Do not merge or enable the new -path by default until coverage and cleanup are confirmed. diff --git a/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md b/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md deleted file mode 100644 index 3c42c38..0000000 --- a/docs/superpowers/specs/2026-07-28-tenant-parallel-bvt-design.md +++ /dev/null @@ -1,137 +0,0 @@ -# Direct Tenant-Parallel BVT Design - -## Decision - -Keep one GitHub Actions runner and one MatrixOne deployment per existing BVT -group. Inside that runner, invoke `mo-tester/run.sh` directly: - -1. run ordering-sensitive directories as `sys`; -2. create two ordinary accounts; -3. run two isolated `mo-tester` processes concurrently, one account per process; -4. wait for both processes and drop both accounts; -5. run the remaining directories as `sys`. - -This replaces the Python planner and JSON policy. Directory lists are explicit -Bash arrays, and each `mo-tester -i` argument contains directory paths rather -than individual test files. - -The new path remains opt-in through `tenant_parallel_enabled: false`. When it -is disabled, each workflow runs its existing `run_bvt_group.sh` command -unchanged. - -## Why CI-level processes - -`mo-tester` accepts one case root through `-p` and comma-separated include -substrings through `-i`. An include ending in `/` safely selects a whole -directory. Its dormant in-process parallel code is not suitable because it is -disabled, fixed to two threads, and uses a hard-coded account. Separate -processes also require separate `mo-tester` and resource directories because -each invocation cleans databases and writes fixed report paths. - -## Directory rule - -The immediate child directory of `test/distributed/cases` is the scheduling -unit. If any case in a directory needs account-management privileges, -cross-account state, cluster-global state, snapshots/PITR, publication state, -explicit credentials, system tables, failpoints, tasks, or external shared -state, the whole directory runs serially as `sys`. - -Unknown future directories are assigned to the same complementary BVT group as -`run_bvt_group.sh` and run in `sys-after`. `optimistic` remains excluded, -matching the current group runner. - -The classification was rescanned against MatrixOne main -`f6dab28046d70412cec132f0068840896852101c`: - -| Phase | Directories | Scripts | -|---|---:|---:| -| sys-before | 5 | 29 | -| ordinary-tenant parallel | 27 | 175 | -| sys-after | 40 | 935 | -| excluded `optimistic` | 1 | 22 | - -Runtime improvement must be measured in the opt-in trial because only about -15% of selected scripts are initially safe enough for ordinary tenants. - -## Fixed classification - -### Sys-before - -```text -log -result_count -sql_source_type -statement_query_type -zz_statement_query_type -``` - -These directories inspect statement and log metadata, so they run before -parallel test traffic. - -### Tenant-parallel - -Group 0: - -```text -worker 0: view -worker 1: auto_increment sequence procedure keyword sample pg_cast plugin - time_window union fake_pk dataXtest -``` - -Group 1: - -```text -worker 0: dtype expression comment recursive_cte qexec replace_statement -worker 1: window fulltext operator geo charset_collation distinct udf cte - plan_cache -``` - -The split is fixed and balances current directory sizes without runtime -planning. Every directory stays on exactly one worker. - -### Sys-after - -```text -analyze array benchmark database ddl disttae dml feature_limit foreign_key -function git4data hint iceberg join load_data metadata mo_cloud optimizer -pessimistic_transaction pitr prepare publication_subscription query_result -save_query_result security set snapshot sql_inject stage subquery system -system_variable table task temporary tenant tenxcloud_xx util vector -zz_accesscontrol -``` - -`analyze` contains snapshot DDL and `benchmark` contains account/cluster -snapshot DDL, so both require `sys`. - -## Runtime contract - -- The orchestrator accepts a case root, tester directory, group number, - directory configuration, output directory, and optional resource directory. -- It derives the selected group from the explicit directory arrays and applies - the same `cksum % 2` fallback as `run_bvt_group.sh` for unknown directories. -- Every include value is an absolute directory path ending in `/`. -- Each phase gets an isolated tester copy, report directory, log directory, - `mo.yml`, and optional resource copy. -- Worker users are `bvtw_g_w:admin`; account names are unique to - the group and worker. Account creation and deletion use the sys connection. -- Both workers are always waited for. A worker assertion failure does not skip - sibling completion, account cleanup, or reachable sys-after execution. -- On a signal, worker process groups are terminated and waited for before - account deletion. -- Generated `mo.yml` artifacts redact both `password` and `syspass`. -- The final exit status is nonzero when any required phase, worker, cleanup, or - leak check fails. - -## Trial and rollback - -The first trial invokes the reusable workflows from a branch while setting: - -```yaml -tenant_parallel_enabled: true -ci_ref: codex/tenant-parallel-bvt -``` - -The job still uses one runner. Its artifacts contain phase logs, reports, -directory includes, and `summary.tsv`. Compare wall-clock time, failures, and -case coverage with an unchanged run from the same MatrixOne commit. Turning the -boolean off immediately restores the original test path. diff --git a/scripts/test_bvt_workflow_contract.py b/scripts/test_bvt_workflow_contract.py deleted file mode 100644 index a725e02..0000000 --- a/scripts/test_bvt_workflow_contract.py +++ /dev/null @@ -1,166 +0,0 @@ -import json -import subprocess -import unittest -from pathlib import Path - - -REPOSITORY_ROOT = Path(__file__).resolve().parents[1] - - -def load_yaml(path): - ruby = r""" -require "json" -require "yaml" -puts JSON.generate(YAML.load_file(ARGV.fetch(0))) -""" - result = subprocess.run( - ["ruby", "-e", ruby, str(path)], - check=True, - capture_output=True, - text=True, - ) - return json.loads(result.stdout) - - -class WorkflowContract: - def __init__(self, test_case, filename, job_name): - self.test_case = test_case - self.document = load_yaml(REPOSITORY_ROOT / ".github" / "workflows" / filename) - workflow_on = self.document.get("on") or self.document.get("true") - self.inputs = workflow_on["workflow_call"]["inputs"] - self.steps = self.document["jobs"][job_name]["steps"] - - def step(self, name): - matches = [step for step in self.steps if step.get("name") == name] - self.test_case.assertEqual( - len(matches), - 1, - f"expected one step named {name!r}, found {len(matches)}", - ) - return matches[0] - - def assert_inputs(self): - expected = { - "tenant_parallel_enabled": { - "description": "Run directory-safe BVT cases in tenant workers", - "required": False, - "type": "boolean", - "default": False, - }, - "ci_ref": { - "description": "matrixorigin/CI ref containing tenant BVT scripts", - "required": False, - "type": "string", - "default": "main", - }, - } - for name, value in expected.items(): - self.test_case.assertIn(name, self.inputs) - self.test_case.assertEqual(self.inputs[name], value) - self.test_case.assertNotIn("tenant_parallel_workers", self.inputs) - - def assert_ci_checkout(self): - step = self.step("Checkout tenant-parallel BVT scripts") - self.test_case.assertEqual(step["if"], "${{ inputs.tenant_parallel_enabled }}") - self.test_case.assertEqual( - step["uses"], - "actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10", - ) - self.test_case.assertEqual(step["with"]["repository"], "matrixorigin/CI") - self.test_case.assertEqual(step["with"]["ref"], "${{ inputs.ci_ref }}") - self.test_case.assertEqual(step["with"]["path"], ".ci/tenant-parallel") - - -class TestComposeWorkflow(unittest.TestCase): - def setUp(self): - self.workflow = WorkflowContract( - self, - "e2e-compose-parallel.yaml", - "multi-CN-bvt-docker-compose-proxy", - ) - - def test_inputs_and_ci_checkout(self): - self.workflow.assert_inputs() - self.workflow.assert_ci_checkout() - - def test_start_step_has_opt_in_and_fallback_branches(self): - run = self.workflow.step("Start BVT Test")["run"] - - self.assertIn( - "if [ '${{ inputs.tenant_parallel_enabled }}' = 'true' ]; then", - run, - ) - self.assertIn( - 'bash "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/run_bvt_tenant_parallel.sh"', - run, - ) - self.assertIn('--case-root "$GITHUB_WORKSPACE/test/distributed/cases"', run) - self.assertNotIn("--resource-dir", run) - self.assertIn( - '--directories "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/' - 'bvt_tenant_directories.sh"', - run, - ) - self.assertNotIn("--workers", run) - self.assertNotIn("--planner", run) - self.assertNotIn("--policy", run) - self.assertNotIn("--group-runner", run) - self.assertIn('else\n bash "${bvt_runner}"', run) - - def test_artifact_contains_tenant_output(self): - path = self.workflow.step("Upload Compose BVT execution log")["with"]["path"] - - self.assertIn("${{ runner.temp }}/bvt-compose.log", path) - self.assertIn("${{ runner.temp }}/bvt-tenant-compose", path) - - -class TestStandaloneWorkflow(unittest.TestCase): - def setUp(self): - self.workflow = WorkflowContract( - self, - "e2e-standalone-parallel.yaml", - "pessimistic-bvt-linux-x86", - ) - - def test_inputs_and_ci_checkout(self): - self.workflow.assert_inputs() - self.workflow.assert_ci_checkout() - - def test_start_step_has_opt_in_resource_and_fallback_branches(self): - run = self.workflow.step("Start BVT Test")["run"] - - self.assertIn( - "if [ '${{ inputs.tenant_parallel_enabled }}' = 'true' ]; then", - run, - ) - self.assertIn( - 'bash "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/run_bvt_tenant_parallel.sh"', - run, - ) - self.assertIn('--case-root "$GITHUB_WORKSPACE/head/test/distributed/cases"', run) - self.assertIn( - '--resource-dir "$GITHUB_WORKSPACE/head/test/distributed/resources"', - run, - ) - self.assertIn( - '--directories "$GITHUB_WORKSPACE/.ci/tenant-parallel/scripts/' - 'bvt_tenant_directories.sh"', - run, - ) - self.assertNotIn("--workers", run) - self.assertNotIn("--planner", run) - self.assertNotIn("--policy", run) - self.assertNotIn("--group-runner", run) - self.assertIn('else\n bash "${bvt_runner}"', run) - - def test_artifact_contains_tenant_output(self): - path = self.workflow.step("Upload Launch + Pessimistic BVT execution log")[ - "with" - ]["path"] - - self.assertIn("${{ runner.temp }}/bvt-pessimistic.log", path) - self.assertIn("${{ runner.temp }}/bvt-tenant-pessimistic", path) - - -if __name__ == "__main__": - unittest.main() diff --git a/scripts/test_run_bvt_tenant_parallel.sh b/scripts/test_run_bvt_tenant_parallel.sh deleted file mode 100644 index ed78639..0000000 --- a/scripts/test_run_bvt_tenant_parallel.sh +++ /dev/null @@ -1,512 +0,0 @@ -#!/usr/bin/env bash - -set -euo pipefail - -script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) -orchestrator="${script_dir}/run_bvt_tenant_parallel.sh" -directory_config="${script_dir}/bvt_tenant_directories.sh" - -fail() { - echo "not ok - $*" >&2 - exit 1 -} - -assert_file() { - [[ -f "$1" ]] || fail "expected file: $1" -} - -assert_contains() { - local path=$1 - local expected=$2 - grep -F -- "${expected}" "${path}" >/dev/null || - fail "expected '${expected}' in ${path}" -} - -assert_not_contains() { - local path=$1 - local unexpected=$2 - if grep -F -- "${unexpected}" "${path}" >/dev/null; then - fail "did not expect '${unexpected}' in ${path}" - fi -} - -line_number() { - local path=$1 - local pattern=$2 - grep -n -m1 -F -- "${pattern}" "${path}" | cut -d: -f1 -} - -assert_order() { - local path=$1 - shift - local previous=0 - local pattern current - for pattern in "$@"; do - current=$(line_number "${path}" "${pattern}") - [[ -n "${current}" ]] || fail "missing event '${pattern}'" - (( current > previous )) || - fail "event '${pattern}' was out of order in ${path}" - previous=${current} - done -} - -wait_for_event() { - local path=$1 - local expected=$2 - local attempt=0 - while (( attempt < 100 )); do - if grep -F -- "${expected}" "${path}" >/dev/null; then - return 0 - fi - sleep 0.05 - ((attempt+=1)) - done - return 1 -} - -array_contains() { - local expected=$1 - shift - local value - for value in "$@"; do - [[ "${value}" == "${expected}" ]] && return 0 - done - return 1 -} - -test_directory_contract() { - local bvt_group=0 - # shellcheck source=/dev/null - source "${directory_config}" - array_contains view "${bvt_worker_0[@]}" || - fail "group 0 worker 0 must contain view" - array_contains auto_increment "${bvt_worker_1[@]}" || - fail "group 0 worker 1 must contain auto_increment" - array_contains benchmark "${bvt_serial_after[@]}" || - fail "group 0 sys-after must contain benchmark" - ! array_contains benchmark "${bvt_worker_0[@]}" || - fail "benchmark must not run in an ordinary tenant" - ! array_contains benchmark "${bvt_worker_1[@]}" || - fail "benchmark must not run in an ordinary tenant" - - local -a group_0_selected=( - "${bvt_serial_before[@]}" - "${bvt_worker_0[@]}" - "${bvt_worker_1[@]}" - "${bvt_serial_after[@]}" - ) - local duplicate - duplicate=$(printf '%s\n' "${group_0_selected[@]}" | sort | uniq -d | head -n 1) - [[ -z "${duplicate}" ]] || - fail "group 0 directory appears in more than one phase: ${duplicate}" - - bvt_group=1 - # shellcheck source=/dev/null - source "${directory_config}" - array_contains analyze "${bvt_serial_after[@]}" || - fail "group 1 sys-after must contain analyze" - ! array_contains analyze "${bvt_worker_0[@]}" || - fail "analyze must not run in an ordinary tenant" - ! array_contains analyze "${bvt_worker_1[@]}" || - fail "analyze must not run in an ordinary tenant" - - local -a group_1_selected=( - "${bvt_serial_before[@]}" - "${bvt_worker_0[@]}" - "${bvt_worker_1[@]}" - "${bvt_serial_after[@]}" - ) - duplicate=$(printf '%s\n' "${group_1_selected[@]}" | sort | uniq -d | head -n 1) - [[ -z "${duplicate}" ]] || - fail "group 1 directory appears in more than one phase: ${duplicate}" -} - -setup_fixture() { - fixture_root=$(mktemp -d) - case_root="${fixture_root}/cases" - tester_dir="${fixture_root}/mo-tester" - resource_dir="${fixture_root}/resources" - fake_bin="${fixture_root}/bin" - event_log="${fixture_root}/events.log" - mysql_log="${fixture_root}/mysql.log" - worker_pid_file="${fixture_root}/worker.pid" - output_dir="${fixture_root}/output" - run_log="${fixture_root}/orchestrator.log" - fixture_group=1 - - unknown_directory="unknown_bvt_case" - local unknown_group - unknown_group=$( - printf '%s' "${unknown_directory}" | - cksum | - awk '{ print $1 % 2 }' - ) - if [[ "${unknown_group}" != "1" ]]; then - unknown_directory="unknown_bvt_suite" - fi - unknown_group=$( - printf '%s' "${unknown_directory}" | - cksum | - awk '{ print $1 % 2 }' - ) - [[ "${unknown_group}" == "1" ]] || - fail "test fixture needs an unknown directory assigned to group 1" - - mkdir -p \ - "${case_root}/log" \ - "${case_root}/dtype" \ - "${case_root}/window" \ - "${case_root}/analyze" \ - "${case_root}/${unknown_directory}" \ - "${case_root}/optimistic" \ - "${tester_dir}/lib" \ - "${tester_dir}/log" \ - "${tester_dir}/report" \ - "${resource_dir}/nested" \ - "${fake_bin}" - printf 'select 1;\n' > "${case_root}/log/a.sql" - printf 'select 2;\n' > "${case_root}/dtype/a.sql" - printf 'select 3;\n' > "${case_root}/window/a.sql" - printf 'select 4;\n' > "${case_root}/analyze/a.test" - printf 'select 5;\n' > "${case_root}/${unknown_directory}/a.sql" - printf 'select 6;\n' > "${case_root}/optimistic/a.sql" - printf 'resource\n' > "${resource_dir}/nested/payload.txt" - : > "${tester_dir}/lib/fake.jar" - : > "${event_log}" - : > "${mysql_log}" - - printf '%s\n' \ - 'jdbc:' \ - ' server:' \ - ' - addr: "127.0.0.1:6001"' \ - 'user:' \ - ' name: "dump"' \ - ' password: "111"' \ - ' sysuser: "dump"' \ - ' syspass: "111"' > "${tester_dir}/mo.yml" - printf '%s\n' 'method: "run"' > "${tester_dir}/run.yml" - printf '%s\n' 'log4j.rootLogger=INFO' > "${tester_dir}/log4j.properties" - printf '%s\n' 'bootstrap.servers: localhost:9092' > "${tester_dir}/kafka.yml" - printf '%s\n' '#!/usr/bin/env bash' 'exit 0' > "${tester_dir}/pprof.sh" - chmod +x "${tester_dir}/pprof.sh" - - cat > "${tester_dir}/run.sh" <<'TESTER' -#!/usr/bin/env bash -set -euo pipefail -phase=$(basename "$(dirname "$PWD")") -include="" -case_path="" -declare -a received=() -while (( $# > 0 )); do - received+=("$1") - case "$1" in - -i) - received+=("$2") - include=$2 - shift 2 - ;; - -p) - received+=("$2") - case_path=$2 - shift 2 - ;; - -s) - received+=("$2") - shift 2 - ;; - *) - shift - ;; - esac -done -user=$(sed -n 's/^ name: *"\([^"]*\)".*/\1/p' mo.yml) -printf '%s\n' "${phase}:start:${user}:${case_path}:${include}" >> "${FAKE_EVENT_LOG}" -printf '%s\n' "${received[*]}" > received-arguments.txt - -if [[ "${FAKE_REQUIRE_CONCURRENCY:-0}" == "1" && "${phase}" == "worker-0" ]]; then - attempt=0 - while (( attempt < 100 )); do - if grep -F "worker-1:start:" "${FAKE_EVENT_LOG}" >/dev/null; then - break - fi - sleep 0.02 - ((attempt+=1)) - done - (( attempt < 100 )) || exit 19 -fi - -mkdir -p report -printf '%s\n' "${phase}" > report/report.txt -if [[ "${FAKE_FAIL_PHASE:-}" == "${phase}" ]]; then - exit 7 -fi -if [[ "${FAKE_BLOCK_PHASE:-}" == "${phase}" ]]; then - printf '%s\n' "${BASHPID}" > "${FAKE_WORKER_PID_FILE}" - printf '%s\n' "${phase}:blocked" >> "${FAKE_EVENT_LOG}" - trap 'printf "%s\n" "${phase}:stopped" >> "${FAKE_EVENT_LOG}"; exit 143' INT TERM - while true; do - sleep 1 - done -fi -printf '%s\n' "${phase}:done" >> "${FAKE_EVENT_LOG}" -TESTER - chmod +x "${tester_dir}/run.sh" - - cat > "${fake_bin}/mysql" <<'MYSQL' -#!/usr/bin/env bash -set -euo pipefail -sql="${*: -1}" -printf '%s\n' "${sql}" >> "${FAKE_MYSQL_LOG}" -lower=$(printf '%s' "${sql}" | tr '[:upper:]' '[:lower:]') -if [[ "${lower}" == *"create account"* ]]; then - printf '%s\n' "create-account" >> "${FAKE_EVENT_LOG}" -elif [[ "${lower}" == *"drop account"* ]]; then - if [[ -s "${FAKE_WORKER_PID_FILE}" ]] && - kill -0 "$(<"${FAKE_WORKER_PID_FILE}")" 2>/dev/null; then - printf '%s\n' "drop-while-worker-alive" >> "${FAKE_EVENT_LOG}" - fi - printf '%s\n' "drop-account" >> "${FAKE_EVENT_LOG}" -elif [[ "${lower}" == *"select 1"* ]]; then - printf '%s\n' "mysql-ready" >> "${FAKE_EVENT_LOG}" -fi -if [[ "${lower}" == *"count(*)"* ]]; then - printf '%s\n' "${FAKE_LEAK_COUNT:-0}" -fi -if [[ -n "${FAKE_MYSQL_FAIL_MATCH:-}" && "${lower}" == *"${FAKE_MYSQL_FAIL_MATCH}"* ]]; then - exit 9 -fi -MYSQL - chmod +x "${fake_bin}/mysql" -} - -run_fixture() { - local -a command=( - env - "PATH=${fake_bin}:${PATH}" - "FAKE_EVENT_LOG=${event_log}" - "FAKE_MYSQL_LOG=${mysql_log}" - "FAKE_FAIL_PHASE=${FAKE_FAIL_PHASE:-}" - "FAKE_BLOCK_PHASE=${FAKE_BLOCK_PHASE:-}" - "FAKE_REQUIRE_CONCURRENCY=${FAKE_REQUIRE_CONCURRENCY:-0}" - "FAKE_WORKER_PID_FILE=${worker_pid_file}" - "FAKE_LEAK_COUNT=${FAKE_LEAK_COUNT:-0}" - "FAKE_MYSQL_FAIL_MATCH=${FAKE_MYSQL_FAIL_MATCH:-}" - bash "${orchestrator}" - --tester-dir "${tester_dir}" - --case-root "${case_root}" - --group "${fixture_group}" - --directories "${directory_config}" - --output-dir "${output_dir}" - --tenant-password "tenant-secret" - --resource-dir "${resource_dir}" - ) - if [[ "${RUN_FIXTURE_IN_PLACE:-0}" == "1" ]]; then - exec "${command[@]}" - fi - "${command[@]}" -} - -test_successful_direct_commands_and_isolation() { - setup_fixture - export FAKE_REQUIRE_CONCURRENCY=1 - - run_fixture > "${run_log}" - - assert_contains "${run_log}" "BVT directory plan: group 1:" - assert_file "${output_dir}/summary.tsv" - assert_file "${output_dir}/serial-before.include" - assert_file "${output_dir}/worker-0.include" - assert_file "${output_dir}/worker-1.include" - assert_file "${output_dir}/serial-after.include" - assert_file "${output_dir}/phases/worker-0/tester/report/report.txt" - assert_file "${output_dir}/phases/worker-1/tester/report/report.txt" - assert_file "${output_dir}/phases/worker-0/resources/nested/payload.txt" - assert_contains "${output_dir}/worker-0.include" "${case_root}/dtype/" - assert_contains "${output_dir}/worker-1.include" "${case_root}/window/" - assert_contains "${output_dir}/serial-after.include" "${case_root}/analyze/" - assert_contains "${output_dir}/serial-after.include" \ - "${case_root}/${unknown_directory}/" - assert_not_contains "${output_dir}/serial-after.include" "/optimistic/" - assert_not_contains "${output_dir}/worker-0.include" ".sql" - assert_not_contains "${output_dir}/worker-1.include" ".test" - local resolved_case_root - resolved_case_root=$(cd "${case_root}" && pwd -P) - assert_contains \ - "${output_dir}/phases/worker-0/tester/received-arguments.txt" \ - "-n -g -o -p ${resolved_case_root} -i ${resolved_case_root}/dtype/" - assert_contains "${output_dir}/phases/worker-0/tester/mo.yml" \ - 'name: "bvtw_g1_w0:admin"' - assert_contains "${output_dir}/phases/worker-1/tester/mo.yml" \ - 'name: "bvtw_g1_w1:admin"' - assert_not_contains "${output_dir}/phases/worker-0/tester/mo.yml" \ - "tenant-secret" - assert_not_contains "${output_dir}/phases/worker-0/tester/mo.yml" \ - 'syspass: "111"' - assert_contains "${output_dir}/phases/worker-0/tester/mo.yml" \ - 'password: "***"' - assert_contains "${output_dir}/phases/worker-0/tester/mo.yml" \ - 'syspass: "***"' - assert_order "${event_log}" \ - "serial-before:start:dump:" \ - "create-account" \ - "worker-" \ - "drop-account" \ - "serial-after:start:dump:" - assert_contains "${output_dir}/summary.tsv" $'parallel\tworker-0\tpassed' - assert_contains "${output_dir}/summary.tsv" $'parallel\tworker-1\tpassed' - unset FAKE_REQUIRE_CONCURRENCY -} - -test_group_0_direct_commands_skip_empty_serial_before() { - setup_fixture - fixture_group=0 - mkdir -p \ - "${case_root}/view" \ - "${case_root}/auto_increment" \ - "${case_root}/benchmark" - printf 'select 7;\n' > "${case_root}/view/a.sql" - printf 'select 8;\n' > "${case_root}/auto_increment/a.sql" - printf 'select 9;\n' > "${case_root}/benchmark/a.sql" - - run_fixture > "${run_log}" - - assert_contains "${run_log}" "BVT directory plan: group 0:" - assert_contains "${output_dir}/summary.tsv" \ - $'serial\tserial-before\tskipped' - assert_contains "${output_dir}/worker-0.include" "${case_root}/view/" - assert_contains "${output_dir}/worker-1.include" \ - "${case_root}/auto_increment/" - assert_contains "${output_dir}/serial-after.include" \ - "${case_root}/benchmark/" - assert_contains "${event_log}" "worker-0:start:bvtw_g0_w0:admin:" - assert_contains "${event_log}" "worker-1:start:bvtw_g0_w1:admin:" - assert_not_contains "${event_log}" "serial-before:start:" - assert_order "${event_log}" \ - "create-account" \ - "worker-" \ - "drop-account" \ - "serial-after:start:dump:" -} - -test_worker_failure_waits_for_sibling_and_runs_cleanup_and_after() { - setup_fixture - export FAKE_FAIL_PHASE=worker-0 - - if run_fixture; then - fail "worker failure should fail the orchestrator" - fi - - assert_contains "${event_log}" "worker-0:start:bvtw_g1_w0:admin:" - assert_contains "${event_log}" "worker-1:done" - assert_contains "${event_log}" "drop-account" - assert_contains "${event_log}" "serial-after:start:dump:" - assert_contains "${output_dir}/summary.tsv" $'parallel\tworker-0\tfailed' - assert_contains "${output_dir}/summary.tsv" $'parallel\tworker-1\tpassed' - unset FAKE_FAIL_PHASE -} - -test_serial_before_failure_prevents_accounts_and_workers() { - setup_fixture - export FAKE_FAIL_PHASE=serial-before - - if run_fixture; then - fail "serial-before failure should fail the orchestrator" - fi - - assert_contains "${event_log}" "serial-before:start:dump:" - assert_not_contains "${event_log}" "create-account" - assert_not_contains "${event_log}" "worker-0:start:" - assert_not_contains "${event_log}" "worker-1:start:" - unset FAKE_FAIL_PHASE -} - -test_leaked_account_fails_before_serial_after() { - setup_fixture - export FAKE_LEAK_COUNT=1 - - if run_fixture; then - fail "leaked account should fail the orchestrator" - fi - - assert_contains "${event_log}" "drop-account" - assert_not_contains "${event_log}" "serial-after:start:" - unset FAKE_LEAK_COUNT -} - -test_signal_stops_workers_before_cleanup() { - setup_fixture - export FAKE_BLOCK_PHASE=worker-0 - - RUN_FIXTURE_IN_PLACE=1 run_fixture & - local orchestrator_pid=$! - if ! wait_for_event "${event_log}" "worker-0:blocked"; then - kill -TERM "${orchestrator_pid}" 2>/dev/null || true - wait "${orchestrator_pid}" 2>/dev/null || true - fail "worker did not enter blocking phase" - fi - - kill -TERM "${orchestrator_pid}" - local status=0 - wait "${orchestrator_pid}" || status=$? - - local worker_pid - worker_pid=$(<"${worker_pid_file}") - local worker_was_alive=0 - if kill -0 "${worker_pid}" 2>/dev/null; then - worker_was_alive=1 - kill -TERM "${worker_pid}" 2>/dev/null || true - fi - - unset FAKE_BLOCK_PHASE - (( status == 130 )) || fail "expected signal exit 130, got ${status}" - (( worker_was_alive == 0 )) || fail "worker remained alive after exit" - assert_not_contains "${event_log}" "drop-while-worker-alive" - assert_order "${event_log}" "worker-0:blocked" "drop-account" -} - -test_account_creation_failure_prevents_workers() { - setup_fixture - export FAKE_MYSQL_FAIL_MATCH="create account" - - if run_fixture; then - fail "account creation failure should fail the orchestrator" - fi - - unset FAKE_MYSQL_FAIL_MATCH - assert_contains "${output_dir}/summary.tsv" \ - $'parallel\taccount-setup\tfailed' - assert_not_contains "${event_log}" "worker-0:start:" - assert_not_contains "${event_log}" "worker-1:start:" - assert_not_contains "${event_log}" "serial-after:start:" -} - -test_unreachable_matrixone_skips_serial_after() { - setup_fixture - export FAKE_MYSQL_FAIL_MATCH="select 1;" - - if run_fixture; then - fail "MatrixOne readiness failure should fail the orchestrator" - fi - - unset FAKE_MYSQL_FAIL_MATCH - assert_contains "${event_log}" "drop-account" - assert_not_contains "${event_log}" "serial-after:start:" - assert_contains "${output_dir}/summary.tsv" \ - $'serial\tserial-after\tskipped' -} - -run_test() { - local name=$1 - "$2" - echo "ok - ${name}" -} - -run_test "static directory contract" test_directory_contract -run_test "direct commands and isolation" test_successful_direct_commands_and_isolation -run_test "group 0 direct commands" test_group_0_direct_commands_skip_empty_serial_before -run_test "worker failure aggregation" test_worker_failure_waits_for_sibling_and_runs_cleanup_and_after -run_test "serial-before barrier" test_serial_before_failure_prevents_accounts_and_workers -run_test "leaked account detection" test_leaked_account_fails_before_serial_after -run_test "signal cleanup ordering" test_signal_stops_workers_before_cleanup -run_test "account creation barrier" test_account_creation_failure_prevents_workers -run_test "unreachable MatrixOne handling" test_unreachable_matrixone_skips_serial_after From 5b023abe181f15b74dbf2060553e5a9002d18b2e Mon Sep 17 00:00:00 2001 From: "Ariznawl@163.com" Date: Wed, 29 Jul 2026 19:54:15 +0800 Subject: [PATCH 15/16] ci: fall back to GitHub token for BVT checkout --- .github/workflows/e2e-compose-parallel.yaml | 2 +- .github/workflows/e2e-standalone-parallel.yaml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/e2e-compose-parallel.yaml b/.github/workflows/e2e-compose-parallel.yaml index dcae3bd..1681c42 100644 --- a/.github/workflows/e2e-compose-parallel.yaml +++ b/.github/workflows/e2e-compose-parallel.yaml @@ -220,7 +220,7 @@ jobs: - name: checkout uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 with: - token: ${{ secrets.TOKEN_ACTION }} + token: ${{ secrets.TOKEN_ACTION || github.token }} fetch-depth: "3" repository: ${{ github.event.pull_request.head.repo.full_name }} ref: ${{ github.event.pull_request.head.sha }} diff --git a/.github/workflows/e2e-standalone-parallel.yaml b/.github/workflows/e2e-standalone-parallel.yaml index 0ca218e..9b23f01 100644 --- a/.github/workflows/e2e-standalone-parallel.yaml +++ b/.github/workflows/e2e-standalone-parallel.yaml @@ -268,7 +268,7 @@ jobs: - name: checkout head uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 with: - token: ${{ secrets.TOKEN_ACTION }} + token: ${{ secrets.TOKEN_ACTION || github.token }} fetch-depth: "3" path: ./head repository: ${{ github.event.pull_request.head.repo.full_name }} From 5139a0841f76fbc6aa9f021700bae4d34200fa12 Mon Sep 17 00:00:00 2001 From: "Ariznawl@163.com" Date: Wed, 29 Jul 2026 21:09:40 +0800 Subject: [PATCH 16/16] fix: preserve BVT phase dependencies --- scripts/bvt_tenant_directories.sh | 4 ++-- scripts/run_bvt_tenant_parallel.sh | 3 +-- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/scripts/bvt_tenant_directories.sh b/scripts/bvt_tenant_directories.sh index 4c2e617..4f00209 100644 --- a/scripts/bvt_tenant_directories.sh +++ b/scripts/bvt_tenant_directories.sh @@ -26,11 +26,9 @@ bvt_group_1_directories=( ) bvt_serial_before_all=( - log result_count sql_source_type statement_query_type - zz_statement_query_type ) bvt_parallel_group_0_worker_0=( @@ -73,6 +71,7 @@ bvt_parallel_group_1_worker_1=( ) bvt_serial_after_all=( + log analyze array benchmark @@ -113,6 +112,7 @@ bvt_serial_after_all=( util vector zz_accesscontrol + zz_statement_query_type ) bvt_excluded=( diff --git a/scripts/run_bvt_tenant_parallel.sh b/scripts/run_bvt_tenant_parallel.sh index 9a2b177..bc6f681 100644 --- a/scripts/run_bvt_tenant_parallel.sh +++ b/scripts/run_bvt_tenant_parallel.sh @@ -400,8 +400,7 @@ prepare_phase() { prepared_resource="" if [[ -n "${resource_dir}" ]]; then - cp -a "${resource_dir}" "${phase_root}/resources" - prepared_resource="${phase_root}/resources" + prepared_resource="${resource_dir}" fi if [[ -n "${account_user}" ]]; then