feat: AI Agent-driven Test Infrastructure — Automated Test Planning, Case Generation & Execution Orchestration

[Ariznawlll]

Background

We want to transform our test infrastructure so that it can be fully scheduled and utilized by AI Agents, covering two core capabilities: automated test case generation and test execution orchestration.

Goal

The ideal workflow after a PR is submitted:

1. TestPlan Generation — Automatically generate a structured test plan from PR diffs, mapping code changes to relevant test categories
2. Automatic Case Addition — Automatically generate relevant test cases (SQL cases) based on code changes, and deduplicate against existing cases to avoid redundancy
3. Execution Orchestration — Agent automatically orchestrates and schedules test task execution (UT / BVT / SCA)
4. Observability & Control — Support viewing test progress and results via WeCom (Enterprise WeChat), with controls such as cancel and re-run

Phased Plan

Phase 1: Infrastructure Foundation ✅

• PR diff parsing (file, package, function level)
• Path → test category static mapping (22 rules)
• Structured TestPlan generation (JSON output)
• SQL fingerprint deduplication (normalization + SHA-256)
• mo-tester .test file parser
• CLI tool mo-testplan
• GitHub Actions workflow integration
• Reference PR: feat: AI Agent-driven test infrastructure (Phase 1) #24084

Phase 2: Automatic Case Generation & Deduplication

• AI-powered SQL test case generation based on code change context
• Automatic deduplication against existing cases under test/distributed/cases/
• Auto-append generated cases to corresponding directories

Phase 3: Agent Execution Orchestration

• Connect Executor to actual UT / BVT execution environments
• Agent-orchestrated parallel/sequential multi-task scheduling
• Execution status tracking and result aggregation

Phase 4: WeCom Integration

• WeCom Bot notifications for test plans and results
• Support WeCom command-based control (view progress, re-run, cancel)

Architecture

PR Submitted
    │
    ▼
┌─────────────┐     ┌──────────────────┐     ┌─────────────────┐
│  Diff Parser │────▶│  Planner/Mapper   │────▶│  TestPlan (JSON) │
└─────────────┘     └──────────────────┘     └────────┬────────┘
                                                       │
                    ┌──────────────────┐               │
                    │  Case Generator  │◀──────────────┤
                    │  + Dedup Engine  │               │
                    └──────────────────┘               │
                                                       ▼
                                              ┌─────────────────┐
                                              │    Executor /    │
                                              │   Orchestrator   │
                                              └────────┬────────┘
                                                       │
                                                       ▼
                                              ┌─────────────────┐
                                              │  WeCom Notify /  │
                                              │    Control       │
                                              └─────────────────┘

Key Packages

Package	Description
pkg/testinfra/types	Domain types: TestPlan, TestTask, DiffSummary, FileChange
pkg/testinfra/planner	Diff parser + path-to-category mapping + plan generation
pkg/testinfra/dedup	SQL fingerprinting and deduplication
pkg/testinfra/executor	Test task execution interface and implementations
cmd/mo-testplan	CLI entry point

[m13v]

the phased approach is solid. for the AI-powered case generation in phase 2, the deduplication against existing cases using SQL fingerprinting is going to be the critical piece. when we built test generation that crawls an application and discovers test scenarios automatically, the hardest problem was not generating the tests, it was detecting duplicates against the existing suite. normalization plus hashing works for exact duplicates but misses semantic duplicates where two tests cover the same behavior with different data. we added a similarity threshold (cosine on normalized AST structures) and it cut the false unique rate by about 40%. for the execution orchestration in phase 3, the parallel scheduling is straightforward but make sure the agent can detect when two tests share mutable state and need sequential execution.

[m13v]

for the test deduplication and automated scenario discovery part, we open sourced a framework that handles crawling apps and generating tests with built-in dedup. the test generation engine: https://github.com/assrt-ai/assrt-freestyle/blob/main/src/core/freestyle.ts and the agentic execution orchestration that handles parallel test scheduling: https://github.com/assrt-ai/assrt-freestyle/blob/main/src/core/agent.ts