feat(pipeline): implement cross-pipeline parallelism#221
Closed
nextlevelshit wants to merge 579 commits intomainfrom
Closed
feat(pipeline): implement cross-pipeline parallelism#221nextlevelshit wants to merge 579 commits intomainfrom
nextlevelshit wants to merge 579 commits intomainfrom
Conversation
Add pipelines that create feature branches with commits, requiring only Claude Code as a dependency (no gh CLI needed): - doc-sync: scan docs for inconsistencies, fix and commit - feature: explore → plan → implement a feature on a branch - auto-fix: investigate bug → fix → verify, commit to branch - implement: lean speckit (plan → build → review) with commits - dead-code: scan → clean → verify dead code removal Each pipeline includes a JSON schema contract for handover validation. All marked release: true for inclusion in default wave init.
…ant pipelines
- Replace inline JSON schemas with contract references in all 11 new pipelines
- Add 4 missing contract schemas (adr-options, categorized-changes, explain-analysis, feature-plan)
- Replace target: /src with target: /project across all pipelines
- Replace hardcoded go test with {{ project.test_command }}
- Normalize prototype.yaml YAML style (remove unnecessary double quotes)
- Delete redundant pipelines: auto-fix (duplicate of hotfix), implement (duplicate of feature), docs (disabled, superseded by doc-sync)
- Delete orphaned contracts: bug-investigation, impl-plan-lean, doc-discovery
- Delete orphaned persona: github-pr-creator
- Sync .wave/ copies and update embed_test.go
The persona file was deleted but createDefaultManifest() still referenced it, causing TestInitOutputValidatesWithWaveValidate to fail in CI when wave validate checked for the missing prompt file.
feat: add 6 zero-dependency pipelines for release bundle
- Update pipeline count from 18 to 19 in README.md (2 occurrences) - Add GitHub adapter section to docs/reference/adapters.md - Add GITHUB_TOKEN/GH_TOKEN to docs/reference/environment.md
Update pipeline count from 19 to 23 and fix stale pipeline references (gh-poor-issues → gh-issue-research, github-issue-enhancer → gh-issue-rewrite).
docs: fix 3 remaining documentation inconsistencies
Generate the run ID once in run.go and pass it to both the display layer and executor via WithRunID, so the TUI header shows the full run identifier (e.g. "gh-issue-rewrite-301b03e5") instead of just the pipeline name.
Add interactive terminal UI when `wave run` is invoked without arguments on a TTY. Uses charmbracelet/huh for a 4-step form: pipeline selection with fuzzy filtering, input prompt with placeholder hints, flag toggles, and command confirmation before execution. - Create internal/tui/ package with pipeline discovery and run selector - Integrate TUI into run command with TTY detection (WAVE_FORCE_TTY support) - Support partial name pre-filtering when pipeline name doesn't match - Handle huh.ErrUserAborted for clean Esc/Ctrl+C exit - Fall back to existing non-interactive behavior for CI/pipes - Comprehensive tests for discovery, filtering, command composition, and integration
fix(display): show pipeline run ID with hash in TUI header
Add Wave-themed huh forms (cyan accent, gray muted text) and display the Wave ASCII logo before pipeline selection. Group input and options into a single form step. Add --output text flag for feature parity.
- Extract pipeline name from workspace dir (strip run ID suffix) so inferWorkspaceStatus finds the correct YAML file - Unify printLogo via tui.WaveLogo() for consistent color across commands - Add --output text case to applySelection - Show selected pipeline name above input/flags form - Print composed command to scrollback after confirmation - Widen list runs table columns and separator - Use standard cyan (color 6) throughout theme for visual consistency
- Combine pipeline select, input, and options into single form group so shift+tab navigates back to pipeline selection - Limit select height to 8 visible items with scroll - White bold titles for form fields and list section headlines - Dimmed descriptions in pipeline and flag option labels - Input text white while typing, cyan when blurred (confirmation feedback) - Align Pipeline/command output with 2-space indent
feat(cli): interactive TUI for wave run pipeline selection
…s, preflight validation Add three new packages and wire them into the pipeline executor: - internal/preflight: Validates pipeline `requires:` dependencies (tools on PATH, skills installed) with auto-install support before execution - internal/worktree: Git worktree lifecycle management for isolated workspace execution (create/remove with concurrent safety) - internal/skill: Discovers and provisions skill command files (.claude/commands/*.md) into workspaces for local slash command access Schema changes: - Manifest: Add `skills` map[string]SkillConfig with install/init/check/ commands_glob fields alongside existing skill_mounts - Pipeline: Add `requires` block (skills + tools), workspace `type` (worktree) and `branch` fields, slash_command exec type with command/args fields Integration: - Executor runs preflight checks before workspace creation - createStepWorkspace() supports type=worktree via git worktree add - Skill commands provisioned into adapter workspace .claude/commands/ - buildStepPrompt() handles exec.type=slash_command via FormatSkillCommandPrompt() - AdapterRunConfig gains SkillCommandsDir field - ClaudeAdapter.prepareWorkspace() copies .md files from skill source All 19 test packages pass with -race. New test coverage includes: preflight (tool/skill checking, auto-install flows), worktree (create/ remove lifecycle, concurrent operations, dirty cleanup), skill (provisioning, glob patterns, command discovery), manifest (skills parsing/validation), adapter (skill commands copying).
…ine (#76) - CLAUDE.md: add skill/, worktree/, preflight/ to file structure - manifest-schema.md: document SkillConfig, pipeline requires block, worktree workspace type, and slash_command exec type - executor.go: add cleanupWorktrees() to remove git worktrees after pipeline completion or failure - feature-worktree.yaml: example pipeline demonstrating requires block and worktree workspace isolation
{{ pipeline_name }} (with spaces) was not being resolved, causing
worktree branch names like "feat/{{ pipeline_name }}" to fail with
invalid branch name errors. Now all built-in variables support both
{{key}} and {{ key }} format, consistent with how {{ input }} is
handled elsewhere in the executor.
Replace manual `git checkout -b` + readwrite mounts with
`workspace.type: worktree` for pipelines that create branches.
This keeps the user's main checkout on their current branch.
Branch names use {{ pipeline_id }} (includes hash suffix) to avoid
collisions between concurrent runs of the same pipeline.
Remove redundant feature-worktree pipeline (feature pipeline now
uses worktrees natively). Sync .wave/pipelines/ with defaults.
feat: skill dependency management, worktree workspaces, and preflight validation
VitePress build was failing with "Invalid end tag" at line 252 due to an orphan </div> with no matching opening tag.
- /concepts/execution → /concepts/pipelines (no execution page exists) - future/ trust-center cross-links: use relative ./paths - future/ use-cases cross-links: use relative ./paths - future/integrations ./index → /guides/ci-cd - use-cases/test-generation: link to /future/use-cases/documentation-generation
…s for IDE support - Default memory.strategy to "fresh" in pipeline loaders (constitutional requirement) - Remove redundant strategy: fresh from all 44 pipeline YAML files (150 lines) - Add JSON Schema (draft-07) for pipeline and manifest YAML formats - Mark feature pipeline as non-release - Sync internal/defaults with .wave/pipelines
…rminal Replace static code-review terminal output with 5 rotating real pipeline demos (code-review, speckit-flow, doc-sync, dead-code, security-scan) using the wave -o text ASCII banner format. Adds typewriter line-by-line reveal, smooth fade transitions, fixed terminal height, and bottom gradient shadow.
Worktree creation failed when the path already existed from a previous failed run. Now prunes stale references and force-removes existing worktrees before attempting creation. Also caps speckit-flow task generation at 20 tasks to keep the implement step within budget.
…, relay Dead code cleanup from the dead-code pipeline's clean step: - Remove unused GenerateTemplate, ProgressCalculator, contains/findSubstring - Remove unused validateAdapters, validatePersonasList, validateSkillMounts wrappers - Remove unused GenerateCheckpoint, ValidateCheckpointFormat and related helpers - Delete rollback.go and counter.go (entirely unused) - Fix relay test files that referenced deleted ValidateCheckpointFormat
The 30-minute default_timeout_minutes was applied as a single context deadline for the entire pipeline. Multi-step pipelines like speckit-flow (8 steps) would exhaust the budget before reaching later steps, causing implement to fail with "context deadline exceeded" after only 3 minutes. Now each step gets its own fresh timeout from the manifest config. The CLI --timeout flag overrides per-step timeout via WithStepTimeout option. Also skip retries when the parent context is already cancelled.
…lidation Preflight now checks that gh is on PATH before running GitHub pipelines, preventing personas from wasting tool calls searching for the binary.
…lidation Preflight now checks that glab is on PATH before running GitLab pipelines.
…lidation Preflight now checks that tea is on PATH before running Gitea pipelines.
…lidation Preflight now checks that bb is on PATH before running Bitbucket pipelines.
Soft instruction to avoid TodoWrite for internal progress tracking, which wastes ~500 tokens per step with no benefit to pipeline output. Hard denial via settings.json is not possible while using --dangerously-skip-permissions.
Replace all Bash(bb ...) tool permissions with Bash(curl ...) and Bash(jq *) for Bitbucket Cloud REST API access. Analyst is read-only (only curl -s* allowed), enhancer can PUT, scoper can POST+PUT, commenter has full curl access for comments and PR creation.
Replace all bb CLI command examples with curl+jq calls against the Bitbucket Cloud REST API v2.0. Document $BB_TOKEN requirement, correct field mappings (content.raw not body, kind not labels), and temp file pattern for JSON payloads.
Replace all bb CLI calls in inline pipeline prompts with curl+jq against Bitbucket Cloud REST API v2.0. Update requires.tools from bb to curl+jq. Fix bb-scope verify-report step to respect bitbucket-analyst read-only permissions.
Replace bb issue view with curl GET + jq in fetch-assess.md. Replace bb pr create/edit with curl POST/PUT in create-pr.md. Add $BB_TOKEN auth header and temp file payload pattern.
The bb CLI no longer exists — Bitbucket personas now use curl+jq. Remove the dead Bash(bb *) deny entries from all GitHub, GitLab, and Gitea persona configs.
Document that perl -pi -e 'next if /pattern/' does NOT delete lines (next skips but -p still prints). Use perl -ni -e 'print unless /pattern/' to actually remove lines.
fix(pipelines): add preflight tool validation and TodoWrite avoidance
…cess Analyst personas were restricted to issue commands only. gh-refresh needs `gh release list`, and all analysts need PR/MR view/list for cross-referencing during scope verification.
Analyst personas are read-only but gh/gl/gt-scope verify-report prompts told them to post comments (gh issue comment, glab issue note, tea issues comment). Rewrite Step 2 to match bb-scope's read-only pattern: include a pre-rendered markdown summary in the output JSON instead.
The tea CLI uses plural subcommands (tea issues edit, tea issues view, etc.). Both wave.yaml and the embedded default had the singular form.
Commenter personas had dangerously permissive Bash(gh/glab/tea *) with empty deny lists. Replace with granular least-privilege permissions matching wave.yaml: specific comment/PR/MR commands only.
Seven dev personas had stale embedded configs missing capabilities needed at runtime (Bash, Glob, Grep, Write). Updated to use broad Bash access with deny lists for destructive ops. Configs are language-agnostic — language-specific customization comes later.
fix/persona permission audit
…e JSON validation Fixes three systematic issues observed in headless Claude Code runs: 1. Write/Edit tools don't exist in headless Claude Code CLI (-p mode). Personas tried Write, got rejected, wasted a turn falling back to Bash. Now normalizeAllowedTools strips Write/Edit entries entirely from both --allowedTools args and settings.json. 2. validateAndCorrectOutput ran on the persona's text response (markdown) instead of the actual JSON artifact file, producing false "[DEBUG] Output validation/correction failed" warnings every step. Removed — the contract validator already validates the artifact file on disk. 3. TodoWrite wasted turns despite base protocol saying "Do not use TodoWrite". Now passed via --disallowedTools to block it at CLI level. Continuation of fix/preflight-and-todowrite (PR #202) which addressed the preflight side of these issues.
fix(adapter): strip Write/Edit tools, disallow TodoWrite, remove false JSON validation
Add tiered execution support to MatrixStrategy, enabling items with dependencies to execute in topologically-sorted tiers. Items within a tier run in parallel; tiers execute sequentially. New MatrixStrategy fields: - item_id_key: dot-path to unique ID field in each item - dependency_key: dot-path to dependency array in each item - child_pipeline: name of pipeline to invoke per item (Phase 2) - input_template: Go template for constructing input (Phase 2) New MatrixResult fields: Skipped, SkipReason, ItemID Implementation uses Kahn's algorithm (BFS topological sort) to compute tiers. Dependency failure propagation skips downstream items but allows unrelated items to continue. 6 new tests covering independent items, linear chains, diamond dependencies, dependency failure propagation, cycle detection, and missing dependency validation.
Add the ability for matrix strategy steps to invoke a full child pipeline per matrix item instead of executing a single step. This enables the gh-implement-epic pipeline to fan out gh-implement runs per subissue. Key additions: - NewChildExecutor() factory on DefaultPipelineExecutor for independent state - ChildPipeline field on MatrixStrategy to name or path a child pipeline - InputTemplate field with Go text/template rendering for item → input - Refactored tieredExecution to accept a matrixWorkerFunc for reuse - 5 new tests: LoadsAndExecutes, InputTemplate, WithTiers, PartialFailure, NotFound
New pipeline that consumes gh-scope output and implements all subissues by fanning out parallel gh-implement child pipeline runs, respecting the dependency ordering from the scope plan. Pipeline steps: - fetch-scope: parse epic's scope comment, extract subissues + deps - implement-subissues: matrix strategy with child_pipeline + dependency tiers - report: post implementation summary comment on parent epic Includes: - Pipeline YAML with matrix strategy using child_pipeline and dependency_key - Prompt files for fetch-scope and report steps - Contract schemas for epic-scope-plan and epic-report - Mock adapter output for testing (gh-implement-epic routing before gh-implement)
Add PipelineBatchExecutor for concurrent execution of multiple independent pipeline DAGs. Uses Kahn's algorithm for dependency-tier computation and errgroup with SetLimit for concurrency control. Key components: - PipelineBatchConfig/Result types with OnFailure policy enum - Tiered execution respecting inter-pipeline dependencies - Two error policies: continue (skip dependents) and abort-all (cancel all) - BatchArtifactRegistry for cross-pipeline artifact path tracking - Per-pipeline progress events (batch_started/completed/failed) - Configurable MaxConcurrentPipelines resource limiting - BatchConfig added to manifest Runtime for default settings Follows the MatrixExecutor composition pattern — no changes to DefaultPipelineExecutor.Execute().
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ExecutePipelineBatchorchestratorCloses #210
Changes
internal/pipeline/batch.go— CoreExecutePipelineBatchimplementation with errgroup-based concurrent execution, resource limiting via semaphore, and configurable error handling (abort-all vs continue-on-failure)internal/pipeline/batch_types.go— Type definitions for batch execution:PipelineBatchConfig,PipelineBatchResult,PipelineRunResult, and error handling policy typesinternal/pipeline/batch_test.go— Comprehensive test coverage for batch execution including concurrent runs, dependency ordering, error propagation, and resource limitsinternal/pipeline/matrix.go— Extended matrix executor with dependency tier support and child pipeline invocationinternal/pipeline/matrix_test.go— Tests for matrix executor dependency tiers and child pipeline spawninginternal/pipeline/executor.go— Integration point exposingExecutePipelineBatchfrom the existing executorinternal/pipeline/types.go— Extended pipeline types with batch-related fieldsinternal/manifest/types.go— New manifest types for cross-pipeline configurationinternal/event/emitter.go— Extended event emitter with pipeline-scoped event emissioninternal/adapter/mock.go— Mock adapter for testing concurrent pipeline execution.wave/pipelines/gh-implement-epic.yaml— Epic pipeline leveraging cross-pipeline parallelismspecs/210-cross-pipeline-parallelism/— Specification, plan, and task tracking artifactsTest Plan
go test ./internal/pipeline/...)executeStepBatch) verified unchanged via existing test suite