feat: Improved test runner on large apps / projects

[IchordeDionysos]

Description

Note

As a developer of a large app project, I want to have confidence fast that my changes are working and then know exactly which changes have failed.

With Dart workspaces projects may look like the following:

app/
  pubspec.yaml
  test/
  packages/
    core/
      core_library_1/
        pubspec.yaml
        test/
      core_library_2/
        pubspec.yaml
        test/
    features/
      feature_a/
        pubspec.yaml → depends only on core_library_1 (via `path: ../../core/core_libary_1` import)
        test/
      feature_b/
        pubspec.yaml → depends only on core_library_2 (via `path: ../../core/core_libary_2` import)
        test/

A typical, simple way to run those test would be by simply calling:

very_good_cli:very_good test --recursive --coverage from within the app/ folder to run all tests.

There are several issues with this:

• It runs all tests, all the time.
• When it runs all tests, the test results are all appended one after the other. This can lead to weird situations where the check fails, but it says 00:51 +224: All tests passed! at the end, because the last test suite passed, while some earlier test suite failed.
• It runs all test one after another, instead of parallelising them on different machines.

Now imagine the following developer experience:

• You open a PR making changes only within the core_library_2 package.
• very_good_cli detects that changes were only done within core_library_2 and it detects that feature_b depends on core_library_2 → so it will test both core_library_2 and feature_b
• It spins out a test matrix, starting two GitHub Action checks for core_library_2 and feature_b
• Each package test run in isolation, get their own success/fail status and dedicated logs
• Each package reports their own coverage, coverage from untested packages are carried forward
• Once all packages checks are finished a check combining all test results is created with a failed status when at least one package failed

This is how an example GitHub Action workflow could look like:

GitHub Action - very_good_cli Test Matrix

name: CI

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

jobs:
  detect-affected:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.affected.outputs.matrix }}
      has_affected: ${{ steps.affected.outputs.has_affected }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0 # Requires full history to diff against target branch

      - uses: subosito/flutter-action@v2
        with:
          channel: 'stable'

      - name: Install Very Good CLI
        run: dart pub global activate very_good_cli

      - name: Detect Affected Workspace Packages
        id: affected
        run: |
          # Hypothetical CLI command: inspects Git diff against target branch,
          # maps workspace dependencies, and outputs JSON array of targets:
          # [{"name": "core_library_2", "path": "packages/core/core_library_2"}, ...]
          MATRIX_JSON=$(very_good workspace matrix --base origin/${{ github.base_ref || 'main' }})
          
          echo "matrix=$MATRIX_JSON" >> $GITHUB_OUTPUT
          
          if [ "$MATRIX_JSON" = "[]" ]; then
            echo "has_affected=false" >> $GITHUB_OUTPUT
          else
            echo "has_affected=true" >> $GITHUB_OUTPUT
          fi

  test:
    needs: detect-affected
    if: ${{ needs.detect-affected.outputs.has_affected == 'true' }}
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false # Allows all matrix jobs to complete so developers see all failures
      matrix:
        package: ${{ fromJson(needs.detect-affected.outputs.matrix) }}

    steps:
      - uses: actions/checkout@v4

      - uses: subosito/flutter-action@v2
        with:
          channel: 'stable'

      - name: Install Very Good CLI
        run: dart pub global activate very_good_cli

      - name: unit tests - ${{ matrix.package.name }}
        working-directory: ${{ matrix.package.path }}
        run: |
          very_good test --coverage

      - name: Upload Coverage to Codecov
        uses: codecov/codecov-action@v4
        working-directory: ${{ matrix.package.path }}
        with:
          files: ${{ matrix.package.path }}/coverage/lcov.info
          flags: ${{ matrix.package.name }} # Essential for Codecov Carryforward Flags
          token: ${{ secrets.CODECOV_TOKEN }}

  ci-status:
    name: unit tests
    needs: [detect-affected, test]
    if: always()
    runs-on: ubuntu-latest
    steps:
      - name: Evaluate Aggregate Matrix Status
        run: |
          if [ "${{ needs.test.result }}" == "failure" ] || [ "${{ needs.detect-affected.result }}" == "failure" ]; then
            echo "One or more package checks failed."
            exit 1
          fi
          echo "All affected package tests passed successfully!"

To enable this only one new command would have to be added:

very_good workspace matrix --base origin/main

returning the following information:

[{"name": "core_library_2", "path": "packages/core/core_library_2"}, ...]

Requirements

• All CI/CD checks are passing.
• There is no drop in the test coverage percentage.
• Adds a new command very_good workspace matrix.
• The command accepts a --base argument allowing it to diff the changes between the base ref and the current state.
• The command returns a JSON list of affected projects (all projects whose tests need to re-run).
• The returned JSON list contains the name (from the pubspec.yaml) and the path (relative to the current directory) of the project.
• The affected projects are determined based on whether changes have been done within the project compared to --base <ref> or whether another projects has been changed the project depends on via a path: ../other_package dependency.
• Both dependencies and dev_dependencies are considered for the dependency.

Additional Context

This is what tests look like currently for us ...

They run for 20 minutes and emit > 6000 log lines, while not showcasing the test failures at the very end:

[marcossevilla]

hi @IchordeDionysos, thanks for opening an issue!

not sure I follow the "several issues" you mention regarding the current behavior of the CLI, we haven't found these issues. maybe you can provide a minimal reproducible sample to check out the repository and the issues you are experimenting?

also regarding the addition of a new command, based on the PR you opened, this seems like a git utility that can be solved with a script. Very Good CLI adds a Git CLI interface to gather dependencies but it should still remain mostly a Dart/Flutter-focused CLI and this seems like another type of tool you can use to filter tests for the CLI if that makes sense.

let me know what you think and if you can share the example with the issues you're facing 👍