Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ repos:
rev: v4.0.1
hooks:
- id: end-of-file-fixer
exclude: ^smart_tests/docs/

- repo: https://github.com/PyCQA/flake8
rev: 7.0.0
Expand Down
24 changes: 24 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
This is a CLI to interact with CloudBees Smart Tests.

# Usage

See https://help.launchableinc.com/resources/cli-reference/ and
Expand Down Expand Up @@ -89,6 +91,13 @@ uv add some-package
uv add --dev some-dev-package
```

## Adding new command
`smart_tests/args4p` defines a small command line argument parsing framework for this CLI. See its README.md for
the capabilities.

`smart_tests/__main__.py` defines the top level commands. Follow the code from there to see where sub-commands
are registered.

## Updating Python Version

When updating the Python version requirement, update the following files:
Expand All @@ -106,6 +115,21 @@ When updating the Python version requirement, update the following files:
Create new release on Github, then Github Actions automatically uploads the
module to PyPI.

## How to update bundled documentation

The product documentation lives in `smart_tests/docs/` and is bundled into the
distributed package. To pull in the latest from the docsite repository:

```shell
uv run poe update-docs
```

Users can extract the bundled docs to the current directory with:

```shell
smart-tests get docs
```

## How to update smart_tests/jar/exe_deploy.jar

```
Expand Down
3 changes: 2 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ lint-warn = "flake8 --count --exit-zero --max-complexity=15 --max-line-length=13
test = "python -m unittest"
test-xml = "python -m test-runner"
type = "mypy smart_tests tests"
update-docs = "bash scripts/update_docs.sh"

[build-system]
requires = ["setuptools>=45", "wheel", "setuptools_scm"]
Expand All @@ -61,6 +62,6 @@ build-backend = "setuptools.build_meta"
packages = ["smart_tests"]

[tool.setuptools.package-data]
smart_tests = ["jar/exe_deploy.jar"]
smart_tests = ["jar/exe_deploy.jar", "docs/**/*"]

[tool.setuptools_scm]
15 changes: 15 additions & 0 deletions scripts/update_docs.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
#!/bin/bash
# Fetch the latest docs from the docsite repository into smart_tests/docs/.
# Run via: uv run poe update-docs
set -euo pipefail

REPO="git@github.com:cloudbees/docsite-cloudbees-smart-tests.git"
DOCS_DST="$(dirname "$0")/../smart_tests/docs"

tmpdir=$(mktemp -d)
trap 'rm -rf "$tmpdir"' EXIT

git clone --depth=1 "$REPO" "$tmpdir"
rm -rf "$DOCS_DST"
cp -r "$tmpdir/docs" "$DOCS_DST"
echo "Docs updated at $DOCS_DST"
2 changes: 2 additions & 0 deletions smart_tests/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
from smart_tests.commands.compare import compare
from smart_tests.commands.detect_flakes import detect_flakes
from smart_tests.commands.gate import gate
from smart_tests.commands.get import get
from smart_tests.commands.inspect import inspect
from smart_tests.commands.record import record
from smart_tests.commands.stats import stats
Expand All @@ -25,6 +26,7 @@
cli.add_command(compare)
cli.add_command(detect_flakes)
cli.add_command(gate)
cli.add_command(get)


def _load_test_runners():
Expand Down
11 changes: 11 additions & 0 deletions smart_tests/commands/get/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
from ... import args4p
from ...app import Application
from .docs import docs


@args4p.group(help="Retrieve resources")
def get(app: Application):
return app


get.add_command(docs)
27 changes: 27 additions & 0 deletions smart_tests/commands/get/docs.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
import importlib.resources
import shutil
import sys
from pathlib import Path

import click

from ... import args4p
from ...app import Application

OUTPUT_DIR = 'smart-tests-docs'


@args4p.command(help="Copy product documentation into ./smart-tests-docs")
def docs(app: Application):
output = Path(OUTPUT_DIR)
if output.exists():
click.secho(
f"'{OUTPUT_DIR}' already exists. Please delete it first, then re-run this command.",
fg='red', err=True,
)
sys.exit(1)

docs_src = Path(str(importlib.resources.files('smart_tests') / 'docs'))
click.echo(f"Copying docs to ./{OUTPUT_DIR} ...")
shutil.copytree(docs_src, output)
click.echo(f"Done. Documentation is in ./{OUTPUT_DIR}")
14 changes: 14 additions & 0 deletions smart_tests/docs/antora.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
name: cloudbees-smart-tests
title: CloudBees Smart Tests
version: latest
start_page: ROOT:index.adoc
nav:
- modules/ROOT/nav.adoc
asciidoc:
attributes:
sectanchors: ""
experimental: ""
toc: ""
toc-title: On this page
toc-placement: manual
toclevels: 1@
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
116 changes: 116 additions & 0 deletions smart_tests/docs/modules/ROOT/nav.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
include::ROOT:partial$abbr.adoc[]

* xref:ROOT:index.adoc[Overview]

* xref:send-data-to-smart-tests:send-data-to-smart-tests.adoc[Send data to {PRODUCT}]
** xref:send-data-to-smart-tests:getting-started/getting-started.adoc[Getting started]
*** xref:send-data-to-smart-tests:getting-started/use-the-cli-with-a-public-repository.adoc[Use the CLI with a public repository]
*** xref:send-data-to-smart-tests:getting-started/migration-to-github-oidc-auth.adoc[Update tokenless authentication to use GitHub OIDC]
*** xref:send-data-to-smart-tests:getting-started/use-the-cli-with-docker-image.adoc[Use the CLI with Docker image]
** xref:send-data-to-smart-tests:record-builds/record-builds.adoc[Record builds]
*** xref:send-data-to-smart-tests:record-builds/choose-a-value-for-build-name.adoc[Choose a value for build name]
*** xref:send-data-to-smart-tests:record-builds/record-builds-from-multiple-repositories.adoc[Record builds from multiple repositories]
*** xref:send-data-to-smart-tests:record-builds/run-under-restricted-networks.adoc[Run under restricted networks]
*** xref:send-data-to-smart-tests:record-builds/deal-with-shallow-clones.adoc[Deal with shallow clones]
** xref:send-data-to-smart-tests:record-sessions/record-sessions.adoc[Record sessions]
** xref:send-data-to-smart-tests:subset/subset-predictive-test-selection.adoc[Subset for Predictive test selection]
** xref:send-data-to-smart-tests:record-test-results/record-test-results.adoc[Record test results]
*** xref:send-data-to-smart-tests:record-test-results/convert-test-reports-to-junit-format.adoc[Convert test reports to JUnit format]
*** xref:send-data-to-smart-tests:record-test-results/ensure-record-tests-always-runs.adoc[Ensure record tests always runs]
*** xref:send-data-to-smart-tests:record-test-results/manage-complex-test-session-layouts.adoc[Manage complex test session layouts]
*** xref:send-data-to-smart-tests:record-test-results/use-flavors-to-run-the-best-tests-for-an-environment.adoc[Use 'flavors' to run the best tests for an environment]
*** xref:send-data-to-smart-tests:record-test-results/separate-out-test-suites.adoc[Separate out test suites]
*** xref:send-data-to-smart-tests:record-test-results/attach-log-files.adoc[Attach log files]

.Features
* xref:features:intelligent-test-failure-diagnostics.adoc[Intelligent Test Failure Diagnostics]
* xref:features:predictive-test-selection.adoc[Predictive Test Selection]
** xref:features:predictive-test-selection/use-cases-for-predictive-test-selection.adoc[Use cases for Predictive Test Selection]
** xref:features:predictive-test-selection/observe-subset-behavior.adoc[Observe subset behavior]
** xref:features:predictive-test-selection/request-and-run-a-subset-of-tests/request-and-run-a-subset-of-tests.adoc[Request and run a subset of tests]
*** xref:features:predictive-test-selection/request-and-run-a-subset-of-tests/choose-a-subset-optimization-target/choose-a-subset-optimization-target.adoc[Choose a subset optimization target]
*** xref:features:predictive-test-selection/request-and-run-a-subset-of-tests/subset-with-the-smart-tests-cli/subset-with-the-smart-tests-cli.adoc[Subset with the CLI]
*** xref:features:predictive-test-selection/request-and-run-a-subset-of-tests/choose-a-subset-optimization-target/smart-subset-optimization-target.adoc[Smart subset optimization targets]
*** xref:features:predictive-test-selection/request-and-run-a-subset-of-tests/subset-with-the-smart-tests-cli/combine-with-rule-based-test-selection.adoc[Combine with rule-based test selection]
*** xref:features:predictive-test-selection/request-and-run-a-subset-of-tests/subset-with-the-smart-tests-cli/replace-static-parallel-suites-dynamic-parallel-subset.adoc[Replace static parallel suites with a dynamic parallel subset]
*** xref:features:predictive-test-selection/request-and-run-a-subset-of-tests/subset-with-the-smart-tests-cli/zero-input-subsetting/zero-input-subsetting.adoc[Zero Input Subsetting]
*** xref:features:predictive-test-selection/request-and-run-a-subset-of-tests/subset-with-the-smart-tests-cli/zero-input-subsetting/use-groups-to-split-subsets.adoc[Use groups to split subsets]
** xref:features:predictive-test-selection/smart-tests-extension-for-VS-code.adoc[{PRODUCT} extension for VS code]
** xref:features:predictive-test-selection/viewing-time-savings.adoc[View time savings]
** xref:features:predictive-test-selection/how-we-select-tests.adoc[How we select tests]
** xref:features:predictive-test-selection/faq.adoc[FAQ]
* xref:features:trends.adoc[Test Suite Trends]
* xref:features:unhealthy-tests.adoc[Unhealthy Test Insights]
* xref:features:test-results-and-reports.adoc[Test Reports]
* xref:features:test-suite-parallelization.adoc[Test Suite Parallelization]
** xref:features:test-suite-parallelization/parallelize-your-test-suite-with-the-smart-tests-cli.adoc[Parallelize your test suite with the {PRODUCT} CLI]
* xref:features:test-notifications.adoc[Test Notifications]
** xref:features:test-notifications/github-app-for-test-sessions.adoc[Github app for test sessions]
** xref:features:test-notifications/test-notifications-via-slack.adoc[Test Notifications via Slack]

.Concepts
* xref:concepts:object-model.adoc[Object model]
* xref:concepts:organization.adoc[Organization]
* xref:concepts:workspace.adoc[Workspace]
* xref:concepts:build.adoc[Build]
* xref:concepts:test-session.adoc[Test Session]
* xref:concepts:subset.adoc[Subset]
* xref:concepts:insight.adoc[Insight]
* xref:concepts:branch.adoc[Branch]
* xref:concepts:test-suite.adoc[Test Suite]

.Resources
* xref:resources:onboarding-guide.adoc[{PRODUCT} Onboarding guide]
* xref:resources:cli-reference.adoc[CLI reference]
* xref:resources:integrations.adoc[Integrations]
** xref:resources:integrations/adb.adoc[Android Debug Bridge (adb)]
** xref:resources:integrations/ant.adoc[Ant]
** xref:resources:integrations/bazel.adoc[Bazel]
** xref:resources:integrations/behave.adoc[Behave]
** xref:resources:integrations/ctest.adoc[CTest]
** xref:resources:integrations/cucumber.adoc[cucumber]
** xref:resources:integrations/cypress.adoc[Cypress]
** xref:resources:integrations/dotnet-test.adoc[dotnet test]
** xref:resources:integrations/use-the-generic-file-based-runner-integration.adoc['file' profile for unsupported test runners]
** xref:resources:integrations/go-test.adoc[Go Test]
** xref:resources:integrations/googletest.adoc[GoogleTest]
** xref:resources:integrations/gradle.adoc[Gradle]
** xref:resources:integrations/jest.adoc[Jest]
** xref:resources:integrations/karma.adoc[Karma]
** xref:resources:integrations/maven.adoc[Maven]
** xref:resources:integrations/minitest.adoc[minitest]
** xref:resources:integrations/playwright.adoc[Playwright]
** xref:resources:integrations/prove.adoc[prove for Perl]
** xref:resources:integrations/pytest.adoc[pytest]
** xref:resources:integrations/raw.adoc['raw' profile for custom test runners]
** xref:resources:integrations/robot.adoc[Robot]
** xref:resources:integrations/rspec.adoc[RSpec]
* xref:resources:supported-languages.adoc[Supported languages]
** xref:resources:supported-languages/c-plus-plus.adoc[C++]
** xref:resources:supported-languages/dotnet.adoc[.NET]
** xref:resources:supported-languages/go.adoc[Go]
** xref:resources:supported-languages/java.adoc[Java]
** xref:resources:supported-languages/javascript.adoc[JavaScript]
** xref:resources:supported-languages/perl.adoc[Perl]
** xref:resources:supported-languages/python.adoc[Python]
** xref:resources:supported-languages/ruby.adoc[Ruby]
* xref:resources:supported-test-frameworks.adoc[Supported test frameworks]
** xref:resources:supported-test-frameworks/appium.adoc[Appium]
** xref:resources:supported-test-frameworks/cucumber.adoc[Cucumber]
** xref:resources:supported-test-frameworks/googletest.adoc[GoogleTest]
** xref:resources:supported-test-frameworks/jest.adoc[Jest]
** xref:resources:supported-test-frameworks/junit.adoc[JUnit]
** xref:resources:supported-test-frameworks/minitest.adoc[minitest]
** xref:resources:supported-test-frameworks/nunit.adoc[NUnit]
** xref:resources:supported-test-frameworks/prove.adoc[prove for Perl]
** xref:resources:supported-test-frameworks/robot.adoc[Robot]
** xref:resources:supported-test-frameworks/rspec.adoc[RSpec]
** xref:resources:supported-test-frameworks/selenium.adoc[Selenium]
** xref:resources:supported-test-frameworks/testng.adoc[TestNG]
* xref:resources:ci-tool-integrations.adoc[CI tool integrations]
** xref:resources:ci-tool-integrations/github-actions.adoc[GitHub Actions]
* xref:resources:policies.adoc[Policies]
** xref:resources:policies/data-privacy-and-protection.adoc[Data privacy and protection]
** xref:resources:policies/data-examples.adoc[Data examples]
** xref:resources:policies/security-policies.adoc[Security policies]
* xref:resources:troubleshooting.adoc[Troubleshoot]
9 changes: 9 additions & 0 deletions smart_tests/docs/modules/ROOT/pages/getting-started.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
include::ROOT:partial$abbr.adoc[]
:slug: getting-started
= Getting Started

After reviewing the xref:ROOT:index.adoc[Product Overview], you can sign up for a {PRODUCT} account at https://app.launchableinc.com/signup[https://app.launchableinc.com/signup].

After creating your user account and verifying your email address (if necessary), you'll be prompted to create an xref:concepts:organization.adoc[Organization].or your company and a xref:concepts:workspace.adoc[Workspace].or your test suite.

Then you can start sending data. For more information, refer to xref:send-data-to-smart-tests:send-data-to-smart-tests.adoc[Send data to {PRODUCT}].
55 changes: 55 additions & 0 deletions smart_tests/docs/modules/ROOT/pages/index.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
= Overview

https://www.launchableinc.com/[{PRODUCT}] is a *software development intelligence platform* focused on continuous integration (CI). Using data from your CI runs, {PRODUCT} provides various features to speed up your testing workflow so you can ship high-quality software faster.

[#predictive-test-selection]
== Predictive test selection

xref:features:predictive-test-selection.adoc[Predictive test selection] uses large language models (LLMs) to understand your code changes and automatically identify the most relevant tests to run.

By analyzing both source code and test files, {PRODUCT} builds a deep understanding of how your code is structured and how different components relate to one another. Using commit information, it calculates the similarity between changed files and test files to determine which tests are most likely impacted — producing an optimized test execution plan where critical tests are prioritized and redundant ones are safely skipped.

With Predictive test selection, {PRODUCT} brings code-aware test execution intelligence to your CI process — helping teams run fewer but relevant tests, gain faster feedback, and ship high-quality software with confidence.

.Predictive test selection data flow
image::ROOT:pts-v2-working.png[Predictive test selection data flow,role="screenshot"]

For more information, refer to xref:features:predictive-test-selection.adoc[Predictive test selection].

[#test-suite-trends]
== Test suite trends

To see aggregate info about your test sessions, including average test session duration, test session frequency, and how often sessions fail refer to xref:features:trends.adoc[test suite trends].

Seeing this data over time gives you a picture of how your test suite evolves; for example, perhaps tests are taking twice as long as they did six months ago, and you need to cut it down! Similarly, perhaps your team's running tests a lot more often than expected, driving up resource costs. Or maybe there are some broken tests that are driving up the overall failure rate.

[#unhealthy-tests]
== Unhealthy tests

{PRODUCT} also surfaces unhealthy tests within your test suite allowing you to fix them and run tests more reliably.

[#test-reports]
== Test reports

As test results are sent to {PRODUCT} using the {PRODUCT} CLI, you can view those xref:features:test-results-and-reports.adoc[test results and reports] in the {PRODUCT} dashboard. {PRODUCT} provides a richer view of test results, helping developers triage and fix failures more quickly.

For quick access, the {PRODUCT} CLI prints out a link to the results view every time you record results. In addition, {PRODUCT} shows all of your test runs in one place for easy navigation. No more digging around build logs.

[#test-suite-parallelization]
== Test suite parallelization

Using duration information from past test runs, {PRODUCT} can automatically create evenly-sized bins of tests for you to run your test suite in parallel. For more information, refer to xref:features:test-suite-parallelization.adoc[test suite parallelization].

[#test-notifications]
== Test Notifications

{PRODUCT} provides test notifications so developers can immediately take action on the results, whether that's to triage failures or merge a PR. Developers can create subscriptions to receive personal notifications about test sessions run against their branches/pull requests or other test sessions they care about.
The following apps are available for receiving notifications:

* GitHub app: xref:features:test-notifications/github-app-for-test-sessions.adoc[GitHub app for test sessions]

* Slack app: xref:features:test-notifications/test-notifications-via-slack.adoc[Test Notifications via Slack]

The apps will send notifications _directly_ to developers eliminating the need to manually check their email or navigate to their pull request to see if their tests passed.

To sign up for a {PRODUCT} account, refer to https://app.launchableinc.com/signup[https://app.launchableinc.com/signup]. Once done, you can start xref:send-data-to-smart-tests:send-data-to-smart-tests.adoc[send data to {PRODUCT}].
11 changes: 11 additions & 0 deletions smart_tests/docs/modules/ROOT/partials/abbr.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
//include user-defined abbreviation for conditional directives for this specific product repo
include::ROOT:partial$conditionals.adoc[]

//include shared abbreviations across all docsite repos
include::docsite-global::partial$abbr.adoc[]

//add additional attributes specific to this component ONLY below this line
//attributes that should be available to all docs and training go in the docsite-global abbr.adoc file

//set the value of {PRODUCT} that should be used throughout this component
:PRODUCT: CloudBees Smart Tests
Empty file.
12 changes: 12 additions & 0 deletions smart_tests/docs/modules/concepts/pages/branch.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
= Branch
:slug: branch

Branch is an attribute of a xref:concepts:build.adoc[build] that tracks which branch of development the build came from. This information helps you focus on the area you want to look into. For example, if you are responsible for the overall quality in the "main" branch, then tests run against builds from the "main" branch are of interest, but tests running in the feature branches might not be.

By default, branch information is obtained automatically from the Git workspace when `smart-tests record build` runs.

== Manual branch configuration

If your build setup is complex, for example if your build spans multiple Git repositories, then you will be asked to record this information explicitly via `smart-tests record build --branch NAME` option.

In these situations, the notion of "branch' starts to deviate from that of Git branch, but conceptually there's still a logical thread of development, and you should be able to name it. Really the key idea here is that when you have parallel threads of developments (such as production vs dev, or version4 vs version5), we'd like to be able to separate them.
Loading
Loading