Skip to content

CLI.md

Latest commit

 

History

History
315 lines (221 loc) · 7.08 KB

CLI.md

File metadata and controls

315 lines (221 loc) · 7.08 KB

ETNA CLI

Overview

ETNA is a CLI for creating experiments, adding workloads, running tests, and managing result data.

etna --help

Top-level commands:

  • experiment — manage experiments and run/visualize tests
  • workload — add/remove/list workloads in an experiment
  • config — show CLI configuration
  • setup — initialize global ETNA config
  • check — integrity checks and cleanup/restore helpers
  • analyze — analysis helpers
  • mutation — inspect and control mutation variants
  • bash — generate bash script from workload config (Unix only)
  • completions — generate shell completions

Most commands operate on an experiment. The experiment is resolved in this order:

  1. explicit --name <NAME> flag (where supported);
  2. current directory — if $PWD is inside a registered experiment, that experiment is used.

If neither resolves and the command requires an experiment, it fails with a message pointing at etna experiment list.

Experiment Commands

etna experiment --help

experiment new

Create a new experiment directory.

Usage: etna experiment new [OPTIONS] <NAME> [PATH]

Options:
  -o, --overwrite

experiment register

Register an existing experiment directory in tracking metadata.

Usage: etna experiment register [NAME] [PATH]

Both arguments are optional: if NAME is omitted it is inferred from the directory, and if PATH is omitted the current directory is used.

experiment run

Run one or more test files from the experiment's tests/ directory.

Usage: etna experiment run [OPTIONS]

Options:
  -n, --name <NAME>
      --tests <TESTS>
  -s, --short-circuit
  -p, --parallel
      --params <KEY=VALUE>

--params may be repeated; values passed here override parameters defined in the test JSON files. --parallel requires the tested run to be effect-free — effectful runs in parallel are not guaranteed to produce deterministic results.

experiment show

Show one experiment by name.

Usage: etna experiment show --name <NAME>

experiment create-test

Create a new test file with default values.

Usage: etna experiment create-test [OPTIONS] --language <LANGUAGE> --workload <WORKLOAD>

Options:
  -n, --name <NAME>
      --language <LANGUAGE>
      --workload <WORKLOAD>
      --test <TEST>                # default: <workload>-<language>
      --trials <TRIALS>            # default: 10
      --timeout <TIMEOUT>          # default: 60 (seconds)
      --mode <MODE>                # default: solve  | solve | sample | test | shrink | cross
      --mutation <MUTATION>        # repeatable

experiment amend-test

Amend an existing test file by assigning a strategy.

Usage: etna experiment amend-test [OPTIONS] --test <TEST> --strategy <STRATEGY>

Options:
  -n, --name <NAME>
      --test <TEST>
      --strategy <STRATEGY>
      --mutation <MUTATION>
      --property <PROPERTY>

Behavior:

  • if the task has no strategy: set it;
  • if the task has a different strategy: duplicate the task with the new strategy;
  • if the task already has the same strategy: no-op.

experiment visualize

Visualize experiment results.

Usage: etna experiment visualize [OPTIONS] --figure <FIGURE>

Options:
      --name <NAME>
      --figure <FIGURE>
  -t, --tests <TESTS>...
  -g, --groupby <GROUPBY>...       # default: language workload strategy mode
  -a, --aggby <AGGBY>...           # default: language workload strategy property mutations mode
  -m, --metric <METRIC>            # default: time     | time | memory | size | coverage
  -b, --buckets <BUCKETS>...       # default: 0.1 1.0 10.0 60.0
      --max <MAX>
  -v, --visualization-type <TYPE>  # default: bucket   | bucket | bar | line
      --hatched <IDX,IDX,...>      # 0-indexed group indices to render with hatching

experiment visualize-json

Render a bucket chart from a pre-computed JSON file.

Usage: etna experiment visualize-json --input <INPUT> --output <OUTPUT>

experiment report

Generate an interactive HTML report for the experiment.

Usage: etna experiment report [OPTIONS]

Options:
  -n, --name <NAME>
  -o, --output <OUTPUT>       # default: <experiment_path>/report.html
      --publish               # publish as a public GitHub Gist via `gh` and print a gisthost.github.io URL

experiment list

Usage: etna experiment list

Workload Commands

etna workload --help

workload add

Usage: etna workload add [OPTIONS] <LANGUAGE> <WORKLOAD>

Options:
  -e, --experiment <EXPERIMENT>

If docs/workloads/<workload>.json exists, ETNA auto-generates a test file at tests/<workload>-<language>.json (lowercased).

workload remove

Usage: etna workload remove [OPTIONS] <LANGUAGE> <WORKLOAD>

Options:
  -e, --experiment <EXPERIMENT>

workload list

Usage: etna workload list [OPTIONS]

Options:
  -e, --experiment <EXPERIMENT>
  -l, --language <LANGUAGE>   # default: all
  -k, --kind <KIND>           # default: experiment | available | experiment

Mutation Commands

etna mutation --help

mutation list

Usage: etna mutation list [OPTIONS]
  -p, --path <PATH>   # default: .

mutation set

Usage: etna mutation set [OPTIONS] <VARIANT>
  -p, --path <PATH>   # default: .
  -g, --glob <GLOB>

mutation reset

Usage: etna mutation reset [OPTIONS]
  -p, --path <PATH>   # default: .

Other Commands

config show

Usage: etna config show

setup

Usage: etna setup [OPTIONS]
  -o, --overwrite

check

Usage: etna check [OPTIONS]
      --restore
      --remove

analyze bucket

Usage: etna analyze bucket [OPTIONS]
  -n, --name <NAME>

bash (Unix only)

Usage: etna bash [OPTIONS]
  -p, --path <PATH>

completions

Usage: etna completions <SHELL>

<SHELL> is one of the shells supported by clap_complete (bash, zsh, fish, powershell, elvish). Write the output to the shell's completions directory, e.g.:

etna completions zsh > ~/.zfunc/_etna

Always Check Live Help

CLI flags evolve. For exact current behavior, prefer:

etna <command> --help

Environment Variables

  • ETNA_HOME — override the default ~/.etna directory for config, the .etna_cache checkout, and the experiments index. Used by the integration test harness to isolate runs; handy for experimenting without disturbing your real setup.
  • ETNA_OFFLINE=1 — skip git pull inside .etna_cache during workload add and bash. Lets you run the CLI against a manually-primed cache without network access.
  • ETNA_REMOTE — override the repo URL cloned during etna setup (defaults to the upstream etna-cli repo).
  • ETNA_LOG — standard tracing filter. When set, stdout logs are colored with the module/level/file/line decorations; when unset, the CLI emits plain INFO messages to stdout and only decorates WARN/ERROR. Logs are always mirrored to etna.log in the current directory.