Track CLI invocation source and full command by ItIsUday · Pull Request #1293 · cloudbees-oss/smart-tests-cli

ItIsUday · 2026-04-28T09:55:32Z

Summary

Sends a COMMAND_INVOCATION tracking event on every CLI invocation with the full command string (rawCommand), exit code, caller, and CI provider
Auto-detects the CLI command from argv (maps to Command enum, falls back to UNKNOWN for typos)
SMART_TESTS_CALLER env var lets wrappers (e.g. GitHub Action) identify themselves
CI provider auto-detected from standard env vars (GITHUB_ACTIONS, JENKINS_URL, CIRCLECI, CODEBUILD_BUILD_ID)
caller, ciProvider, exitCode sent inside metadata JSONB — no DB migration needed
Existing error/performance tracking unchanged

Test plan

Unit tests for _detect_command() across all commands + typos + global options
Unit tests for send_command_tracking() payload structure
Unit tests for caller/ciProvider in metadata for all event types
Full test suite passes (376 tests, 0 new failures)

ItIsUday · 2026-05-05T07:56:47Z

There is two pending items though, would like to have an opinion on how to proceed

There's no input sanitization yet
The full raw command is captured, so if a command goes something like this, smart-tests record tests file --session 10 ./test-results/*.xml, then the full list of /test-results/ would be expanded in the raw string, which could be huge. Maybe we can just truncate the raw command?

ItIsUday · 2026-05-06T06:10:53Z

Ok I think I'm gonna truncate the raw command to 2000 characters

psakthivel04 · 2026-05-06T14:37:21Z

+        exit_code = e.code if isinstance(e.code, int) else 1
+    finally:
+        try:
+            send_command_tracking(argv=argv, exit_code=exit_code)


Can we invoke the CLI tracking asynchronously instead of blocking the execution?

It is invoked when all the command is processed already, and the CLI is about to exit. Asynchronous execution here might not be needed.

While it's true that tracking happens in the finally block after the command completes,
the HTTP request still blocks the CLI exit. This means every user waits an extra
50-200ms (or more with slow networks) for the tracking call to complete

But since the CLI is about to exit, we have these options

Forking: But forking will add extra overhead and might not be worth

Daemon threads or asyncio process get killed the moment sys.exit() is called

But these are not helpful unless we change the shutdown process some how.

I will add a short timeout, because if the server is unreachable, the cli will hang, this is more of an issue.

Addresses LCHUX-315. The backend can now identify whether CLI invocations come from direct usage, our GitHub Action wrapper, or another integration by reading the User-Agent header on every request. - SMART_TESTS_CALLER env var (defaults to "cli") lets wrappers identify themselves (e.g. github-action, jenkins-plugin) - detect_ci_provider() auto-detects the CI environment from standard env vars (GITHUB_ACTIONS, JENKINS_URL, CIRCLECI, CODEBUILD_BUILD_ID) - Both values are appended to User-Agent as Caller/{value} CI/{value} - Both values are also included in the cli_tracking payload Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Add test_tracking.py covering detect_ci_provider() and caller/ ciProvider fields in tracking payloads - Add User-Agent tests for caller and CI provider segments - Replace clear=True with surgical env cleanup in test_http_client.py to avoid wiping SMART_TESTS_BASE_URL (same issue as PR #1279) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Add send_command_tracking() to send a COMMAND_INVOCATION event with the full CLI argv, exit code, caller, and CI provider in metadata - Add _detect_command() to map argv to Command enum values; falls back to UNKNOWN for typos or unrecognized commands - Refactor TrackingClient: extract construct_payload() from _post_payload() so both send_command_tracking and existing methods share payload construction - Move caller/ciProvider into metadata JSONB (not top-level fields) - Add UNKNOWN and COMMAND_INVOCATION to respective enums Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Capture sys.argv and exit code, send COMMAND_INVOCATION tracking event after every CLI run (including typos and failures). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Caller and CI provider are now sent via cli_tracking metadata only. Remove Caller/ and CI/ segments from User-Agent header. Update test_tracking with tests for _detect_command, send_command_tracking, and caller/ciProvider in metadata. Revert test_http_client to original User-Agent assertions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add record attachment, inspect model, inspect subset, stats test_sessions, compare subsets, and get docs to both the Command enum and _COMMAND_MAP. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…lues Also adds individual detection tests for the 7 newly mapped commands. If a new Command is added to the enum without updating _COMMAND_MAP, test_command_map_covers_all_enum_values will fail. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…considering option values as commands

…command identification

psakthivel04

LGTM

ono-max · 2026-05-13T03:12:42Z

        metadata["stackTrace"] = stack_trace
-        metadata["organization"] = org or ""
-        metadata["workspace"] = workspace or ""
        metadata["api"] = api


Why did you leave stackTrace and api in this method instead of putting them in construct_payload method?

That was intentional because the stackTrace and api are visible to only send_error_event(). And that method can create a partial metadata dict with those values already. It is similar to other method like send_event() and send_command_tracking() where a partial metadata is generated with values under that specific method's scope.

ItIsUday self-assigned this Apr 28, 2026

ItIsUday changed the title ~~Track CLI invocation source via User-Agent~~ Track CLI invocation source and full command May 4, 2026

ItIsUday force-pushed the feature/track-cli-invokation branch from 3d78b59 to 8d52688 Compare May 6, 2026 06:00

psakthivel04 reviewed May 6, 2026

View reviewed changes

Comment thread smart_tests/utils/tracking.py

psakthivel04 reviewed May 6, 2026

View reviewed changes

Comment thread smart_tests/utils/tracking.py

psakthivel04 reviewed May 6, 2026

View reviewed changes

Comment thread smart_tests/utils/tracking.py

ItIsUday and others added 11 commits May 7, 2026 12:40

feat: track every CLI invocation from __main__.py

30af022

Capture sys.argv and exit code, send COMMAND_INVOCATION tracking event after every CLI run (including typos and failures). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

refactor: rawCommand can be None instead of an empty string

f8a0937

fix: send exitCode as string in tracking metadata

a24628e

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Truncate the raw command length to 2000 characters

8930132

feat: add missing commands to tracking map and enum

8001fce

Add record attachment, inspect model, inspect subset, stats test_sessions, compare subsets, and get docs to both the Command enum and _COMMAND_MAP. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Pre-commit fix

228ca35

ItIsUday force-pushed the feature/track-cli-invokation branch from 1c9cb77 to 228ca35 Compare May 7, 2026 07:13

ItIsUday added 3 commits May 11, 2026 16:30

Fix wrongful identification of command, where the sliding window was …

3525a35

…considering option values as commands

Ignore commands used with dry-run flag since that adds complexity to …

34ebbf6

…command identification

Add timeout for cli tracking API call

b118b6f

psakthivel04 approved these changes May 12, 2026

View reviewed changes

ono-max reviewed May 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Track CLI invocation source and full command#1293

Track CLI invocation source and full command#1293
ItIsUday wants to merge 14 commits into
mainfrom
feature/track-cli-invokation

ItIsUday commented Apr 28, 2026 •

edited

Loading

Uh oh!

ItIsUday commented May 5, 2026 •

edited

Loading

Uh oh!

ItIsUday commented May 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

psakthivel04 May 6, 2026 •

edited

Loading

Uh oh!

ItIsUday May 7, 2026

Uh oh!

psakthivel04 May 11, 2026

Uh oh!

ItIsUday May 11, 2026

Uh oh!

Uh oh!

Uh oh!

psakthivel04 left a comment

Uh oh!

ono-max May 13, 2026

Uh oh!

ItIsUday May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ItIsUday commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

ItIsUday commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ItIsUday commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

psakthivel04 May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ItIsUday May 7, 2026

Choose a reason for hiding this comment

Uh oh!

psakthivel04 May 11, 2026

Choose a reason for hiding this comment

Uh oh!

ItIsUday May 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

psakthivel04 left a comment

Choose a reason for hiding this comment

Uh oh!

ono-max May 13, 2026

Choose a reason for hiding this comment

Uh oh!

ItIsUday May 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ItIsUday commented Apr 28, 2026 •

edited

Loading

ItIsUday commented May 5, 2026 •

edited

Loading

ItIsUday commented May 6, 2026 •

edited

Loading

psakthivel04 May 6, 2026 •

edited

Loading