AIR CLI Integration: Port code_source Snapshotting#5897
Open
riddhibhagwat-db wants to merge 4 commits into
Open
AIR CLI Integration: Port code_source Snapshotting#5897riddhibhagwat-db wants to merge 4 commits into
riddhibhagwat-db wants to merge 4 commits into
Conversation
First slice of the code_source snapshot port (Go equivalent of the Python CLI's cli/utils/snapshot.py). Adds the snapshot subpackage with the pure cache-key function and a golden-fixture harness: - cachekey.go: ComputeCacheKey + PackagingVersion="v1", ported verbatim from compute_snapshot_cache_key so both CLIs agree on the key algorithm. - testdata/cache_keys.json: golden keys captured from the exact Python algorithm over the local-only matrix (commit + include_paths permutations). - cachekey_test.go: byte-for-byte golden table test plus a properties test pinning the normalization (trim + sort, no dedup, nil == empty). Co-authored-by: Isaac
…only) Continues the code_source snapshot port. Flattens the snapshot files into the aircmd package (matching the rest of experimental/air/cmd; an organizational layer can land later) and adds Phase 2/3: - snapshot_git.go: local, no-network git introspection ported from cli/utils/git_state.py — is-repo, HEAD/branch SHA, dirty check (whole-tree and path-scoped), commit-exists, include-path validation. Concrete gitRepo shelling out with arg slices (no shell string); tested against real temp repos, matching the Python tests. - snapshot_resolve.go: the mode/ref decision tree (cli_entrypoint ~1541-1722), local-only. git.commit (must exist locally) / git.branch (local HEAD) -> git_archive; no ref or non-git dir -> plain_tar. Dirty check runs once and is threaded into the plan; dirty + git.branch is a hard error. - snapshot_package.go: git archive / plain tar builders + .gitignore->exclude parsing ported from cli/utils/snapshot.py. Shells out for parity; preserves the top-level dir name (--prefix / -C parent) the remote extraction depends on, and the .git / macOS AppleDouble exclusions. - runconfig.go: reject a truthy git.remote (remote-fetch path deprecated) with an actionable error pointing to git.commit; remote: false stays valid. The remote-fetch flavor (fetch_branch_sha, partial clone, remote HEAD) is intentionally not ported: the snapshot archives the local copy only. Co-authored-by: Isaac
Completes the code_source snapshot port (Phases 4-5): git provenance sidecars, the top-level orchestrator, upload via libs/filer, and the submit wiring. - snapshot_gitstate.go: git_state.json (schema_version 1, Python field names, null-for-absent encoding) + git_diff.patch capture (dirty-only, 1MB cap, 5s timeout). Best-effort — a failure warns and continues, never fails submit. Merge-base/repo-url read local refs only (no fetch), matching the local-only design. - snapshot.go: runSnapshot orchestrator (resolve -> package+upload -> sidecars). git_archive is cache-keyed and a workspace/Volume existence check skips packaging+upload on a hit; plain_tar is timestamp-named. Uploads through libs/filer (WorkspaceFilesClient, or FilesClient for a remote_volume). Also ports root_path resolution (project_root/, ~, env vars). - snapshot_git.go: add currentBranch / remoteURL / mergeBaseWithUpstream and a runBytes variant for the raw diff bytes. - runsubmit.go: drop the code_source rejection; package the snapshot and attach code_source_path / git_state_path / git_diff_path to the ai_runtime_task. Tested end to end against the in-process fake workspace, including the cache-hit skip and dirty-diff sidecar paths. Co-authored-by: Isaac
Phase 6 of the snapshot port — acceptance tests over the user-visible surface. - experimental/air/run: add dry-run cases for the new code_source config surface — a valid snapshot config passes validation, and a git.remote config is rejected with the actionable error. - experimental/air/run-submit: a full (non-dry-run) submit that packages a git code_source, uploads the tarball + git_state sidecar, and asserts code_source_path / git_state_path land on the ai_runtime_task payload. The commit is date-pinned so the cache-key-derived tarball name is stable, and RecordRequests captures the submitted payload. Co-authored-by: Isaac
Contributor
Approval status: pending
|
riddhibhagwat-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
riddhibhagwat-db
changed the title
riddhibhagwat-db
requested review from
ben-hansen-db,
maggiewang-db and
vinchenzo-db
Collaborator
Integration test reportCommit: f0d74ce
10 interesting tests: 4 RECOVERED, 4 SKIP, 2 flaky
Top 15 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Ports code_source snapshotting from the Python air CLI to Go and wires it into air run. When a config sets code_source.type: snapshot, the CLI now packages the local repo into a tarball, uploads it to the workspace (or a Volume), records git provenance sidecars, and attaches the paths (code_source_path, git_state_path, git_diff_path) to the ai_runtime_task payload.
Git-archive tarballs are cached by (commit, include_paths), so re-submitting the same commit skips packaging and upload.
Why
air run needs to ship the user's code to the training workload. This is a local-only port: we archive the local checkout rather than cloning from a remote, since that remote-fetch path is being deprecated. Consequences of that decision, which aren't obvious from the code alone:
Tests