Use native git CLI for worktree status on hook hot paths (45.9s -> 193ms on large worktrees)#1643
Open
sam-fakhreddine wants to merge 4 commits into
Open
Conversation
go-git's worktree.Status() stats and hashes every file on disk, including gitignored directories. On repos with large ignored trees (node_modules/, SwiftPM .build/, ...) this costs tens of seconds per hook invocation: user-prompt-submit measured at 45954ms on a 61GB/686k-file worktree (capture_pre_prompt_state 24397ms + init_session 21540ms), and every turn-end paid the same walk again in DetectFileChanges before the shadow branch commit. Add strategy.GitCLIStatus(), which runs 'git status --porcelain=v1 -z -uall --no-renames' and parses the output into a go-git-compatible git.Status map (porcelain v1 status letters are byte-identical to go-git StatusCode values), and use it in the three remaining hot call sites: - getUntrackedFilesForState (capture_pre_prompt_state) - calculatePromptAttributionAtStart (init_session) - DetectFileChanges (turn-end checkpoint) Same repo, same trace tooling after the change: user-prompt-submit 193ms (capture 102ms, init_session 79ms) — ~238x faster. The hand-rolled test repos in state_test.go gain a .git/refs/ directory, required by the native CLI to recognize a repository. Fixes entireio#1642
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses severe performance regressions on large worktrees by replacing remaining hot-path uses of go-git’s worktree.Status() with a native git status --porcelain implementation, avoiding expensive scans/hashing across large gitignored directories during hooks and checkpointing.
Changes:
- Adds
strategy.GitCLIStatus(ctx)to obtain a go-git-compatiblegit.Statusmap via nativegit status --porcelain=v1 -z -uall --no-renames. - Switches
DetectFileChanges,getUntrackedFilesForState, and prompt attribution startup logic to useGitCLIStatus. - Updates unit test scaffolding for minimal repos to include
.git/refs/, which native git requires for repo recognition.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| cmd/entire/cli/strategy/common.go | Introduces GitCLIStatus helper to replace go-git status calls with native git CLI parsing. |
| cmd/entire/cli/state.go | Uses strategy.GitCLIStatus in state capture and change detection hot paths. |
| cmd/entire/cli/strategy/manual_commit_hooks.go | Uses GitCLIStatus in calculatePromptAttributionAtStart to avoid expensive status walks. |
| cmd/entire/cli/state_test.go | Adjusts hand-rolled repo fixtures to be recognized by native git (adds .git/refs). |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Match the pattern in checkpoint/git_common_dir.go: an ExitError with empty stderr previously produced the awkward message "git status failed: : exit status 128".
Author
|
Addressed the error-formatting finding in ae2b4c4: stderr detail is now only appended when non-empty, matching the existing pattern in |
Comment on lines
68
to
78
| // Initialize git repo. refs/ is required alongside objects/ and HEAD for | ||
| // the native git CLI (used by GitCLIStatus) to recognize the directory. | ||
| if err := os.MkdirAll(".git/objects", 0o755); err != nil { | ||
| t.Fatalf("Failed to create .git: %v", err) | ||
| } | ||
| if err := os.MkdirAll(".git/refs", 0o755); err != nil { | ||
| t.Fatalf("Failed to create .git/refs: %v", err) | ||
| } | ||
| if err := os.WriteFile(".git/HEAD", []byte("ref: refs/heads/main\n"), 0o644); err != nil { | ||
| t.Fatalf("Failed to create HEAD: %v", err) | ||
| } |
Author
There was a problem hiding this comment.
Fixed in 6b86259 — both hand-rolled setups now use testutil.InitRepo, matching the convention used elsewhere in this file.
Comment on lines
354
to
364
| // Initialize git repo. refs/ is required alongside objects/ and HEAD for | ||
| // the native git CLI (used by GitCLIStatus) to recognize the directory. | ||
| if err := os.MkdirAll(".git/objects", 0o755); err != nil { | ||
| t.Fatalf("Failed to create .git: %v", err) | ||
| } | ||
| if err := os.MkdirAll(".git/refs", 0o755); err != nil { | ||
| t.Fatalf("Failed to create .git/refs: %v", err) | ||
| } | ||
| if err := os.WriteFile(".git/HEAD", []byte("ref: refs/heads/main\n"), 0o644); err != nil { | ||
| t.Fatalf("Failed to create HEAD: %v", err) | ||
| } |
Author
There was a problem hiding this comment.
Fixed in 6b86259 — setupTestRepoWithTranscript now uses testutil.InitRepo.
…e tests The manually constructed .git layout (objects/, refs/, HEAD) is brittle across git versions now that production code shells out to the git CLI for rev-parse and status. testutil.InitRepo creates a real repository layout and matches the convention used elsewhere in this file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1642 — on worktrees with large gitignored trees, the Claude Code
user-prompt-submithook takes 40–55s per prompt and every turn-end checkpoint pays the same cost again. Root cause: three hot paths still used go-git'sworktree.Status(), which stats and hashes every file on disk including gitignored directories. Several sibling call sites had already been converted to the native git CLI for exactly this reason (see the existing comments incheckpoint/ephemeral.goandgetStagedFilesinstrategy/manual_commit_hooks.go); this PR finishes the job.Changes
strategy/common.go: new exportedGitCLIStatus(ctx)helper — runsgit status --porcelain=v1 -z -uall --no-renamesat the worktree root and parses the output into a go-git-compatiblegit.Statusmap. Porcelain v1 status letters are byte-identical to go-git'sStatusCodeconstants, so this is a drop-in replacement forworktree.Status().state.go:getUntrackedFilesForState(→capture_pre_prompt_state) andDetectFileChanges(→ turn-end checkpointing) now useGitCLIStatus.strategy/manual_commit_hooks.go:calculatePromptAttributionAtStart(→init_session) now usesGitCLIStatus.state_test.go: the hand-rolled test repos gain a.git/refs/directory — the native git CLI requires it to recognize a repository (go-git did not).Measurements
61GB / ~686k-file worktree (~34GB in gitignored SwiftPM
.build/dirs), measured withentire doctor trace+ENTIRE_LOG_LEVEL=DEBUG:user-prompt-submittotalcapture_pre_prompt_stateinit_sessionTurn-end shadow-branch commits went from 20–50s to sub-second on the same repo.
Testing
mise run lint— cleanmise run test— 989 passed; the single failure (TestRunUninstall_Force_NothingInstalled) is pre-existing and environment-dependent — it fails identically on pristinemainon any machine where the Entire CLI is installed, because the uninstall path finds a real installation.entirebinary on the affected repo since building it.Notes
Entire-Checkpointtrailer on the commit: the investigation and patch were developed from an agent session anchored in the affected repo (where the bug lives), not this clone, so there was no checkpoint in this worktree to attribute. Happy to adjust if you'd like it structured differently.