Move path-listing policy to the server and list git worktrees via ls-files

[SawyerHood]

What was wrong

listPathsRecursively in apps/host-daemon/src/command-handlers/file-list.ts, the single walker behind host.list_files and host.list_paths, unconditionally dropped every dot-leading entry, node_modules and symlinks before looking at the entry kind, and the command schemas carried no knob to relax it. Quick-open, the panel file search and composer @-mentions therefore could not find .github/workflows/ci.yml, .env or anything under .claude/, .bb/, .vscode/, while host.read_file and bb thread open serve the very same file (listing and reading disagreed about whether the file exists). The dot rule was never a product decision; it was a crude stand-in for "skip the stuff nobody wants" and it was also the only thing keeping the uncapped walk cheap. Report: https://get-bb.github.io/reports/issues/2093.html (#2093).

Related PRs: #2103 narrows the skip list to .git/node_modules in the daemon. That fixes the named symptom but keeps policy hardcoded in the daemon and, because the walker has no budget and runs on every keystroke, descends .venv/.turbo/.next/.cache: the report measured host.list_paths on a Python repo going from 6-8 ms to 629-1234 ms per keystroke with .venv/lib/python3.12/site-packages/... pushing the project's own config.py out of the top results, and the skill-file browser starts listing dotfiles its read path denies. #2127 (Monaco builtin) does not fix this issue; it documents it as a known gap. This PR credits both; what differs is below.

What changed

Policy is separated from the primitive: the server owns what is excluded, the daemon owns how it walks cheaply.

• Contract (packages/host-daemon-contract/src/commands.ts): host.list_files and host.list_paths gain required includeHidden: boolean, excludeNames: string[] and respectGitignore: boolean (shared pathListPolicySchema, exported PathListPolicy). HOST_DAEMON_PROTOCOL_VERSION 150 -> 151 with a note in protocol.ts; an older daemon rejects the new fields, a newer daemon rejects an older server's commands rather than defaulting.
• Daemon (file-list.ts, host-files.ts): listRootPaths applies the policy. With respectGitignore and a root inside a git worktree, candidates come from git ls-files -z --cached --others --exclude-standard (tracked + untracked-not-ignored, honours .gitignore, .git/info/exclude and global excludes) through the record-bounded runGitWithNullRecordLimit (newly exported from @bb/host-workspace); directory entries are synthesised from the file paths and the output is sorted once so parents precede children. Roots outside git, roots that are themselves ignored, hosts without git and respectGitignore: false take the readdir walk, which now takes includeHidden/excludeNames as arguments, skips symlinks, and still has exactly one literal: it never lists or descends .git. Both sources stop at a hard 50,000-entry cap (PATH_LIST_ENTRY_LIMIT) and report truncated: true.
• Server: new apps/server/src/routes/path-list-policy.ts fills the product default once at the boundary for workspace search (includeHidden: true, excludeNames: ["node_modules"], respectGitignore: true) for environments.paths, projects.paths, projects.files, files.list and files.listPaths; thread storage (threads/data.ts) uses includeHidden: true with the disk walk since a bb-owned data directory has no gitignore semantics. The two skill consumers (workspace-skills.ts, skill-listing.ts) pass SKILL_DIRECTORY_LIST_POLICY with includeHidden: false because readProjectSkill reads with dotfiles: "deny".
• Routes / SDK / CLI / docs: optional includeHidden on GET /environments/:id/paths, GET /projects/:id/paths and POST /files/paths (omitted = shown); PathListArgs.includeHidden and EnvironmentPathsArgs.includeHidden in @bb/sdk; --no-hidden on bb environment paths, bb project paths and bb file paths; guide templates (bb-guide-environments.md, bb-guide-projects.md, bb-guide-customization.md) and the bb-cli SKILL.md updated per docs/cli-guide-and-skill.md.
• Docs plugin (plugins/docs/server.ts): its four files.listPaths calls pass includeHidden: false because requireVaultPath rejects dot segments and the plugin's own .bb-docs-state.json lives at the vault root; without this the new default would throw during sync.

Deviation from the issue's proposed fix: the issue suggested skipping only .git (that is #2103) or caller-configurable fields. This implements the fields and additionally makes gitignore the exclusion source plus a walk cap, per the report's section 6, so the wider include set does not trade "the file is not there" for "the search is slow and full of junk". Known trade-offs (documented in code): empty directories do not appear in git mode, a submodule shows as one file-kind entry, and untracked files inside an ignored directory are hidden by default (respectGitignore: false is on the wire for callers that need them).

How you verified

Tests that fail before and pass after:

• apps/host-daemon/src/command-handlers/host-files.test.ts: the report's repro — listHostPaths with query: "ci.yml" returns .github/workflows/ci.yml and readHostFile serves the same file; a full listing lists .github/... while a gitignored .venv/lib/ci.yml is absent; host.list_files with includeHidden: false hides .DS_Store.
• apps/host-daemon/src/command-handlers/file-list.test.ts: walker lists dot paths but never .git; includeHidden/excludeNames prune at the entry; the cap truncates; listRootPaths in a git repo lists tracked + untracked-not-ignored with synthesised directories, keeps .venv/.env/node_modules/.git out, hides dot paths when told to, walks the disk when respectGitignore is false, falls back outside git and when the root itself is ignored.
• packages/host-daemon-contract/test/contract.test.ts: protocol 151; a listing command without the policy fields is rejected.
• apps/server/test/public/public-environments.test.ts, test/files/host-file-routes.test.ts: routes fill the default policy and pass an explicit includeHidden=false through. test/public/public-project-skills.test.ts, test/threads/thread-runtime-config.test.ts: skill listings send includeHidden: false so they never offer a file the dotfiles: "deny" read rejects.

Commands: pnpm exec turbo run typecheck --filter=@bb/host-daemon-contract --filter=@bb/host-daemon --filter=@bb/server --filter=@bb/server-contract --filter=@bb/sdk --filter=@bb/cli --filter=bb-plugin-simple-notes --filter=@bb/app (clean); pnpm exec turbo run test for @bb/host-daemon, @bb/host-daemon-contract, @bb/server (197 files; one unrelated timer flake in plugin-update.test.ts passes alone), @bb/server-contract, @bb/sdk, @bb/cli, bb-plugin-simple-notes, @bb/templates.

Live on an isolated dev instance (scripts/bb-dev-app current) against a project and environment on this bb worktree (a git worktree, so .git is a file): the issue's exact endpoint GET /api/v1/environments/<id>/paths?query=ci.yml&limit=5&includeFiles=true&includeDirectories=false now returns .github/workflows/ci.yml first (it returned {"paths":[]} on base); &includeHidden=false hides it again; the full listing has 6,139 entries with zero .turbo, node_modules or .git paths. bb environment paths --no-hidden, bb project paths --no-hidden and bb file paths --no-hidden behave the same. Timing with the report's handler benchmark on ~/browser-use (Python repo with a 16k-file .venv), query: "config", limit: 8: base 6-8 ms, #2103 629-1234 ms, this branch 14-17 ms with browser_use/config.py back in the top three; the same repo with respectGitignore: false (plain capped walk) takes 138-148 ms, which is what the gitignore source avoids.

Fixes #2093

AGENT GENERATED: by Claude Opus 5