Surface unsupported files in index diagnostics

AGENT_GUIDE.md

@@ -143,7 +143,7 @@ CI watching must be bounded. Do not loop indefinitely.
 
 ## Status Contract
 
-- `status --json` and related JSON/MCP payloads currently expose the trust fields documented in `README.md` and `DEVELOPER_GUIDE.md`, including `fold_ready`, `fold_ready_reason`, `graph_table_available`, `graph_data_current`, `index_complete`, `index_incomplete_reasons`, `issues_table_available`, `file_issues_data_current`, `migration_in_progress`, `sql_graph_contract_ready`, `sql_graph_contract_degraded_reason`, `hotspot_family_ready`, `hotspot_family_degraded_reason`, `language_readiness`, `csharp_symbol_name_ready`, `csharp_metadata_target_ready`, `csharp_metadata_target_degraded_reason`, `indexed_head_commit`, `workspace_verified_head_sha`, `worktree_head_changed`, `indexed_head_sha`, `indexed_head_branch`, `indexed_head_timestamp`, `commits_ahead_of_indexed_head`, `head_freshness`, `index_writer_version`, `index_newer_than_reader`, `index_newer_than_reader_reason`, `unknown_extension_file_count`, `unknown_extension_files`, `unknown_extension_files_truncated`, `unknown_extension_file_path_limit`, `unknown_extension_extension_counts`, `unknown_extension_category_counts`, `unknown_extension_groups`, `extractors`, `git_executable`, `path_case_sensitive`, `data_dir`, `data_dir_source`, `data_dir_mode`, `db_file_mode`, `database_permission_policy`, `database_permission_diagnostics`, `mac_profile`, `mac_profile_diagnostics`, `db_size_bytes`, `wal_size_bytes`, `db_pragma_settings` (`journal_mode`, `synchronous`, `wal_autocheckpoint`, `busy_timeout_ms`, `page_count`, `freelist_count`, `page_size`, `auto_vacuum`), `prepared_command_cache` (`count`, `capacity`, `hit_count`, `miss_count`, `eviction_count`), `maintenance_guidance`, WAL checkpoint diagnostics (`read_only_fallback`, `wal_checkpoint_attempted`, `wal_checkpoint_succeeded`, `wal_checkpoint_skipped_reason`, `wal_checkpoint_failure_reason`, `wal_checkpoint_busy`, `wal_checkpoint_log_page_count`, `wal_checkpoint_checkpointed_page_count`, `wal_checkpoint_remaining_page_count`, `read_only_immutable_fallback`, `wal_stale_snapshot_risk`, `wal_stale_snapshot_reason`), `symbol_kinds`, `symbols_by_language`, status kind cap metadata (`symbol_kind_limit`, `symbol_kind_name_limit`, `symbol_kind_total_count`, `symbol_kind_omitted_count`, `symbol_kind_names_truncated`, `symbols_by_language_kind_total_counts`, `symbols_by_language_kind_omitted_counts`, `symbols_by_language_kind_names_truncated`), `process`, `last_index_run`, `last_failed_or_partial_index_run`, `last_failed_or_partial_index_run.progress_persisted`, `last_failed_or_partial_index_run.recovery_hint`, `last_failed_or_partial_index_run.file_errors`, `last_workspace_freshened_at`, `hooks`, `hook_diagnostics`, `trust_overrides`, MCP-only `mcp_session`, `mcp.rate_limit.bucket_limit`, `mcp.rate_limit.bucket_limit_rejection_count`, and the `status --check`-only `stale_after_seconds` / `index_age_seconds` threshold audit fields and `repair_commands`.
+- `status --json` and related JSON/MCP payloads currently expose the trust fields documented in `README.md` and `DEVELOPER_GUIDE.md`, including `fold_ready`, `fold_ready_reason`, `graph_table_available`, `graph_data_current`, `index_complete`, `index_incomplete_reasons`, `issues_table_available`, `file_issues_data_current`, `migration_in_progress`, `sql_graph_contract_ready`, `sql_graph_contract_degraded_reason`, `hotspot_family_ready`, `hotspot_family_degraded_reason`, `language_readiness`, `csharp_symbol_name_ready`, `csharp_metadata_target_ready`, `csharp_metadata_target_degraded_reason`, `indexed_head_commit`, `workspace_verified_head_sha`, `worktree_head_changed`, `indexed_head_sha`, `indexed_head_branch`, `indexed_head_timestamp`, `commits_ahead_of_indexed_head`, `head_freshness`, `index_writer_version`, `index_newer_than_reader`, `index_newer_than_reader_reason`, `unknown_extension_file_count`, `unknown_extension_files`, `unknown_extension_files_truncated`, `unknown_extension_file_path_limit`, `unknown_extension_extension_counts`, `unknown_extension_category_counts`, `unknown_extension_groups`, `unknown_extension_group_count`, `unknown_extension_groups_truncated`, `unknown_extension_group_limit`, `unknown_extension_group_omitted_count`, `unknown_extension_guidance`, `extractors`, `git_executable`, `path_case_sensitive`, `data_dir`, `data_dir_source`, `data_dir_mode`, `db_file_mode`, `database_permission_policy`, `database_permission_diagnostics`, `mac_profile`, `mac_profile_diagnostics`, `db_size_bytes`, `wal_size_bytes`, `db_pragma_settings` (`journal_mode`, `synchronous`, `wal_autocheckpoint`, `busy_timeout_ms`, `page_count`, `freelist_count`, `page_size`, `auto_vacuum`), `prepared_command_cache` (`count`, `capacity`, `hit_count`, `miss_count`, `eviction_count`), `maintenance_guidance`, WAL checkpoint diagnostics (`read_only_fallback`, `wal_checkpoint_attempted`, `wal_checkpoint_succeeded`, `wal_checkpoint_skipped_reason`, `wal_checkpoint_failure_reason`, `wal_checkpoint_busy`, `wal_checkpoint_log_page_count`, `wal_checkpoint_checkpointed_page_count`, `wal_checkpoint_remaining_page_count`, `read_only_immutable_fallback`, `wal_stale_snapshot_risk`, `wal_stale_snapshot_reason`), `symbol_kinds`, `symbols_by_language`, status kind cap metadata (`symbol_kind_limit`, `symbol_kind_name_limit`, `symbol_kind_total_count`, `symbol_kind_omitted_count`, `symbol_kind_names_truncated`, `symbols_by_language_kind_total_counts`, `symbols_by_language_kind_omitted_counts`, `symbols_by_language_kind_names_truncated`), `process`, `last_index_run`, `last_failed_or_partial_index_run`, `last_failed_or_partial_index_run.progress_persisted`, `last_failed_or_partial_index_run.recovery_hint`, `last_failed_or_partial_index_run.file_errors`, `last_workspace_freshened_at`, `hooks`, `hook_diagnostics`, `trust_overrides`, MCP-only `mcp_session`, `mcp.rate_limit.bucket_limit`, `mcp.rate_limit.bucket_limit_rejection_count`, and the `status --check`-only `stale_after_seconds` / `index_age_seconds` threshold audit fields and `repair_commands`.
 - `status --check` repair actions are structured by `name`, `action`, `args`, `mutation_class`, `safety_class`, and `safety_notes`. Preserve compatibility `reason` as the first trigger and ordered `reasons` as the complete trigger set. Deduplicate only exact structured identities; different targets, options, actions, mutation classes, or safety semantics must remain distinct. JSON and human output must use the same deduplicated order. Human output must preserve platform-aware shell quoting, visibly escape control characters to keep each repair action on one diagnostic line, and leave structured JSON `args` unchanged.
 - `maintenance_guidance.fts_optimization` is the shared, read-only recommendation contract for status, explain, optimize preview, and optimize execution. Keep `recommended`, `action`, `reason`, `threshold_writes`, `observed_writes`, and `state` synchronized; stale or unavailable snapshots must not recommend mutation.
 - A successful CLI or MCP `index --rebuild` evaluates the shared freelist warning threshold after the index transaction commits. Incremental-auto-vacuum databases may run bounded `PRAGMA incremental_vacuum`; legacy databases must skip automatic reclaim instead of running a full `VACUUM`. Preserve the immediate index-result and persisted `last_index_run.rebuild_reclaim` telemetry, stable states/reasons, before/after ratios and byte/page counts, and the rule that reclaim failure never reclassifies an already committed index run as failed.
@@ -159,7 +159,7 @@ CI watching must be bounded. Do not loop indefinitely.
 - Successful CLI full/update indexing, immediate status/workspace status, and MCP indexing/status must derive `index_complete`, `index_incomplete_reasons`, `reference_graph_complete`, and `reference_graph_incomplete_reasons` from the same persisted-readiness snapshot. Symbols-only runs and persisted file-size, symbol-count, reference-count, extractor-failure, or reference-cap evidence make the generation incomplete. Legacy databases keep the complete compatibility default only when persisted rows do not prove an omission.
 - `index_writer_version` records the `cdidx` version that last wrote to the DB (stamped into `codeindex_meta` as `cdidx_writer_version` on every full scan, update, and MCP index). `index_newer_than_reader` flips to `true` whenever any persisted numeric contract stamp in `codeindex_meta` (or unknown `PRAGMA user_version` readiness bits) exceeds the current binary's compiled maximum, so an older CLI re-opening a DB written by a newer CLI degrades loudly with an audit trail instead of silently dropping back to text-search fallbacks. `index_newer_than_reader_reason` enumerates the specific newer-than-reader stamps.
 - `status` separates latest-write provenance (`indexed_head_sha`, `indexed_head_branch`, `indexed_head_timestamp`, `commits_ahead_of_indexed_head`) from whole-workspace verification (`workspace_verified_head_sha`) and the full-scan compatibility baseline (`indexed_head_commit`). Every successful full or partial run updates the latest-write triple; only a full scan or a Git-scoped refresh reconciled from the prior verified baseline advances whole-workspace verification. Scoped mutations persist their affected paths until that reconciled refresh revisits them, including net-zero Git ranges; incomplete pending-path coverage fails closed until a normal full scan succeeds. `worktree_head_changed`, status, map, and MCP use that verified value, falling back conservatively to `indexed_head_commit` for legacy DBs. `head_freshness.state=fresh` requires `status --check` to match the workspace, `fresh_but_incomplete` keeps matching-workspace freshness distinct from incomplete extraction coverage, and `state=head_current` only means the runtime HEAD matches the `indexed_head` selected by `indexed_head_source`.
-- `status` also surfaces unknown-extension scan coverage via `unknown_extension_file_count`, stamped by successful full-repository index runs (`cdidx index <projectPath>` and MCP `index_project`) as the number of non-indexed files with non-empty extensions that do not map to a known language. Current scans also stamp `unknown_extension_files` as a path sample bounded by `unknown_extension_file_path_limit` items and the string-list decoded-character budget, `unknown_extension_files_truncated` when more paths existed than were emitted for either bound, and `unknown_extension_file_path_limit` as the item cap rather than a guarantee that that many paths are returned. Newer scans also expose `unknown_extension_extension_counts`, `unknown_extension_category_counts`, and `unknown_extension_groups`; groups classify common non-code buckets such as repository metadata, licenses, binary assets, configuration, structural metadata, and language-support candidates, and include `recommended_action` values of `ignore_configuration`, `first_class_structural_extraction`, or `language_support`. These fields are omitted on legacy DBs or before a current full scan has stamped them.
+- `status` also surfaces unsupported-language scan coverage via `unknown_extension_file_count`, stamped by successful full-repository index runs (`cdidx index <projectPath>` and MCP `index_project`) as the number of readable, text-like, non-indexed files whose filename or extension does not map to a known language. Extensionless files without a recognized shebang are included as `<none>`; binary, oversized, unreadable, explicitly ignored, and internal index-artifact files are excluded. Current scans also stamp a diagnostics contract version and `unknown_extension_files` as a path sample bounded by `unknown_extension_file_path_limit` items and the string-list decoded-character budget, `unknown_extension_files_truncated` when more paths existed than were emitted for either bound, and `unknown_extension_file_path_limit` as the item cap rather than a guarantee that that many paths are returned. Newer scans also expose `unknown_extension_extension_counts`, `unknown_extension_category_counts`, and `unknown_extension_groups`; groups classify common non-code buckets such as repository metadata, licenses, binary assets, configuration, structural metadata, and language-support candidates, and include `recommended_action` values of `ignore_configuration`, `first_class_structural_extraction`, or `language_support`. These fields are omitted on legacy DBs, when the persisted diagnostics version is absent or unsupported, or before a current full scan has stamped them. Guidance for actionable extensionless files starts with adding a recognized shebang or renaming the file with a supported extension.
 - `status` also surfaces extractor plugin and pattern-config runtime diagnostics via `extractors`, including loaded counts, the zero parent `retained_load_context_count`, the isolated-worker `load_context_lifecycle`, skipped file counts, and a bounded diagnostics list for incompatible or malformed plugin/pattern files. `extractors.pattern_configs[]` reports accepted sidecars with sanitized `path`, `source` (`workspace` or `user`), normalized `language`, and `rule_count`; workspace discovery stops at its explicit root and path identity follows the live filesystem case policy. Pattern rules and their 128-rule budget live in immutable workspace snapshots; reindex replaces the owning snapshot, and timeout cooldowns/diagnostics remain workspace-scoped. `extractors.snapshot_scope` and highest-first `registration_precedence` expose snapshot selection and the fixed `built_in > user_plugin > user_pattern > workspace_plugin > workspace_pattern` resolution order; replacing one workspace snapshot never mutates another. Diagnostic paths, categories, and messages are sanitized before output; `diagnostics[].category` is the stable machine-readable failure code.
 - `status` surfaces worker-discovered post-extraction hook manifests and callback budgets through `hooks[]`, including stable assembly-qualified `hooks[].id`, `hooks[].callback_budget_ms`, and the worker-only `hooks[].load_context_lifecycle`. Assembly loading, module initialization, type inspection, and constructor validation occur only in deadline-, memory-, and output-bounded discovery workers, which are terminated after returning a manifest. `hook_diagnostics[]` reports sanitized discovery and callback diagnostics such as candidate-limit truncation, assembly load failure, constructor failure, callback failure, and timeout; `hook_diagnostics[].category` is the stable machine-readable failure code and `hook_diagnostics[].hook_id` links a concrete-hook failure to the corresponding manifest. Index runs enforce `CDIDX_HOOK_CALLBACK_BUDGET_MS` (default: 5000 ms) on scratch copies, discard timed-out mutations, and disable only the timed-out assembly-qualified hook ID for the remainder of the current run.
 - `status` also surfaces accepted trust-boundary environment overrides through `trust_overrides[]`. Entries include `kind`, `environment_variable`, sanitized `value`, optional sanitized `path`, and `message`; current entries cover `CDIDX_TRUST_WORKSPACE_PLUGINS` workspace plugin discovery, `CDIDX_HOOKS_DIR` hook directory overrides, and the absolute `CDIDX_GIT_EXECUTABLE` executable override. `git_executable` reports the selected source, acceptance, stable reason, sanitized path, owner-only-write result, Unix mode, owner category, owner/ancestor trust (including POSIX sticky-ancestor and Windows ACL policy), and bounded `git --version` execution-probe result even when an explicit Git override is rejected. Build `git_executable` and its matching `trust_overrides[]` entry from the same resolution snapshot. Keep this visible runtime field registered in `status --explain` and in both README status-field tables.