fix: improve logging on nil-resource case

[gusfcarvalho]

Summary by CodeRabbit

• New Features

  • Consolidated end-of-run summary with per-check run rows and bounded diagnostic snapshots; zero-evidence outcome now returns an enriched error including those diagnostics
  • Per-check summaries show baseline, matched resources, evidence/error counts, and region info; baselines now emit a warning when zero resources are found

• Bug Fixes

  • Custodian run log tail is always captured (including on success) but not surfaced as a top-level error

• Tests

  • Added tests validating zero-evidence diagnostics, mixed execution reporting, and diagnostic-size bounding

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: f14810e2-5523-4459-a783-10de51ca436c

📥 Commits

Reviewing files that changed from the base of the PR and between c95fd23 and 1d37961.

📒 Files selected for processing (1)

• main.go

---

📝 Walkthrough

Walkthrough

This PR always captures custodian run log tails, records bounded per-execution diagnostics and per-check run summaries during Eval, and returns a bounded, detailed zero-evidence error (including per-execution snapshots) when no evidence is produced, while preserving original per-policy errors.

Changes

Zero-Evidence Error Reporting with Diagnostic Detail

Layer / File(s)	Summary
Data model and diagnostic structures main.go	Imports maps, adds LogTail to CustodianExecutionResult, and adds diagnostic/capacity constants plus executionDiagnostic and checkRunSummary types and helpers used to format bounded diagnostic detail.
Unconditional log artifact collection main.go	Command executor now always reads and stores custodian-run.log paths and tails for every execution and includes aws_regions in the execution log statement.
Per-execution diagnostic tracking during Eval main.go	Eval now builds executionDiagnostics and runSummaries, records baseline/parse/missing-baseline/execution-failure cases, tracks payload/evidence counts and had-error flags, and calls warnZeroResourceBaseline for empty-but-successful baselines.
Zero-evidence detection and bounded error composition main.go	Detects zero-evidence outcomes and returns composeZeroEvidenceError(accumulatedErrors, executionDiagnostics), which merges original errors with bounded diagnostic snapshots capped by custodianDiagnosticDetailCap; artifacts are preserved for zero-evidence outcomes.
Baseline zero-resource diagnostics main.go	Adds warnZeroResourceBaseline to emit warnings when an inventory baseline exits 0 but reports zero resources, including bounded stderr/log tails for diagnostics.
Test verification of log artifacts and zero-evidence errors main_test.go	Updates executor test to verify log tail capture on success without surfacing as an error; adds TestEvalZeroEvidenceErrorIncludesCustodianDiagnostics with subtests verifying inclusion of per-execution diagnostics, preserved per-policy errors, and enforced diagnostic size bounding.

sequenceDiagram
  participant Executor as CommandCustodianExecutor
  participant Eval as CloudCustodianPlugin.Eval
  participant Composer as composeZeroEvidenceError
  Executor->>Eval: return CustodianExecutionResult (LogPaths, LogTail, exitCode, stderr)
  Eval->>Eval: accumulate executionDiagnostics and runSummaries
  Eval->>Composer: on zero-evidence send accumulatedErrors + executionDiagnostics
  Composer->>Eval: composed, bounded diagnostic error string

Loading

🎯 4 (Complex) | ⏱️ ~45 minutes

🐰
I nibble lines of log and tail,
Collect the whiskered details,
When evidence is none or slim,
I stitch a tidy diagnostic hymn,
So humans find the missing clue.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Title check	⚠️ Warning	The title 'fix: improve logging on nil-resource case' is partially related to the changeset, but does not reflect the main scope of changes which include zero-evidence error handling, diagnostic detail bounding, and comprehensive log artifact capture.	Revise the title to better reflect the primary changes, such as 'fix: improve zero-evidence diagnostics and log capture' or similar, to more accurately describe the enriched error diagnostics and artifact preservation on zero-evidence outcomes.
Docstring Coverage	⚠️ Warning	Docstring coverage is 27.27% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

This PR improves troubleshooting for Cloud Custodian runs that exit successfully (exit code 0) but yield no resources/evidence by always capturing custodian-run.log tails and emitting consolidated, per-check diagnostics when the evaluation produces zero evidence.

Changes:

• Always collect custodian-run.log artifact paths and tail content into CustodianExecutionResult (new LogTail field).
• Add consolidated per-check run summaries and preserve execution artifacts when the run produces zero evidence.
• Enrich the “zero evidence” failure error with bounded (capped) per-execution diagnostics (stderr/log tails) and add targeted warnings for zero-resource inventory baselines.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
main.go	Captures log tails unconditionally, adds zero-evidence diagnostic composition, consolidated summary logging, and baseline zero-resource warnings.
main_test.go	Updates executor behavior tests for always-captured log tails and adds coverage for enriched zero-evidence diagnostics and size bounding.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.