Skip to content

Add alerting investigation skill reference#1356

Merged
thebiglabasky merged 3 commits into
mainfrom
herve/investigate-alerting-skill
Jun 19, 2026
Merged

Add alerting investigation skill reference#1356
thebiglabasky merged 3 commits into
mainfrom
herve/investigate-alerting-skill

Conversation

@thebiglabasky

@thebiglabasky thebiglabasky commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a dedicated checkly skills investigate alerting reference for read-only alerting investigations.

The reference guides agents through identifying a concrete check, preferring JSON/API evidence, inspecting check-local fields, group behavior, alert channel subscriptions, retry/double-check settings, and reporting unavailable evidence without inventing account/global alert settings.

Validation

  • pnpm exec vitest --run -c ./vitest.config.mts src/ai-context/__tests__/references.spec.ts
  • CHECKLY_API_KEY=dummy CHECKLY_ACCOUNT_ID=dummy NODE_CONFIG_DIR=./e2e/config pnpm exec vitest --run -c ./vitest.config.e2e.mts e2e/__tests__/skills.spec.ts
  • pnpm exec eslint src/ai-context/context.ts src/ai-context/__tests__/references.spec.ts e2e/__tests__/skills.spec.ts
  • ./bin/run skills
  • ./bin/run skills investigate alerting

CLI Eval Evidence

Focused scenario: cli-skill:alerting-settings-overrides, baseline checkly@latest, candidate checkly/checkly-cli#1356, profile skills.

  • Before tightening: run 27802013041, candidate @ 641ecb62cc18, failed on opus-4.6 because token budgets were exceeded: 248,317 total tokens, 244,388/240,000 input tokens, 16 steps. The LLM judge was skipped because hard token budgets failed.
  • After tightening: run 27802787199, candidate @ d37f8e1a44, passed on opus-4.6: 71,957 total tokens, 8 steps, judge 0.92.
  • Delta: -176,360 candidate tokens and -8 steps for the same focused alerting scenario.
  • Interpretation: the follow-up guidance reduced expensive speculative discovery, especially guessed account/global alert endpoints and broad alert-channel fetching, while preserving answer quality.

@thebiglabasky thebiglabasky marked this pull request as ready for review June 19, 2026 03:41
- If neither field is available, say retry or double-check behavior was not
available in the inspected output.

## Response Template

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's an interesting pattern. Do we have that somewhere already that we instruct the output?


Apply this tree from confirmed evidence only:

1. Check run and notification gates:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quite something. 😅 No surprise that we need to document that. :D

@stefanjudis stefanjudis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@thebiglabasky thebiglabasky enabled auto-merge (squash) June 19, 2026 08:45
@thebiglabasky thebiglabasky merged commit 4b338b4 into main Jun 19, 2026
7 of 8 checks passed
@thebiglabasky thebiglabasky deleted the herve/investigate-alerting-skill branch June 19, 2026 08:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants