Add alerting investigation skill reference#1356
Merged
Merged
Conversation
stefanjudis
reviewed
Jun 19, 2026
| - If neither field is available, say retry or double-check behavior was not | ||
| available in the inspected output. | ||
|
|
||
| ## Response Template |
Collaborator
There was a problem hiding this comment.
That's an interesting pattern. Do we have that somewhere already that we instruct the output?
stefanjudis
reviewed
Jun 19, 2026
|
|
||
| Apply this tree from confirmed evidence only: | ||
|
|
||
| 1. Check run and notification gates: |
Collaborator
There was a problem hiding this comment.
This is quite something. 😅 No surprise that we need to document that. :D
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a dedicated
checkly skills investigate alertingreference for read-only alerting investigations.The reference guides agents through identifying a concrete check, preferring JSON/API evidence, inspecting check-local fields, group behavior, alert channel subscriptions, retry/double-check settings, and reporting unavailable evidence without inventing account/global alert settings.
Validation
pnpm exec vitest --run -c ./vitest.config.mts src/ai-context/__tests__/references.spec.tsCHECKLY_API_KEY=dummy CHECKLY_ACCOUNT_ID=dummy NODE_CONFIG_DIR=./e2e/config pnpm exec vitest --run -c ./vitest.config.e2e.mts e2e/__tests__/skills.spec.tspnpm exec eslint src/ai-context/context.ts src/ai-context/__tests__/references.spec.ts e2e/__tests__/skills.spec.ts./bin/run skills./bin/run skills investigate alertingCLI Eval Evidence
Focused scenario:
cli-skill:alerting-settings-overrides, baselinecheckly@latest, candidatecheckly/checkly-cli#1356, profileskills.641ecb62cc18, failed onopus-4.6because token budgets were exceeded:248,317total tokens,244,388/240,000input tokens,16steps. The LLM judge was skipped because hard token budgets failed.d37f8e1a44, passed onopus-4.6:71,957total tokens,8steps, judge0.92.-176,360candidate tokens and-8steps for the same focused alerting scenario.