From abd857be89f28a4c8c3bc7692711182c22191ee2 Mon Sep 17 00:00:00 2001 From: scottschreckengaust <345885+scottschreckengaust@users.noreply.github.com> Date: Wed, 29 Jul 2026 00:59:09 +0000 Subject: [PATCH 1/2] feat(infra): add Lambda-error alarm to GitHub webhook processor (#284) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #284 asked for an SQS DLQ + a CloudWatch alarm on the async screenshot processor. Most of it already landed on main (PR #241 follow-up): github-screenshot-integration.ts already defines WebhookProcessorDlq (14-day retention, enforceSSL), wires it as the Lambda async-invoke deadLetterQueue, carries the AwsSolutions-SQS3 suppression, and has a WebhookProcessorDlqDepthAlarm on the DLQ's ApproximateNumberOfMessages metric. So ACs 1 (DLQ onFailure), 3 (retention/SSE), and 4 (cdk-nag) were already satisfied. Two ACs were UNMET and are addressed here: - AC 2 — an alarm on the processor's Lambda **Errors** metric (>= 1 in 5min, 2 eval periods). This is DISTINCT from the existing DLQ-depth alarm: the depth alarm only fires once a failure has survived Lambda's async-retry ladder and landed on the queue, whereas the Errors alarm fires as soon as invocations start faulting — catching a systemic break (IAM regression, AgentCore quota exhaustion, OAuth-rotation issue, dependency outage) earlier. Mirrors the OrchestratorErrorAlarm idiom in task-orchestrator.ts (threshold, evaluationPeriods, treatMissingData=NOT_BREACHING). I did NOT duplicate the pre-existing DLQ-depth alarm. - AC 5 (docs) — DEPLOY_PREVIEW_SCREENSHOTS_GUIDE.md now has a Troubleshooting subsection explaining both alarms and how to inspect/drain the DLQ; Starlight mirror regenerated via docs:sync. SNS: AC #5 said "notify via existing alarm SNS topic if one exists, otherwise create one". There is NO SNS topic or addAlarmAction anywhere in cdk/src today — every existing alarm (FanOutConsumer.dlqDepthAlarm, OrchestratorErrorAlarm, the pre-existing WebhookProcessorDlqDepthAlarm) is a bare alarm with no notification action. To reuse-over-reinvent and stay consistent with the repo, this alarm is also bare; a stack-wide SNS notification plane is out of scope for a single construct. Closes #284 Co-authored-by: Claude Opus 4.8 --- .../github-screenshot-integration.ts | 44 +++++++++++++++++++ .../github-screenshot-integration.test.ts | 37 +++++++++++++++- .../DEPLOY_PREVIEW_SCREENSHOTS_GUIDE.md | 29 ++++++++++++ .../using/Deploy-preview-screenshots-guide.md | 29 ++++++++++++ 4 files changed, 138 insertions(+), 1 deletion(-) diff --git a/cdk/src/constructs/github-screenshot-integration.ts b/cdk/src/constructs/github-screenshot-integration.ts index b48c70864..3446a02c8 100644 --- a/cdk/src/constructs/github-screenshot-integration.ts +++ b/cdk/src/constructs/github-screenshot-integration.ts @@ -40,6 +40,12 @@ const PROCESSOR_DLQ_RETENTION_DAYS = 14; /** DLQ-depth alarm metric period (minutes). */ const DLQ_ALARM_PERIOD_MINUTES = 5; +/** Processor Lambda-Errors alarm metric period (minutes). */ +const ERROR_ALARM_PERIOD_MINUTES = 5; + +/** Processor Lambda-Errors alarm sustained evaluation periods. */ +const ERROR_ALARM_EVALUATION_PERIODS = 2; + /** Async screenshot-processor Lambda memory (MB). */ const PROCESSOR_MEMORY_MB = 512; @@ -128,6 +134,18 @@ export class GitHubScreenshotIntegration extends Construct { * "for operator inspection" that no operator is ever told to make. */ public readonly processorDlqDepthAlarm: cloudwatch.Alarm; + /** Fires when the processor Lambda's ``Errors`` metric breaches. The + * handler swallows its per-step operational failures (log + return), + * so this metric ticks only on faults that ESCAPE the handler — + * init-time crashes (missing env at cold start, bundling defect), + * unhandled throws in an unguarded path, or the 120s hard timeout. + * For that population it fires one evaluation window sooner than the + * DLQ-depth alarm (which waits for Lambda's async-retry ladder to land + * a message). Follows the ``metricErrors`` + 2-eval-period shape of + * ``task-orchestrator.ts`` OrchestratorErrorAlarm (with threshold 1, + * since any escaped error here is significant). */ + public readonly processorErrorAlarm: cloudwatch.Alarm; + constructor(scope: Construct, id: string, props: GitHubScreenshotIntegrationProps) { super(scope, id); @@ -219,6 +237,32 @@ export class GitHubScreenshotIntegration extends Construct { treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING, }); + // Alarm on the processor's Errors metric (#284 AC). Complementary to + // the DLQ-depth alarm above, not redundant. The handler is best-effort + // by design — it catches its per-step operational failures (bad token, + // AgentCore/S3/comment-post errors) and returns success, so those + // surface as tagged log events, NOT as Lambda Errors. This alarm + // therefore fires on the faults that ESCAPE the handler: an init-time + // crash (missing env at cold start, bundling defect), an unhandled + // throw in an unguarded path, or the 120s hard timeout. That is the + // same failure class the DLQ eventually catches, but the Errors metric + // trips one evaluation window sooner — before Lambda's async-retry + // ladder has landed a message on the DLQ. threshold 1 / 2 eval periods + // matches the AC; the metricErrors + 2-eval-period shape follows + // ``task-orchestrator.ts`` OrchestratorErrorAlarm (which uses + // threshold 3 — we use 1 because any escaped error here is + // significant). + this.processorErrorAlarm = new cloudwatch.Alarm(this, 'WebhookProcessorErrorAlarm', { + metric: this.webhookProcessorFn.metricErrors({ + period: Duration.minutes(ERROR_ALARM_PERIOD_MINUTES), + }), + threshold: 1, + evaluationPeriods: ERROR_ALARM_EVALUATION_PERIODS, + alarmDescription: + 'Screenshot webhook processor Lambda errors breached threshold — deploy-preview screenshots are failing (check IAM, AgentCore Browser quota, and dependency health)', + treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING, + }); + this.screenshotBucket.bucket.grantPut(this.webhookProcessorFn); props.githubTokenSecret.grantRead(this.webhookProcessorFn); diff --git a/cdk/test/constructs/github-screenshot-integration.test.ts b/cdk/test/constructs/github-screenshot-integration.test.ts index 3e415c87a..68c7b4383 100644 --- a/cdk/test/constructs/github-screenshot-integration.test.ts +++ b/cdk/test/constructs/github-screenshot-integration.test.ts @@ -70,7 +70,12 @@ describe('GitHubScreenshotIntegration construct', () => { // crashes reach the queue — each one means the screenshot pipeline is // silently down. Without the alarm the queue is "for operator // inspection" that no operator is ever told to make. - template.resourceCountIs('AWS::CloudWatch::Alarm', 1); + // + // Two alarms total: this DLQ-depth alarm (a record LANDED) and the + // Lambda-Errors alarm below (invocations are FAILING, before Lambda's + // async retries have exhausted onto the DLQ). They are complementary, + // not redundant — see the next test. + template.resourceCountIs('AWS::CloudWatch::Alarm', 2); template.hasResourceProperties('AWS::CloudWatch::Alarm', { MetricName: 'ApproximateNumberOfMessagesVisible', Namespace: 'AWS/SQS', @@ -90,6 +95,36 @@ describe('GitHubScreenshotIntegration construct', () => { }); }); + test('alarms on the processor Lambda Errors metric (>=1 in 5min, 2 eval periods)', () => { + // #284 AC: an alarm on the processor's Errors metric. This is DISTINCT + // from the DLQ-depth alarm above — the DLQ-depth alarm only fires once + // a failure has survived Lambda's async retry ladder and LANDED on the + // queue, whereas an Errors alarm fires as soon as invocations start + // failing, catching a systemic break (IAM regression, AgentCore quota + // exhaustion, dependency outage) at the first faulting invocation. + // Mirrors `task-orchestrator.ts` OrchestratorErrorAlarm's idiom. + template.hasResourceProperties('AWS::CloudWatch::Alarm', { + MetricName: 'Errors', + Namespace: 'AWS/Lambda', + // metricErrors() defaults to Sum — the correct statistic for an + // error COUNT (vs Average/Maximum). Pin it so a future edit that + // swaps the statistic can't silently change the alarm's semantics. + Statistic: 'Sum', + Period: 300, + Threshold: 1, + EvaluationPeriods: 2, + TreatMissingData: 'notBreaching', + Dimensions: Match.arrayWith([ + Match.objectLike({ + Name: 'FunctionName', + Value: Match.objectLike({ + Ref: Match.stringLikeRegexp('WebhookProcessorFn'), + }), + }), + ]), + }); + }); + test('wires the DLQ as the processor Lambda async-invoke dead-letter target', () => { // The queue existing is not enough — it must be bound to the // processor function's DeadLetterConfig or failed async invokes diff --git a/docs/guides/DEPLOY_PREVIEW_SCREENSHOTS_GUIDE.md b/docs/guides/DEPLOY_PREVIEW_SCREENSHOTS_GUIDE.md index f3c4ff40f..d9283254b 100644 --- a/docs/guides/DEPLOY_PREVIEW_SCREENSHOTS_GUIDE.md +++ b/docs/guides/DEPLOY_PREVIEW_SCREENSHOTS_GUIDE.md @@ -175,6 +175,35 @@ Then tail the function's CloudWatch log group. Common silent skips: - `skipped_no_url` — the `success` status didn't include `environment_url`. Some providers post URL-less success events; the next push usually carries the URL. - `No open PR found for SHA after retries` — the deploy provider built and reported faster than the agent could `gh pr create` (race window > 35s). Rare; redeliver the webhook from GitHub's UI to retry. +### No screenshots at all: check the processor alarms and DLQ + +The receiver Lambda async-invokes the processor (`InvocationType: Event`) and returns `200` to GitHub as soon as that invoke is accepted, so a *processor*-side fault never propagates back — GitHub sees success and never redelivers. (Only a failure to even enqueue the invoke returns `500`.) Two operator-visible signals catch a hard processor fault that would otherwise stop screenshots silently. + +**Important:** the processor handler is best-effort by design — it catches its own per-step operational failures (bad token, AgentCore/S3/comment-post errors) and returns success, so those show up as tagged log events (see the section above), **not** as Lambda `Errors`. Both alarms below fire only on faults that *escape* the handler: an init-time crash (missing env at cold start, bundling defect), an unhandled throw in an unguarded path, or the 120s hard timeout. For a swallowed operational failure (e.g. a revoked S3/AgentCore permission), watch the processor **logs**, not these alarms. + +- **`WebhookProcessorErrorAlarm`** — fires on the processor's Lambda `Errors` metric (`>= 1` error in a 5-minute period, sustained for 2 evaluation periods). Early signal: an invocation is faulting *now*. +- **`WebhookProcessorDlqDepthAlarm`** — fires when such a failed invocation has survived Lambda's built-in async retries and landed on the DLQ (construct id `WebhookProcessorDlq`; 14-day retention, SSL-enforced). Backstop: a payload is now parked undelivered. (The queue also gets SSE via SQS's service-side default; the construct doesn't set an explicit `encryption`.) + +The two catch the same failure class, but the `Errors` alarm trips one evaluation window sooner (before retries exhaust onto the DLQ). Find and inspect the DLQ — the construct sets no explicit queue name, so its physical name is CloudFormation-generated and *contains* the `WebhookProcessorDlq` construct id: + +```bash +# Find the DLQ URL (physical name contains the construct id) +aws sqs list-queues --region us-east-1 \ + --query "QueueUrls[?contains(@, 'WebhookProcessorDlq')]" --output text + +# How many failed invocations are parked? +aws sqs get-queue-attributes --region us-east-1 \ + --queue-url \ + --attribute-names ApproximateNumberOfMessages + +# Peek at a parked event (the original async-invoke payload + Lambda error context) +aws sqs receive-message --region us-east-1 \ + --queue-url --max-number-of-messages 1 \ + --visibility-timeout 0 +``` + +The message body is the original async-invoke event; the `RequestContext`/error attributes show why Lambda gave up. Fix the root cause (re-check the processor's IAM grants, AgentCore Browser quota, and the GitHub/Linear token secrets). The event source is GitHub, not the DLQ, so recovery is to **redeliver the webhook from GitHub's UI** (a Lambda async-invoke DLQ has no automatic re-invoke path — the parked messages are event copies for diagnosis). Purge the queue (`aws sqs purge-queue`) once the alarm has cleared and you no longer need the payloads. + ### Screenshot lands on GitHub PR but not on Linear The GitHub-side post is the primary path; Linear is opt-in and best-effort. Skipping the Linear post is normal if you don't have Linear configured. If you do, look for the processor log line `Linear identifier did not resolve to an issue` — usually means: diff --git a/docs/src/content/docs/using/Deploy-preview-screenshots-guide.md b/docs/src/content/docs/using/Deploy-preview-screenshots-guide.md index cd01304e3..bc28e0d40 100644 --- a/docs/src/content/docs/using/Deploy-preview-screenshots-guide.md +++ b/docs/src/content/docs/using/Deploy-preview-screenshots-guide.md @@ -179,6 +179,35 @@ Then tail the function's CloudWatch log group. Common silent skips: - `skipped_no_url` — the `success` status didn't include `environment_url`. Some providers post URL-less success events; the next push usually carries the URL. - `No open PR found for SHA after retries` — the deploy provider built and reported faster than the agent could `gh pr create` (race window > 35s). Rare; redeliver the webhook from GitHub's UI to retry. +### No screenshots at all: check the processor alarms and DLQ + +The receiver Lambda async-invokes the processor (`InvocationType: Event`) and returns `200` to GitHub as soon as that invoke is accepted, so a *processor*-side fault never propagates back — GitHub sees success and never redelivers. (Only a failure to even enqueue the invoke returns `500`.) Two operator-visible signals catch a hard processor fault that would otherwise stop screenshots silently. + +**Important:** the processor handler is best-effort by design — it catches its own per-step operational failures (bad token, AgentCore/S3/comment-post errors) and returns success, so those show up as tagged log events (see the section above), **not** as Lambda `Errors`. Both alarms below fire only on faults that *escape* the handler: an init-time crash (missing env at cold start, bundling defect), an unhandled throw in an unguarded path, or the 120s hard timeout. For a swallowed operational failure (e.g. a revoked S3/AgentCore permission), watch the processor **logs**, not these alarms. + +- **`WebhookProcessorErrorAlarm`** — fires on the processor's Lambda `Errors` metric (`>= 1` error in a 5-minute period, sustained for 2 evaluation periods). Early signal: an invocation is faulting *now*. +- **`WebhookProcessorDlqDepthAlarm`** — fires when such a failed invocation has survived Lambda's built-in async retries and landed on the DLQ (construct id `WebhookProcessorDlq`; 14-day retention, SSL-enforced). Backstop: a payload is now parked undelivered. (The queue also gets SSE via SQS's service-side default; the construct doesn't set an explicit `encryption`.) + +The two catch the same failure class, but the `Errors` alarm trips one evaluation window sooner (before retries exhaust onto the DLQ). Find and inspect the DLQ — the construct sets no explicit queue name, so its physical name is CloudFormation-generated and *contains* the `WebhookProcessorDlq` construct id: + +```bash +# Find the DLQ URL (physical name contains the construct id) +aws sqs list-queues --region us-east-1 \ + --query "QueueUrls[?contains(@, 'WebhookProcessorDlq')]" --output text + +# How many failed invocations are parked? +aws sqs get-queue-attributes --region us-east-1 \ + --queue-url \ + --attribute-names ApproximateNumberOfMessages + +# Peek at a parked event (the original async-invoke payload + Lambda error context) +aws sqs receive-message --region us-east-1 \ + --queue-url --max-number-of-messages 1 \ + --visibility-timeout 0 +``` + +The message body is the original async-invoke event; the `RequestContext`/error attributes show why Lambda gave up. Fix the root cause (re-check the processor's IAM grants, AgentCore Browser quota, and the GitHub/Linear token secrets). The event source is GitHub, not the DLQ, so recovery is to **redeliver the webhook from GitHub's UI** (a Lambda async-invoke DLQ has no automatic re-invoke path — the parked messages are event copies for diagnosis). Purge the queue (`aws sqs purge-queue`) once the alarm has cleared and you no longer need the payloads. + ### Screenshot lands on GitHub PR but not on Linear The GitHub-side post is the primary path; Linear is opt-in and best-effort. Skipping the Linear post is normal if you don't have Linear configured. If you do, look for the processor log line `Linear identifier did not resolve to an issue` — usually means: From b822a393a02e392e0d73bbbea56e0fb89e66ef1a Mon Sep 17 00:00:00 2001 From: scottschreckengaust <345885+scottschreckengaust@users.noreply.github.com> Date: Wed, 29 Jul 2026 12:35:09 +0000 Subject: [PATCH 2/2] fix(infra): make webhook Errors alarm 1-of-2 (datapointsToAlarm) + correct timing docs (#284) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reviewers @theagenticguy + @isadeks caught that WebhookProcessorErrorAlarm omitted datapointsToAlarm, so CloudWatch defaulted it to evaluationPeriods (2) — a 2-of-2 consecutive alarm. Because Lambda Errors is a sparse invocation metric, a single-event crash lands its retries in one 5-minute period and NOT_BREACHING fills the neighbour, so the alarm never fires. - Add `datapointsToAlarm: 1` → 1-of-2 (any single breaching period alarms; evaluation range stays 2). treatMissingData: NOT_BREACHING unchanged. - Pin `DatapointsToAlarm: 1` and `ComparisonOperator: GreaterThanOrEqualToThreshold` in the construct test (both carry the alarm semantics and previously rode on CDK defaults). - Reword the ERROR_ALARM_EVALUATION_PERIODS JSDoc and the timing comment: 1-of-2, not "sustained"; "fires no later than — and, when the retry ladder straddles a period boundary, one window before — the DLQ-depth alarm." - Update DEPLOY_PREVIEW_SCREENSHOTS_GUIDE runbook (+ regenerated Starlight mirror): Errors alarm fires on ANY 5-min period with >=1 error. Refs #284 Co-authored-by: Claude Opus 4.8 --- .../github-screenshot-integration.ts | 25 +++++++++++++------ .../github-screenshot-integration.test.ts | 8 ++++++ .../DEPLOY_PREVIEW_SCREENSHOTS_GUIDE.md | 4 +-- .../using/Deploy-preview-screenshots-guide.md | 4 +-- 4 files changed, 29 insertions(+), 12 deletions(-) diff --git a/cdk/src/constructs/github-screenshot-integration.ts b/cdk/src/constructs/github-screenshot-integration.ts index 3446a02c8..739e5a7c3 100644 --- a/cdk/src/constructs/github-screenshot-integration.ts +++ b/cdk/src/constructs/github-screenshot-integration.ts @@ -43,7 +43,10 @@ const DLQ_ALARM_PERIOD_MINUTES = 5; /** Processor Lambda-Errors alarm metric period (minutes). */ const ERROR_ALARM_PERIOD_MINUTES = 5; -/** Processor Lambda-Errors alarm sustained evaluation periods. */ +/** Processor Lambda-Errors alarm evaluation range (periods). Paired with + * `datapointsToAlarm: 1` below, this is 1-of-2, NOT 2-of-2: any single + * breaching period alarms. Do not remove `datapointsToAlarm` — CloudWatch + * would default it back to this value and a one-off crash would never fire. */ const ERROR_ALARM_EVALUATION_PERIODS = 2; /** Async screenshot-processor Lambda memory (MB). */ @@ -245,19 +248,25 @@ export class GitHubScreenshotIntegration extends Construct { // therefore fires on the faults that ESCAPE the handler: an init-time // crash (missing env at cold start, bundling defect), an unhandled // throw in an unguarded path, or the 120s hard timeout. That is the - // same failure class the DLQ eventually catches, but the Errors metric - // trips one evaluation window sooner — before Lambda's async-retry - // ladder has landed a message on the DLQ. threshold 1 / 2 eval periods - // matches the AC; the metricErrors + 2-eval-period shape follows - // ``task-orchestrator.ts`` OrchestratorErrorAlarm (which uses - // threshold 3 — we use 1 because any escaped error here is - // significant). + // same failure class the DLQ eventually catches, but before Lambda's async-retry + // ladder has landed a message on the DLQ. The Errors metric is stamped + // at invocation time, so this alarm fires no later than — and, when the + // retry ladder straddles a period boundary, one window before — the + // DLQ-depth alarm. threshold 1 / 1-of-2 eval periods matches the AC; + // the metricErrors shape follows ``task-orchestrator.ts`` + // OrchestratorErrorAlarm (which uses threshold 3 — we use 1 because any + // escaped error here is significant). this.processorErrorAlarm = new cloudwatch.Alarm(this, 'WebhookProcessorErrorAlarm', { metric: this.webhookProcessorFn.metricErrors({ period: Duration.minutes(ERROR_ALARM_PERIOD_MINUTES), }), threshold: 1, evaluationPeriods: ERROR_ALARM_EVALUATION_PERIODS, + // Fire on the FIRST breaching datapoint. Unset, this defaults to + // evaluationPeriods (2 consecutive), which for a sparse invocation + // metric means a single-event crash never alarms: its retries land + // in one period and NOT_BREACHING fills the neighbour. + datapointsToAlarm: 1, alarmDescription: 'Screenshot webhook processor Lambda errors breached threshold — deploy-preview screenshots are failing (check IAM, AgentCore Browser quota, and dependency health)', treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING, diff --git a/cdk/test/constructs/github-screenshot-integration.test.ts b/cdk/test/constructs/github-screenshot-integration.test.ts index 68c7b4383..d519da46b 100644 --- a/cdk/test/constructs/github-screenshot-integration.test.ts +++ b/cdk/test/constructs/github-screenshot-integration.test.ts @@ -112,7 +112,15 @@ describe('GitHubScreenshotIntegration construct', () => { Statistic: 'Sum', Period: 300, Threshold: 1, + // >= 1, not > 1. Relies on a CDK default otherwise, and + // GreaterThanThreshold would silently require TWO errors to fire. + ComparisonOperator: 'GreaterThanOrEqualToThreshold', EvaluationPeriods: 2, + // 1-of-2, not 2-of-2. Unset, DatapointsToAlarm defaults to + // EvaluationPeriods, and a single-event crash would never alarm + // (its Errors datapoints land in one period; NOT_BREACHING fills + // the neighbour). Pinned so that regression is caught here. + DatapointsToAlarm: 1, TreatMissingData: 'notBreaching', Dimensions: Match.arrayWith([ Match.objectLike({ diff --git a/docs/guides/DEPLOY_PREVIEW_SCREENSHOTS_GUIDE.md b/docs/guides/DEPLOY_PREVIEW_SCREENSHOTS_GUIDE.md index d9283254b..71a315cd1 100644 --- a/docs/guides/DEPLOY_PREVIEW_SCREENSHOTS_GUIDE.md +++ b/docs/guides/DEPLOY_PREVIEW_SCREENSHOTS_GUIDE.md @@ -181,10 +181,10 @@ The receiver Lambda async-invokes the processor (`InvocationType: Event`) and re **Important:** the processor handler is best-effort by design — it catches its own per-step operational failures (bad token, AgentCore/S3/comment-post errors) and returns success, so those show up as tagged log events (see the section above), **not** as Lambda `Errors`. Both alarms below fire only on faults that *escape* the handler: an init-time crash (missing env at cold start, bundling defect), an unhandled throw in an unguarded path, or the 120s hard timeout. For a swallowed operational failure (e.g. a revoked S3/AgentCore permission), watch the processor **logs**, not these alarms. -- **`WebhookProcessorErrorAlarm`** — fires on the processor's Lambda `Errors` metric (`>= 1` error in a 5-minute period, sustained for 2 evaluation periods). Early signal: an invocation is faulting *now*. +- **`WebhookProcessorErrorAlarm`** — fires on the processor's Lambda `Errors` metric: **any** 5-minute period with `>= 1` error alarms (evaluation range 2 periods, 1 datapoint to alarm). Early signal: an invocation is faulting *now*. - **`WebhookProcessorDlqDepthAlarm`** — fires when such a failed invocation has survived Lambda's built-in async retries and landed on the DLQ (construct id `WebhookProcessorDlq`; 14-day retention, SSL-enforced). Backstop: a payload is now parked undelivered. (The queue also gets SSE via SQS's service-side default; the construct doesn't set an explicit `encryption`.) -The two catch the same failure class, but the `Errors` alarm trips one evaluation window sooner (before retries exhaust onto the DLQ). Find and inspect the DLQ — the construct sets no explicit queue name, so its physical name is CloudFormation-generated and *contains* the `WebhookProcessorDlq` construct id: +The two catch the same failure class, but the `Errors` alarm fires no later than — and, when the retry ladder straddles a period boundary, one window before — the DLQ-depth alarm (the `Errors` metric is stamped at invocation time, before retries exhaust onto the DLQ). Find and inspect the DLQ — the construct sets no explicit queue name, so its physical name is CloudFormation-generated and *contains* the `WebhookProcessorDlq` construct id: ```bash # Find the DLQ URL (physical name contains the construct id) diff --git a/docs/src/content/docs/using/Deploy-preview-screenshots-guide.md b/docs/src/content/docs/using/Deploy-preview-screenshots-guide.md index bc28e0d40..36c49637f 100644 --- a/docs/src/content/docs/using/Deploy-preview-screenshots-guide.md +++ b/docs/src/content/docs/using/Deploy-preview-screenshots-guide.md @@ -185,10 +185,10 @@ The receiver Lambda async-invokes the processor (`InvocationType: Event`) and re **Important:** the processor handler is best-effort by design — it catches its own per-step operational failures (bad token, AgentCore/S3/comment-post errors) and returns success, so those show up as tagged log events (see the section above), **not** as Lambda `Errors`. Both alarms below fire only on faults that *escape* the handler: an init-time crash (missing env at cold start, bundling defect), an unhandled throw in an unguarded path, or the 120s hard timeout. For a swallowed operational failure (e.g. a revoked S3/AgentCore permission), watch the processor **logs**, not these alarms. -- **`WebhookProcessorErrorAlarm`** — fires on the processor's Lambda `Errors` metric (`>= 1` error in a 5-minute period, sustained for 2 evaluation periods). Early signal: an invocation is faulting *now*. +- **`WebhookProcessorErrorAlarm`** — fires on the processor's Lambda `Errors` metric: **any** 5-minute period with `>= 1` error alarms (evaluation range 2 periods, 1 datapoint to alarm). Early signal: an invocation is faulting *now*. - **`WebhookProcessorDlqDepthAlarm`** — fires when such a failed invocation has survived Lambda's built-in async retries and landed on the DLQ (construct id `WebhookProcessorDlq`; 14-day retention, SSL-enforced). Backstop: a payload is now parked undelivered. (The queue also gets SSE via SQS's service-side default; the construct doesn't set an explicit `encryption`.) -The two catch the same failure class, but the `Errors` alarm trips one evaluation window sooner (before retries exhaust onto the DLQ). Find and inspect the DLQ — the construct sets no explicit queue name, so its physical name is CloudFormation-generated and *contains* the `WebhookProcessorDlq` construct id: +The two catch the same failure class, but the `Errors` alarm fires no later than — and, when the retry ladder straddles a period boundary, one window before — the DLQ-depth alarm (the `Errors` metric is stamped at invocation time, before retries exhaust onto the DLQ). Find and inspect the DLQ — the construct sets no explicit queue name, so its physical name is CloudFormation-generated and *contains* the `WebhookProcessorDlq` construct id: ```bash # Find the DLQ URL (physical name contains the construct id)