Skip to content

feat(lambda-runtime): log non-2xx Lambda Runtime API responses with status and body#1109

Open
darklight3it wants to merge 4 commits intomainfrom
dmelfi/timeout-fix-simple
Open

feat(lambda-runtime): log non-2xx Lambda Runtime API responses with status and body#1109
darklight3it wants to merge 4 commits intomainfrom
dmelfi/timeout-fix-simple

Conversation

@darklight3it
Copy link

@darklight3it darklight3it commented Feb 25, 2026

Summary

When the Lambda Runtime API returns a non-2xx response (e.g., HTTP 410 for timeout, 500 for internal errors), the runtime previously provided no visibility into these failures. Customers had no way to diagnose why their Lambda invocations were failing at the runtime API level.

Solution

Add comprehensive error logging for all non-2xx responses from the Lambda Runtime API, while maintaining backward compatibility by preserving the existing behavior of returning Ok(()) and avoiding backward compatibility issue with RuntimeApiClientFuture which is unfortunately public.

Changes

1. Created log_or_print! macro (lambda-runtime/src/macros.rs)

  • New crate-internal macro that dispatches to tracing::error! when a tracing dispatcher is set
  • Falls back to eprintln! when no tracing subscriber is configured
  • Consistent with existing codebase patterns for dual logging support

2. Enhanced error logging in RuntimeApiClientFuture (lambda-runtime/src/layers/api_client.rs)

  • Added logging for non-2xx HTTP responses with status code details
  • Special handling for HTTP 410 (Gone) with additional "Lambda function timeout" message
  • Added logging for request building errors (First state)
  • Added logging for network/transport errors (Second state)
  • Maintains backward compatibility: Still returns Ok(()) for non-2xx responses to avoid breaking existing runtime behavior

3. Added comprehensive test coverage

  • Added tracing-test dev dependency for testing logging behavior
  • Tests cover all error branches: 200 OK, 410 timeout, 500 error, 404 error, request build errors, and network errors
  • Tests verify both return values and log output using #[traced_test]

4. Updated lambda-runtime/src/runtime.rs

  • Applied log_or_print! macro to existing concurrency warning for consistency

Example log output

**410 Timeout:**
ERROR lambda_runtime::layers::api_client: Lambda Runtime API returned non-200 response status=410 Gone 

ERROR lambda_runtime::layers::api_client: Lambda function timeout!


**500 Internal Server Error:**
ERROR lambda_runtime::layers::api_client: Lambda Runtime API returned non-200 response status=500 Internal Server Error


**Network failure:**
ERROR lambda_runtime::layers::api_client: Lambda Runtime API request failed error=...

Backward Compatibility

This change is fully backward compatible:

  • No changes to public API surface
  • RuntimeApiClientFuture still returns Ok(()) for non-2xx responses (preserving existing behavior)
  • Runtime continues processing events after API errors (no change in control flow)
  • Only adds observability through logging

Testing

Run tests with:

cd lambda-runtime && cargo test layers::api_client::tests --lib
All 6 tests pass, 

@darklight3it
Copy link
Author

Added an issue explaining the problem

#1110

Copy link
Collaborator

@jlizen jlizen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This approach looks good to me. It should keep the worker id span intact if using tracing, which is nice.

@@ -0,0 +1,10 @@
// Logs using tracing `error!` if a dispatcher is set, otherwise falls back to `eprintln!`.
macro_rules! log_or_print {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i like the new macro interface

Ok(resp) if !resp.status().is_success() => {
let status = resp.status();

log_or_print!(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: worth a TODO comment linking to the follow-up issue about the body?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants