Skip to content

feat(common): add serialized error payloads for data-bearing error#3751

Draft
jiengup wants to merge 1 commit into
apache:masterfrom
jiengup:error-payloads
Draft

feat(common): add serialized error payloads for data-bearing error#3751
jiengup wants to merge 1 commit into
apache:masterfrom
jiengup:error-payloads

Conversation

@jiengup

@jiengup jiengup commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

variants

Server now serializes the fields of data-bearing error variants (e.g. PartitionNotFound, StreamIdNotFound) into a binary payload and sends it alongside the error code. Client transports (QUIC, TCP, WebSocket, VSR) read this payload and reconstruct the full error, enabling contextual error messages on the client side. The Python SDK e2e test is updated to match the new dynamic IDs.

Which issue does this PR address?

Closes #3735

Rationale

Due to &[] payload in send_error_response<T> in core/server/src/sender/mod.rs here:

pub(crate) async fn send_error_response<T>(
stream: &mut T,
error: IggyError,
) -> Result<(), IggyError>
where
T: AsyncReadExt + AsyncWriteExt + Unpin,
{
send_response(stream, &error.as_code().to_le_bytes(), &[]).await
}

Error payloads (partition_id, topic_id, stream_id, etc.) were systematically dropped on the wire for
TCP, QUIC, WebSocket, and VSR transports. Clients received fielded variants filled with Default values,
producing byte-identical errors for completely different requests.

Server: PartitionNotFound(5, topic_42, stream_1)
 └─ send_error_response → [3007_u32] + empty body Wire: [0xDF 0x0B 0x00 0x00] [0x00 0x00 0x00 0x00] Client: from_code(3007) → FromRepr → Default fill
 └─ PartitionNotFound(0, Identifier(0), Identifier(0))

The HTTP transport was unaffected (error.to_string() serialized in JSON), proving the data existed
and was merely not transported.

What changed?

Server (send_error_response) now serializes the error’s fields into a structured binary payload
instead of an empty body. Client (TCP, QUIC, WebSocket, VSR) reads the body and reconstructs the
original variant with real field values via from_code_with_payload. The old empty-body path falls
back to from_code for backward compatibility with pre-fix servers.

A declarative macro (implement_error_payload!) generates both serialization and deserialization
from a single list of variant field declarations. An ErrorPayloadField trait encapsulates
the wire encoding for usize, u32, u64, u8, u16, String, and Identifier.
90 data-bearing variants are covered; unit variants and unsupported custom types produce empty payloads.

Local Execution

  • 489 unit tests passed (iggy + iggy_common + server, both non-VSR and VSR features)
  • Python test passes

AI Usage

  • CodeX
  • Serialization trait/macro implementation, unit test generation
  • All generated code compiled and tested; every line reviewed

Discussion

The current approach uses a declarative macro to cover 90 data-bearing variants.
A more elegant implementation should be considered:

Option 1: Proc macro / unified serialization module

A #[derive(ErrorPayload)] proc macro would eliminate hand-listing variants—new
field-bearing variants get payload support automatically. The read/write logic could
be further generalized into a WireField trait shared with WireEncode/WireDecode
in iggy_binary_protocol, avoiding duplicate serialization infrastructure.

Option 2: External crate

Replace the custom ErrorPayloadField trait and macro with a third-party binary
serialization crate. Identifier already derives Serialize/Deserialize, so
integration cost is low.

variants

Server now serializes the fields of data-bearing error variants (e.g.
PartitionNotFound, StreamIdNotFound) into a binary payload and sends
it alongside the error code. Client transports (QUIC, TCP, WebSocket,
VSR) read this payload and reconstruct the full error, enabling
contextual error messages on the client side. The Python SDK e2e test is
updated to match the new dynamic IDs.
@codecov

codecov Bot commented Jul 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.37500% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.99%. Comparing base (d9635b6) to head (f602613).

Files with missing lines Patch % Lines
core/common/src/error/iggy_error.rs 83.52% 24 Missing and 4 partials ⚠️
core/sdk/src/quic/quic_client.rs 66.66% 2 Missing and 1 partial ⚠️
core/sdk/src/tcp/tcp_client.rs 94.59% 0 Missing and 2 partials ⚠️
core/sdk/src/websocket/websocket_client.rs 66.66% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3751      +/-   ##
============================================
- Coverage     74.53%   70.99%   -3.55%     
  Complexity      969      969              
============================================
  Files          1306     1306              
  Lines        150028   144974    -5054     
  Branches     125462   120482    -4980     
============================================
- Hits         111829   102927    -8902     
- Misses        34700    38401    +3701     
- Partials       3499     3646     +147     
Components Coverage Δ
Rust Core 72.67% <84.37%> (-2.16%) ⬇️
Java SDK 62.73% <ø> (+0.02%) ⬆️
C# SDK 33.11% <ø> (-39.15%) ⬇️
Python SDK 92.27% <ø> (ø)
PHP SDK 84.52% <ø> (ø)
Node SDK 92.24% <ø> (ø)
Go SDK 43.21% <ø> (+0.12%) ⬆️
Files with missing lines Coverage Δ
core/server/src/sender/mod.rs 90.55% <100.00%> (+0.07%) ⬆️
core/sdk/src/tcp/tcp_client.rs 75.75% <94.59%> (+1.13%) ⬆️
core/sdk/src/websocket/websocket_client.rs 71.59% <66.66%> (-0.14%) ⬇️
core/sdk/src/quic/quic_client.rs 72.97% <66.66%> (-0.13%) ⬇️
core/common/src/error/iggy_error.rs 86.13% <83.52%> (-13.87%) ⬇️

... and 183 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jiengup

jiengup commented Jul 25, 2026

Copy link
Copy Markdown
Contributor Author

The discussion should be considered.
I need more suggestions about elegant implementation 😢

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Binary protocol drops error payloads — clients rebuild IggyError with Default fields (PartitionNotFound always reports 0/0/0)

1 participant