Add opt-in Cosmos Binary encoding for Direct item, query, and batch operations

[JulienEllie]

Description

Adds opt-in Cosmos Binary encoding for Direct-mode document operations to avoid JSON/base64 expansion for native binary item fields.

Enable with either:

AZURE_COSMOS_BINARY_ENCODING_ENABLED=true
-Dazure.cosmos.binaryEncodingEnabled=true

The feature is disabled by default and does not change the public item or serializer APIs.

Protocol background

Cosmos Binary documents use the 0x80 format marker and support native binary scalar values rather than representing byte[] as JSON base64 strings.

Transactional batch and bulk use the service's existing wire contract:

RNTBD Batch
└── RecordIO
    └── HybridRow V1 BatchOperation / BatchResult
        └── resourceBody: item document bytes

This PR implements only the fixed Cosmos batch schemas; it does not add a general-purpose HybridRow engine.

Supported operations

Surface	Behavior when enabled
Point create/read/replace/upsert/delete	Cosmos Binary in Direct mode
Query / partition-scoped readAllItems	Negotiates `JsonText
Current-version Change Feed	Requests Cosmos Binary with ContentSerializationFormat; live validated
All Versions and Deletes Change Feed	Same binary wire path; unit/contract tested, but not live validated because no AVAD-enabled account was available
Transactional batch	RecordIO/HybridRow framing; create/replace/upsert item bodies use Cosmos Binary
Ordinary bulk	RecordIO/HybridRow framing with Cosmos Binary item bodies
Transactional bulk	Uses the transactional-batch path
Standalone patch	Existing JSON request body
Patch inside transactional batch	JSON patch body inside HybridRow framing
Gateway, emulator, and thin client	Existing JSON behavior
Requests with pre/post triggers	Existing JSON behavior

Pre-serialized JSON byte[] inputs are parsed and re-encoded before a request advertises Cosmos Binary. Existing 0x80 input remains unchanged. Custom CosmosItemSerializer hooks, ID validation, tracking IDs, diagnostics, retry bytes, session tokens, and content-response-disabled behavior remain in their existing paths.

Explicit remaining gap: generic raw ReadFeed

Generic raw OperationType.ReadFeed remains text. This does not include partition-scoped public readAllItems, which Java implements as a SQL query and therefore receives binary query responses.

This boundary follows the service behavior documented and tested by the .NET SDK:

• .NET only sets SupportedSerializationFormats for OperationType.Query and documents partition-key ReadFeed binary responses as a backend bug.
• .NET Direct/Gateway tests require ordinary ReadFeed to remain text even when Cosmos Binary is advertised.
• Change Feed uses a separate, explicit ContentSerializationFormat contract, which this PR mirrors.

Relying on the documented backend bug would make behavior partition- and routing-dependent, so generic raw ReadFeed is intentionally excluded pending a supported service contract. No matching Java issue currently documents this limitation; a follow-up issue can track it if the service adds stable negotiation.

Parser and wire hardening

The implementation includes:

• strict UTF-8 decoding;
• maximum nesting depth enforcement;
• scope length, count, and allocation validation;
• reference target and cycle protection;
• unsigned integer and compressed-length bounds;
• duplicate-key compatibility with physical member-count validation;
• RecordIO CRC and response-count validation;
• strict HybridRow schema, sparse field, duplicate field, and trailing-byte checks;
• structured SDK deserialization failures for malformed server responses;
• exact encoded-size batch splitting.

Unsupported user-dictionary string markers require out-of-band dictionary state and are rejected. Normal supported service responses do not provide such a dictionary.

Validation

Focused Failsafe coverage currently passes:

completed: 48
errors: 0
failures: 0
skipped: 0

Coverage includes marker/golden vectors, every-marker handling, truncation at every byte boundary, hostile lengths/counts, strict UTF-8, native Jackson binary nodes, custom serializer behavior, RNTBD format tokens, exact .NET HybridRow vectors, CRC failures, duplicate sparse fields, and exact batch size boundaries.

Live Direct-mode validation against Azure Cosmos DB covered:

• native binary point create/read/replace/upsert;
• conditional transactional batch operations;
• mixed JSON patch plus native binary batch operations;
• cross-partition bulk;
• binary query responses with exact bytes;
• current-version Change Feed with exact bytes and a no-changes/continuation page.

AVAD uses the same binary response selection and parser path, but remains explicitly not live validated due to the lack of an AVAD-enabled account.

Spotless and git diff --check pass. Meaningful Revapi validation should run under the repository-supported JDK in CI; locally available JDK 25/26 trigger unrelated Revapi scanner failures.

Preliminary size/performance evidence

Directional local measurements for documents dominated by 8, 64, and 256 KiB binary fields showed approximately:

• 25% smaller wire payloads by eliminating base64 expansion;
• 3–4x faster serialization in the local directional loop.

These timings are preliminary and are not presented as JMH benchmark results or universal document-size guarantees.

Risk and rollback

The feature is disabled by default and restricted to Direct document paths. Removing the environment variable or JVM property restores existing JSON behavior without application changes.

Reviewer guide

1. Add Cosmos binary JSON codec — value encoding/decoding and hostile-input validation.
2. Add HybridRow batch wire codec — fixed batch schemas and RecordIO framing.
3. Enable opt-in Cosmos binary encoding — serializer, point, query, Change Feed, batch, bulk, and RNTBD integration.
4. Document Cosmos binary encoding — changelog and supported scope.

All SDK Contribution checklist:

• The pull request does not introduce [breaking changes]
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message.

Testing Guidelines

• Pull request includes test coverage for the included changes.

[github-actions]

Thank you for your contribution @JulienEllie! We will review the pull request and get back to you soon.

[azure-pipelines]

Azure Pipelines:
Successfully started running 2 pipeline(s).
30 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[JulienEllie]

@microsoft-github-policy-service agree company="Walmart Inc."

[azure-pipelines]

Commenter does not have sufficient privileges for PR 49834 in repo Azure/azure-sdk-for-java

[JulienEllie]

@microsoft-github-policy-service agree company="Walmart Inc."

[microsoft-github-policy-service]

@JulienEllie the command you issued was incorrect. Please try again.

Examples are:

@microsoft-github-policy-service agree

and

@microsoft-github-policy-service agree company="your company"

[JulienEllie]

@microsoft-github-policy-service agree