Cosmos: flow transactional batch failures through the execution strategy so they can be retried#38597
Cosmos: flow transactional batch failures through the execution strategy so they can be retried#38597AndriySvyryd with Copilot wants to merge 6 commits into
Conversation
…angesAsync Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR fixes Cosmos transactional batch failures bypassing the execution strategy retry pipeline by throwing on batch failure (instead of returning a “failed result”) and moving the execution-strategy scope to wrap the full SaveChanges batch sequence, allowing transient failures to reach ShouldRetryOn and be retried.
Changes:
- Replace the transactional batch “result” return pattern with a
CosmosTransactionalBatchExceptionthrown fromProcessBatchResponse. - Change
ICosmosClientWrapper.ExecuteTransactionalBatchAsyncto returnTaskand execute once (no per-batch execution strategy wrapping in the client). - Wrap the full set of transactional batches in a single execution strategy invocation inside
CosmosDatabaseWrapper, with retry skipping of already-committed operations, and add a functional test verifying the exception reaches the execution strategy.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| test/EFCore.Cosmos.FunctionalTests/CosmosTransactionalBatchTest.cs | Adds a functional test asserting transactional batch failures flow through the execution strategy and nothing is committed. |
| src/EFCore.Cosmos/Storage/Internal/ICosmosClientWrapper.cs | Updates the transactional batch API to return Task rather than a result object. |
| src/EFCore.Cosmos/Storage/Internal/CosmosTransactionalBatchException.cs | Introduces an internal exception type carrying errored entries and the underlying CosmosException. |
| src/EFCore.Cosmos/Storage/Internal/CosmosDatabaseWrapper.cs | Moves execution strategy retry scope to encompass the entire batch set and adds committed-operation skipping across retries. |
| src/EFCore.Cosmos/Storage/Internal/CosmosClientWrapper.cs | Executes transactional batches once and throws CosmosTransactionalBatchException on failure (instead of returning a failure result). |
…e transactional batch Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>
…fected; add test Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>
…currencyException for suppression Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>
| throw errorCode switch | ||
| { | ||
| HttpStatusCode.PreconditionFailed | ||
| => new DbUpdateConcurrencyException(CosmosStrings.UpdateConflict(effectiveEntries[0].Id), cosmosException, errorUpdateEntries), | ||
| HttpStatusCode.Conflict | ||
| => new DbUpdateException(CosmosStrings.UpdateConflict(effectiveEntries[0].Id), cosmosException, errorUpdateEntries), | ||
| _ | ||
| => new DbUpdateException(CosmosStrings.UpdateStoreException(effectiveEntries[0].Id), cosmosException, errorUpdateEntries) | ||
| }; |
There was a problem hiding this comment.
@copilot Make sure there's no duplication between this code and WrapUpdateException, also check other modified code for opportunities to refactor
There was a problem hiding this comment.
Extracted the CosmosException → DbUpdateException switch to a new public static CosmosClientWrapper.WrapUpdateException(Exception, string id, IReadOnlyList<IUpdateEntry>) method. ProcessBatchResponse now calls it directly, and CosmosDatabaseWrapper.WrapUpdateException computes the document id and delegates to the same method — removing the duplicate switch. Also cleaned up using System.Net (no longer needed in CosmosDatabaseWrapper) and replaced transaction.Entries.First().Entry.Context with allEntries[0].Context since allEntries was already available.
| { | ||
| if (batch.UpdateEntries.Count == 1 && _currentDbContext.Context.Database.AutoTransactionBehavior != AutoTransactionBehavior.Always) | ||
| { | ||
| // Skip the operations that were already committed by a previous execution strategy attempt. |
|
|
||
| foreach (var transaction in CreateTransactions(batch)) | ||
| { | ||
| // Skip the operations that were already committed by a previous execution strategy attempt. |
| var errorCode = response.StatusCode; | ||
| var errorEntries = response | ||
| var cosmosException = new CosmosException(response.ErrorMessage, errorCode, 0, response.ActivityId, response.RequestCharge); | ||
| var errorBatchEntries = response | ||
| .Select((opResult, index) => (opResult, index)) | ||
| .Where(r => r.opResult.StatusCode == errorCode) | ||
| .Select(r => entries[r.index].Entry) | ||
| .Select(r => entries[r.index]) | ||
| .ToList(); | ||
|
|
…pper.WrapUpdateException Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>
Cosmos transactional batch failures returned a result object from
ProcessBatchResponserather than throwing, so the failure surfaced as aDbUpdateExceptionoutside any execution-strategy scope and could never be retried by anExecutionStrategy(ShouldRetryOnwas never called).Changes
CosmosClientWrapper.ProcessBatchResponsenow computes the errored entries (those whose per-item status code matches the batch-level error code), wraps theCosmosExceptioninto aDbUpdateConcurrencyException(PreconditionFailed) orDbUpdateException(Conflict / other), and throws it directly.CosmosTransactionalBatchResultandCosmosTransactionalBatchExceptionare removed.ExecuteTransactionalBatchAsyncno longer wraps itself in the execution strategy; it runs once and returnsTask(interface updated).SaveChangesAsync—CosmosDatabaseWrappertakes an injectedIExecutionStrategyand wraps the whole set of batches in a singleExecuteAsync. ABatchExecutionState.CommittedOperationscounter lets a retry skip already-committed operations while preserving the original ordering of single-entry saves interleaved with transactional batches. The execution strategy unwrapsDbUpdateExceptionviaExecutionStrategy.CallOnWrappedExceptionto reach the innerCosmosExceptionbefore callingShouldRetryOn, so transient failures are correctly identified and retried.DbUpdateConcurrencyExceptionfrom a batch is suppressed viaOptimisticConcurrencyExceptionAsync,RowsAffectedis not incremented (matchingSaveAsyncsingle-entry behavior), whileCommittedOperationsstill advances so a subsequent retry skips the already-processed operation.Because a transactional batch is atomic, a failed batch commits nothing and is retried in place; deterministic transaction serialization across attempts makes index-based skipping of committed operations valid.
Result
The repro from the issue now behaves as expected — a transient batch failure reaches the strategy and is retried:
Two functional tests verify the behavior:
SaveChanges_transactional_batch_failure_flows_through_execution_strategy— a conflicting batch reachesShouldRetryOnand yieldsRetryLimitExceededException, with nothing committed.SaveChanges_suppressed_concurrency_exception_in_transactional_batch_reports_zero_rows_affected— a stale-ETag batch update suppressed by an interceptor returns 0 rows affected and leaves the store unchanged.