Skip to content

Add CRT HTTP/2 stability tests and JMH benchmarks#6723

Open
zoewangg wants to merge 3 commits intofeature/master/crtH2from
zoewang/addPerfTestsStabilityTests
Open

Add CRT HTTP/2 stability tests and JMH benchmarks#6723
zoewangg wants to merge 3 commits intofeature/master/crtH2from
zoewang/addPerfTestsStabilityTests

Conversation

@zoewangg
Copy link
Contributor

@zoewangg zoewangg commented Feb 9, 2026

Motivation and Context

This PR adds stability tests and JMH benchmarks for the CRT HTTP client with HTTP/2 protocol support. These tests validate CRT HTTP/2 behavior under high-concurrency
scenarios using real AWS services (Kinesis and Transcribe Streaming) that support HTTP/2.

Modifications

New Stability Tests:

  • KinesisCrtH2StabilityTest - Tests Kinesis SubscribeToShard streaming with CRT HTTP/2
  • TranscribeStreamingCrtStabilityTest - Tests Transcribe Streaming bidirectional streaming with CRT HTTP/2

New JMH Benchmark:

  • AwsCrtH2ClientBenchmark - Benchmarks CRT HTTP client with HTTP/2 over TLS

Refactoring:

  • Created KinesisBaseStabilityTest to eliminate code duplication between Kinesis stability tests
  • Created TranscribeStreamingBaseStabilityTest to eliminate code duplication between Transcribe Streaming stability tests

Benchmark Results (CRT HTTP/2 vs Netty HTTP/2):

Benchmark CRT (ops/s) Netty OpenSSL (ops/s) Improvement
Concurrent 12,287 ± 433 7,156 ± 91 +72%
Sequential 3,760 ± 185 2,487 ± 54 +51%

Testing

  • Stability tests require AWS credentials and create real Kinesis streams/Transcribe sessions
  • JMH benchmarks run against local mock H2 server

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

Checklist

  • I have read the CONTRIBUTING document
  • Local run of mvn install succeeds
  • My code follows the code style of this project
  • My change requires a change to the Javadoc documentation
  • I have updated the Javadoc documentation accordingly
  • I have added tests to cover my changes
  • All new and existing tests pass
  • I have added a changelog entry (N/A - test-only changes)

@zoewangg zoewangg requested a review from a team as a code owner February 9, 2026 22:29
@zoewangg zoewangg force-pushed the zoewang/addPerfTestsStabilityTests branch 2 times, most recently from 1e59327 to 11cfe9c Compare February 10, 2026 00:02
# Conflicts:
#	services/kinesis/src/it/java/software/amazon/awssdk/services/kinesis/AbstractTestCase.java
@zoewangg zoewangg force-pushed the zoewang/addPerfTestsStabilityTests branch from 1d38de5 to 05f773d Compare February 12, 2026 22:30
Comment on lines +130 to +131
assertThatThrownBy(() -> recorder.completeFuture().get(5, TimeUnit.SECONDS)).hasCauseInstanceOf(IOException.class)
.hasMessageContaining("socket");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated this test because it's a bit flaky, when I run it locally, it fails of socket timeout but when the CI runs it, it sometimes fails of operation timeout

Comment on lines -99 to +98
.hasMessageContaining("operation timed out");
.hasMessageContaining("socket");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here. Updated this test because it's a bit flaky, when I run it locally, it fails of socket timeout but when the CI runs it, it sometimes fails of operation timeout

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if it's flaky wouldn't we want to assert that it has either one of those strings?

@sonarqubecloud
Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
64.9% Coverage on New Code (required ≥ 80%)
4.8% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

Comment on lines +20 to +21
import software.amazon.awssdk.http.Protocol;
import software.amazon.awssdk.http.SdkHttpConfigurationOption;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove unused imports

Comment on lines -99 to +98
.hasMessageContaining("operation timed out");
.hasMessageContaining("socket");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if it's flaky wouldn't we want to assert that it has either one of those strings?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants