IcerbergIO Load test and benchmarking#39270
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces comprehensive load testing and benchmarking for IcebergIO. It includes a significant refactoring of existing integration test utilities to promote code reuse by moving common components into a shared module. Additionally, it adds support for IcebergIO performance tracking on the project's website, providing visibility into read and write performance metrics. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces integration and performance tests for IcebergIO (IcebergIOLT), along with its corresponding Gradle tasks, website documentation, and Looker metrics configurations. Additionally, it reorganizes the integration testing framework by moving common test utilities (such as load/stress test base classes, artifact clients, and resource managers) from the GCP-specific module to a shared common module under the org.apache.beam.it.common package, updating dependencies and imports across several modules. Regarding the feedback, a performance issue was identified in the new IcebergIOLT test where a Random object is instantiated for every processed element, which can cause significant CPU and garbage collection overhead; it is recommended to reuse a single transient Random instance and update its seed instead.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| private static class MapToIcebergFormat extends DoFn<Long, Row> { | ||
|
|
||
| private final int valueSizeBytes; | ||
|
|
||
| private MapToIcebergFormat(int valueSizeBytes) { | ||
| if (valueSizeBytes <= 0) { | ||
| throw new IllegalArgumentException("valueSizeBytes must be positive."); | ||
| } | ||
|
|
||
| this.valueSizeBytes = valueSizeBytes; | ||
| } | ||
|
|
||
| @ProcessElement | ||
| public void processElement(@Element Long index, OutputReceiver<Row> output) { | ||
|
|
||
| byte[] value = new byte[valueSizeBytes]; | ||
|
|
||
| /* | ||
| * Using the row index as the seed makes retries deterministic. Random data also prevents | ||
| * compression from making the generated payload significantly smaller than configured. | ||
| */ | ||
| new Random(index).nextBytes(value); | ||
|
|
||
| output.output(Row.withSchema(ROW_SCHEMA).addValues(index, value).build()); | ||
| } | ||
| } |
There was a problem hiding this comment.
Instantiating a new Random(index) object inside processElement for every single element processed by the pipeline introduces significant CPU overhead and garbage collection pressure. This is especially problematic in load and performance tests where millions or billions of elements are processed, as it can skew the benchmark results.
Instead, declare a single transient Random field in the DoFn class, lazily initialize it, and reuse it by calling setSeed(index) for each element to maintain determinism.
private static class MapToIcebergFormat extends DoFn<Long, Row> {
private final int valueSizeBytes;
private transient Random random;
private MapToIcebergFormat(int valueSizeBytes) {
if (valueSizeBytes <= 0) {
throw new IllegalArgumentException("valueSizeBytes must be positive.");
}
this.valueSizeBytes = valueSizeBytes;
}
@ProcessElement
public void processElement(@Element Long index, OutputReceiver<Row> output) {
if (random == null) {
random = new Random();
}
byte[] value = new byte[valueSizeBytes];
/*
* Using the row index as the seed makes retries deterministic. Random data also prevents
* compression from making the generated payload significantly smaller than configured.
*/
random.setSeed(index);
random.nextBytes(value);
output.output(Row.withSchema(ROW_SCHEMA).addValues(index, value).build());
}
}There was a problem hiding this comment.
I think this is true - would you mind addressing this?
damccorm
left a comment
damccorm
left a comment
There was a problem hiding this comment.
Thanks - just had a couple of questions
| * limitations under the License. | ||
| */ | ||
| package org.apache.beam.it.gcp.bigquery; | ||
| package org.apache.beam.it.common.bigquery; |
There was a problem hiding this comment.
Why did we need to move all of these to common? Some of these do seem GCP specific
There was a problem hiding this comment.
I moved them so Kafka, Iceberg, and other non-GCP IO tests could reuse the load-test infrastructure without depending on the whole GCP test module. I agree some utilities are still GCP-specific, so keeping them in clearly named gcp or dataflow packages within common may be a better separation.
| @@ -0,0 +1 @@ | |||
| ["This is a test artifact."] No newline at end of file | |||
There was a problem hiding this comment.
How is this (and the similar file below) used?
There was a problem hiding this comment.
It is used in GcsResourceManagerTest. I removed the other two that were not used. Thanks!
| private static class MapToIcebergFormat extends DoFn<Long, Row> { | ||
|
|
||
| private final int valueSizeBytes; | ||
|
|
||
| private MapToIcebergFormat(int valueSizeBytes) { | ||
| if (valueSizeBytes <= 0) { | ||
| throw new IllegalArgumentException("valueSizeBytes must be positive."); | ||
| } | ||
|
|
||
| this.valueSizeBytes = valueSizeBytes; | ||
| } | ||
|
|
||
| @ProcessElement | ||
| public void processElement(@Element Long index, OutputReceiver<Row> output) { | ||
|
|
||
| byte[] value = new byte[valueSizeBytes]; | ||
|
|
||
| /* | ||
| * Using the row index as the seed makes retries deterministic. Random data also prevents | ||
| * compression from making the generated payload significantly smaller than configured. | ||
| */ | ||
| new Random(index).nextBytes(value); | ||
|
|
||
| output.output(Row.withSchema(ROW_SCHEMA).addValues(index, value).build()); | ||
| } | ||
| } |
There was a problem hiding this comment.
I think this is true - would you mind addressing this?
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
2 participants
Successful run: https://github.com/apache/beam/actions/runs/29006124448/job/86078012033
Read Dataflow Job: 2026-07-09_02_11_56-9909184907203270916
Write Dataflow Job: 2026-07-09_02_00_15-6048170378328054140
Changes:
it.gcptoit.common:DefaultPipelineLauncherandAbstractPipelineLauncherLoadTestBaseThank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.