Skip to content

impl(bigquery): job level retry for queries - #6318

Open
alvarowolfx wants to merge 6 commits into
googleapis:mainfrom
alvarowolfx:impl-bq-retry-ctx
Open

impl(bigquery): job level retry for queries#6318
alvarowolfx wants to merge 6 commits into
googleapis:mainfrom
alvarowolfx:impl-bq-retry-ctx

Conversation

@alvarowolfx

Copy link
Copy Markdown
Contributor

When a query job fails with a retryable BigQuery error (for example backendError or rateLimitExceeded), the client now automatically creates a new job attempt with backoff instead of failing immediately. Move all logic for query execution away from builder layer (RunQuery) to the execution layer.

Towards #5844 #6218

@product-auto-label product-auto-label Bot added the api: bigquery Issues related to the BigQuery API. label Aug 10, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a RetryContext to handle job-level retries and re-issuances for BigQuery queries, refactoring the query execution path to delegate to this context and updating Query::until_done to retry on transient failures. Feedback on the changes highlights a critical compilation error in until_done where self is partially moved inside a loop, and suggests avoiding destructuring self at the loop's start. Additionally, it is recommended to pass self by value in reissue to avoid an unnecessary clone of the context.

Comment thread src/bigquery/src/query/query_handle.rs
Comment thread src/bigquery/src/query/execution.rs Outdated
@codecov

codecov Bot commented Aug 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.99329% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.27%. Comparing base (4ff2683) to head (49eaaf9).

Files with missing lines Patch % Lines
src/bigquery/src/query/execution.rs 97.50% 2 Missing ⚠️
src/bigquery/src/query/query_handle.rs 99.43% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6318      +/-   ##
==========================================
+ Coverage   96.26%   96.27%   +0.01%     
==========================================
  Files         282      282              
  Lines       73304    73508     +204     
==========================================
+ Hits        70563    70772     +209     
+ Misses       2741     2736       -5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@alvarowolfx

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a RetryContext to manage job-level retries and re-issuances for BigQuery queries, refactoring PostQueryExecutor and InsertJobExecutor to return raw responses and updating Query::until_done to handle retries in a loop. The review feedback focuses on minor Rust idiomatic improvements and optimizations, such as avoiding an unnecessary double reference (&&str) when passing project_id, removing an unnecessary .to_string() allocation on set_project_id, and using pattern matching instead of .take() on the owned retry_context option.

Comment thread src/bigquery/src/query/execution.rs Outdated
Comment thread src/bigquery/src/query/execution.rs Outdated
Comment thread src/bigquery/src/query/query_handle.rs Outdated
@alvarowolfx

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a RetryContext to manage job-level retries and re-issuances for BigQuery queries, refactoring PostQueryExecutor and InsertJobExecutor to return raw responses and updating Query::until_done to handle retryable errors. Feedback on the changes suggests deriving Debug for the public RunQuery struct to adhere to the repository style guide, and addresses an off-by-one error in RetryableJobErrors that allows an extra retry attempt beyond the configured limit.

pub(crate) job_service: Arc<JobService>,
pub(crate) request: RunQueryRequest,
pub(crate) project_id: Option<String>,
pub(crate) job_retry_policy: Arc<dyn JobRetryPolicy>,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

According to the repository style guide, all public types should implement both Debug and Clone to ensure interoperability. Since RunQuery is a public struct, please derive Debug for it (e.g., by changing #[derive(Clone)] to #[derive(Clone, Debug)] on line 31).

References
  1. Public types should implement both Debug and Clone to ensure interoperability. (link)
  2. Public types, such as request builders, should implement both Debug and Clone to ensure interoperability.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will do that in a separated PR. Need to figure out how to add #[derive(Debug)] on the generated types

}

#[derive(Clone, Debug)]
#[allow(dead_code)]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The attempt_limit in RetryableJobErrors actually allows attempt_limit + 1 total attempts due to an off-by-one check in its on_error implementation (which checks state.attempt_count >= self.attempt_limit). Since attempt_count starts at 0 for the first failure, an attempt_limit of 3 will allow attempts 0, 1, 2, and 3 (total of 4 attempts). Consider changing the check to state.attempt_count + 1 >= self.attempt_limit to strictly adhere to the attempt limit while keeping attempt_count 0-indexed for backoff calculations.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will fix in a separated PR

@alvarowolfx
alvarowolfx marked this pull request as ready for review August 10, 2026 18:24
@alvarowolfx
alvarowolfx requested a review from a team as a code owner August 10, 2026 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: bigquery Issues related to the BigQuery API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant