Skip to content

fix(spanner): preserve user call options in execute/read and consolidate test databases#9329

Open
cy-yun wants to merge 7 commits into
googleapis:mainfrom
cy-yun:fix-spanner-tests-9230
Open

fix(spanner): preserve user call options in execute/read and consolidate test databases#9329
cy-yun wants to merge 7 commits into
googleapis:mainfrom
cy-yun:fix-spanner-tests-9230

Conversation

@cy-yun

@cy-yun cy-yun commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Fix Spanner System Tests Reliability and Provisioning Overhead

This PR addresses the ongoing flakiness and high database provisioning overhead of the Spanner system tests (resolving #9230).

While this PR consolidates test database provisioning (reducing overall execution time by reusing databases), the test suite as a whole will still take a long time to run (~1.5 hours) due to the inherent slowness of operations like Backup and Restore. (Parallelizing these tests with paratest to bring the time down to minutes is planned for a subsequent follow-up).

Below is a breakdown of the specific problems and fixes implemented in this PR:

1. Dropped $options Array (Fixing ReadTest flakiness & SDK Bug)

  • Problem: An issue was identified where the user-provided $options array was inadvertently being omitted when making calls through Database::execute() and TransactionalReadTrait methods. In the test suite, this surfaced as flakiness in testReadFailsOnDeadlineExceeded (because the 1ms timeoutMillis option was not passed through, allowing the request to reach the backend). For users, this meant that custom RPC settings (like requestTimeout, headers, or retrySettings) passed to queries or reads were not being applied to the underlying gRPC requests.
  • Fix: Updated Operation::validateOptions() to centrally and explicitly whitelist the internal Spanner SDK options (begin, transactionType, etc.) so that they are cleanly consumed and ignored by OptionsValidator. This allowed us to safely pass the entire $options array straight from the user's Database::execute() call down to the underlying Operation layer without throwing unexpected option errors, restoring the proper client-side configuration behavior and fixing the test.

2. Test Database Provisioning Overhead

  • Problem: The system test suite was creating and dropping a new test database for every single test class, adding massive overhead to the already long execution time.
  • Fix: Introduced TestDatabaseManager to provision a single shared Spanner test database and reuse it across all test classes.

3. PostgreSQL DDL Compatibility

  • Problem: Reusing the same database across multiple test runs or classes caused CREATE TABLE commands to fail because the tables already existed.
  • Fix: Added IF NOT EXISTS to PostgreSQL DDL statements in the Pg*Test classes.

4. Emulator Transaction Leaks

  • Problem: When running tests against the emulator, unhandled or failed transactions were leaking and persisting on the emulator, causing subsequent test failures or slowdowns. Specifically, testLockHintReadWriteTransaction was leaving a transaction hanging without rollback.
  • Fix: Added a manual $res->transaction()->rollback() to testLockHintReadWriteTransaction, and added an unhandled exception catch block in Database::runTransaction to rollback non-single-use active transactions before bubbling up the exception.

5. BackupTest Reliability

  • Problem: BackupTest operations were flaking due to transient errors and improper polling configurations (e.g. timeoutMillis instead of maxPollingDurationSeconds for GAX long-running operations). The backup cancellation test was also flawed because it tried to wait for a cancelled backup to exist.
  • Fix: Updated long-running operation polling to use maxPollingDurationSeconds. Added retry logic for UNAVAILABLE errors in testRestoreDatabase. Split the backup creation and cancellation tests to prevent the cancellation from interfering with tests that depend on a successfully created backup.

6. PHP 8.1 Deprecation Warnings

7. CI Configuration

  • Problem: Spanner system tests were not executing in the standard CI pipeline.
  • Fix: Enabled the Spanner test suite in phpunit-system.xml.dist.

@cy-yun
cy-yun force-pushed the fix-spanner-tests-9230 branch 2 times, most recently from d69b03a to 169514a Compare July 8, 2026 19:41
@cy-yun cy-yun changed the title Fix spanner tests 9230 test(spanner): optimize and fix system tests and enable in CI Jul 8, 2026
@cy-yun
cy-yun force-pushed the fix-spanner-tests-9230 branch 2 times, most recently from bfd11cb to 41272ef Compare July 8, 2026 20:18
@cy-yun
cy-yun marked this pull request as ready for review July 8, 2026 20:34
@cy-yun
cy-yun requested a review from a team as a code owner July 8, 2026 20:34
Comment thread Spanner/tests/System/BackupTest.php Outdated
// Poll for completion with the extended timeout
$op->pollUntilComplete([
$op->pollUntilComplete(
[

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did the linter suggested this change? This change feels wrong to me. I mean it still is valid, but I would expect either the previous iteration or something like:

$op->pollUntilComplete(
    [
        'timeoutMillis' => ...
    ]
)

As it is right now, having the braces on the same level as the key feels off.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I still think the previous one should be valid 🤷

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fixed the formatting of the code such that it is consistent with the original & phpcbf is not upset


$this->assertTrue($backup->exists());
// Cancellation usually drops the backup. We don't assert exists()
// to avoid flakiness with asynchronous deletion.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, isn't this testing that if you execute a backup deletion and cancel it, it shouldn't be still available? Or am i misreading the test?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, this test is meant to test canceling a backup creation operation, not a deletion. Previously, the test was calling $op->pollUntilComplete() before $op->cancel(). That meant it was waiting for the backup to fully finish creating, and then issuing a cancel command on a completed operation (which basically does nothing), hence why it used to assert that the backup still existed.

By removing the polling, we are now properly canceling the operation while it is still in-progress. Because we cancelled it mid-flight, Spanner usually drops the incomplete backup, so asserting that it still exists makes the test extremely flaky.

@cy-yun
cy-yun force-pushed the fix-spanner-tests-9230 branch from 41272ef to 2e68fb4 Compare July 9, 2026 07:57
@product-auto-label product-auto-label Bot added the api: spanner Issues related to the Spanner API. label Jul 9, 2026
@cy-yun
cy-yun force-pushed the fix-spanner-tests-9230 branch from 2e68fb4 to 921a05c Compare July 9, 2026 20:33
@cy-yun cy-yun added kokoro:force-run Add this label to force Kokoro to re-run the tests. and removed kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Jul 14, 2026
@cy-yun
cy-yun force-pushed the fix-spanner-tests-9230 branch from 921a05c to 122401c Compare July 21, 2026 01:43
@cy-yun cy-yun changed the title test(spanner): optimize and fix system tests and enable in CI fix(spanner): preserve user call options in execute/read and consolidate test databases Jul 21, 2026
@cy-yun
cy-yun force-pushed the fix-spanner-tests-9230 branch 2 times, most recently from 4bd521e to bc64d80 Compare July 21, 2026 02:33
@cy-yun
cy-yun force-pushed the fix-spanner-tests-9230 branch from bc64d80 to 5798d9e Compare July 21, 2026 02:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: spanner Issues related to the Spanner API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants