Skip to content

Tests | Fix TVP query hint test timeouts against shared Azure SQL DB - #4593

Open
cheenamalhotra wants to merge 3 commits into
mainfrom
dev/cheena/bookish-guacamole
Open

Tests | Fix TVP query hint test timeouts against shared Azure SQL DB#4593
cheenamalhotra wants to merge 3 commits into
mainfrom
dev/cheena/bookish-guacamole

Conversation

@cheenamalhotra

@cheenamalhotra cheenamalhotra commented Aug 25, 2026

Copy link
Copy Markdown
Member

Problem

The sqlclient_manual_azure_* legs of sqlclient-pr have been failing intermittently with Execution Timeout Expired in TvpQueryHintsTests:

Microsoft.Data.SqlClient.SqlException : Execution Timeout Expired. The timeout period elapsed prior to
completion of the operation or the server is not responding.
   at Microsoft.Data.SqlClient.ManualTesting.Tests.TvpQueryHintsTests..ctor()

The timeouts always land on the CREATE TYPE / CREATE PROC / DROP PROC; DROP TYPE statements — never on a TVP data path.

The failures also appear across many unrelated PRs, and only ever in the Azure SQL legs.

Root cause

Two independent issues combined:

  1. UserDefinedType.DropObject never dropped anything. It guarded the DROP with OBJECT_ID(...), but user-defined types live in sys.types, not sys.objects, so OBJECT_ID always returns NULL and the DROP was silently skipped. Every test run has been orphaning its table types into the target database. Locally, a single pass of the UDT-using manual tests leaked 108 types. In CI these accumulate indefinitely against a shared database, making metadata operations progressively slower — which matches the gradual onset around Aug 20.

  2. TvpQueryHintsTests issued 20 DDL statements for 5 tests. Each test constructor created and dropped an identically shaped table type and stored procedure, multiplying schema-modification lock contention on a database that several platform legs hit concurrently.

Fix

  • OBJECT_IDTYPE_ID in UserDefinedType.DropObject, so types are actually dropped.
  • TvpQueryHintsTests now uses an IClassFixture to create the type and procedure once per class (5x less DDL), with a 120s command timeout on the fixture connection to absorb remaining contention.

Validation

Run against SQL Server 2022:

  • All 5 TvpQueryHintsTests pass.
  • Broader UDT-dependent suite (DateTimeVariantTests, ConnectionSchemaTest, UdtDateTimeOffsetTest, ParametersTest, TvpQueryHintsTests): 357 passed. The 4 failures are time bulk-copy cases that fail identically on baseline main — pre-existing, in a class already tagged [Trait("Category", "flaky")].
  • Leaked user-defined types after a full run: 108 → 0.

Follow-up (not in this PR)

The shared CI databases already hold a backlog of orphaned types from before this fix. Those should be purged separately, ideally during a quiet window since DROP TYPE takes a Sch-M lock.

Checklist

  • Tests added or updated
  • Public API changes documented (n/a — test-only change)
  • Verified against customer repro (n/a — verified against CI failure signature and locally)
  • Ensure no breaking changes introduced

The `sqlclient_manual_azure_*` legs have been failing intermittently with
"Execution Timeout Expired" while executing the CREATE/DROP TYPE and
CREATE/DROP PROCEDURE statements in TvpQueryHintsTests. Two independent
issues combined to cause this:

1. `UserDefinedType.DropObject` guarded its DROP with `OBJECT_ID(...)`.
   User-defined types live in `sys.types`, not `sys.objects`, so
   `OBJECT_ID` always returns NULL for them and the DROP was silently
   skipped. Every test run therefore orphaned its table types in the
   shared test database. Locally, a single pass of the UDT-using manual
   tests leaked 108 types; in CI these accumulate indefinitely and make
   subsequent metadata operations progressively slower. Switched to
   `TYPE_ID(...)`, which is the correct lookup.

2. TvpQueryHintsTests created and dropped an identically shaped table
   type and stored procedure in every test constructor, issuing 20 DDL
   statements for 5 tests. Moved that setup into a class fixture so it
   happens once, and gave the fixture connection a longer default command
   timeout to absorb the schema-modification lock contention that remains
   when several platform legs run against the same Azure SQL database
   concurrently.

Verified against SQL Server 2022: all 5 TvpQueryHintsTests pass and the
UDT-using manual tests now leak zero types (previously 108).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings August 25, 2026 06:39
@cheenamalhotra
cheenamalhotra requested a review from a team as a code owner August 25, 2026 06:39
@github-project-automation github-project-automation Bot moved this to To triage in SqlClient Board Aug 25, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses intermittent ManualTests timeouts in the Azure SQL CI legs by reducing schema-modification (DDL) contention and ensuring transient user-defined table types are actually cleaned up after tests complete.

Changes:

  • Fix UserDefinedType cleanup to correctly detect existing types using TYPE_ID() (instead of OBJECT_ID()), preventing leaked table types.
  • Refactor TvpQueryHintsTests to use an IClassFixture so the table type and stored procedure are created/dropped once per class (instead of per test), and increase command timeout to tolerate shared-DB contention.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ParameterTest/TvpQueryHintsTests.cs Introduces a class fixture to reuse a single UDT + stored procedure across all tests, reducing DDL operations and timeouts.
src/Microsoft.Data.SqlClient/tests/Common/Fixtures/DatabaseObjects/UserDefinedType.cs Fixes drop logic to correctly locate user-defined types via TYPE_ID() so they are actually removed from the shared test database.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@cheenamalhotra cheenamalhotra added this to the 7.1.0-preview3 milestone Aug 25, 2026
@cheenamalhotra cheenamalhotra moved this from To triage to In review in SqlClient Board Aug 25, 2026
@cheenamalhotra cheenamalhotra added the Area\Tests Issues that are targeted to tests or test projects label Aug 25, 2026
Addresses review feedback on the TYPE_ID() guard, and applies the same fix to
the sibling fixtures that had the identical pattern.

Generated object names embed Environment.UserName and Environment.MachineName
(DatabaseObject.GenerateLongName), so interpolating them into a T-SQL string
literal breaks the batch if either contains an apostrophe, silently skipping
the drop and leaking the object. Pass the name as a parameter instead, matching
what ColumnEncryptionKey and ColumnMasterKey already do. The identifiers in the
DROP statements stay safe because GenerateLongName bracket-quotes them.

Covers UserDefinedType (TYPE_ID), Table and StoredProcedure (OBJECT_ID),
DatabaseUser (USER_ID) and ServerLogin (SUSER_ID).

Also makes TvpQueryHintsFixture exception-safe: a failure partway through the
constructor previously orphaned the table type and the connection, and a
transient error while dropping the procedure skipped the type and connection
cleanup entirely - the exact leak this change set exists to prevent.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 25, 2026 06:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

@cheenamalhotra

Copy link
Copy Markdown
Member Author

The sqlclient_manual_azure_123_linux_net8 leg failed on an infrastructure flake unrelated to this PR — TransactionEnlistmentTest.TestManualEnlistment_Enlist hit a TCP connection timeout reaching the server (provider: TCP Provider, error: 35 / The connection attempt timed out), not an assertion failure. That test is untouched by this PR and failed the same way on build 169380 for an unrelated PR. It was 1 failure out of 871 in that leg.

Worth noting the same build confirms the fix: 13 of the 14 Azure legs passed, and every sqlclient_manual_azure_123_* leg is now free of the TvpQueryHintsTests failures that were present on builds 169380 and 169440.

/azp run sqlclient-pr

The required sqlclient-pr pipeline is failing only on tests unrelated to
this change, in a pipeline with a ~48% baseline failure rate across other
PRs. Observed failures, none of which touch any file in this PR:

  MARSTest.MarsScenarioClientJoin              (also failed on PR #4567)
  SqlCommandCancelTest.TimeOutDuringRead_Tcp   (timing sensitive)
  TransactionEnlistmentTest.TestManualEnlistment_Enlist
                                               (also failed on PR #4585)

The /azp run comment trigger is not enabled on this repo, so refreshing
the head SHA is the only available way to re-run the required checks.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: cfc64bc6-a9e6-490d-88b1-4b78d25aa103
@cheenamalhotra
cheenamalhotra force-pushed the dev/cheena/bookish-guacamole branch from 23382c6 to a633e89 Compare August 25, 2026 08:46
@paulmedynski paulmedynski modified the milestones: 8.0.0-preview1, 7.1.0 Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area\Tests Issues that are targeted to tests or test projects

Projects

Status: In review

Development

Successfully merging this pull request may close these issues.

5 participants