Fixes #26225: Add index and FORCE INDEX for listLastTestCaseResultsForTestSuite (MySQL)

[ndtrang]

Fixes ISSUE-26225

PR Description

Short blurb

What changes did you make?

• Added index idx_entity_timestamp_desc on data_quality_data_time_series (entityFQNHash, timestamp DESC) in the 1.12.2 MySQL and Postgres migrations.
• Replaced the single @SqlQuery for listLastTestCaseResultsForTestSuite in CollectionDAO with two @ConnectionAwareSqlQuery variants: MySQL uses FORCE INDEX (idx_entity_timestamp_desc) on the anti-join; Postgres keeps the same SQL without a hint.

Why did you make them?

• The "latest row per entity" query used by listLastTestCaseResultsForTestSuite was very slow (~20–42s) on MySQL because the optimizer did not pick an efficient plan for the anti-join. That query runs on every test suite pipeline completion and caused slow responses and Hikari connection warnings. Adding the index and using FORCE INDEX on MySQL brought the query down to ~0.5s.

How did you test your changes?

• Ran the 1.12.2 migration on a dev MySQL instance, then exercised the flow that calls this query (e.g. test suite pipeline completion or calling the DAO method). Confirmed query time dropped from tens of seconds to under a second with the new index and FORCE INDEX. Postgres was verified to still use the same SQL (no hint). Existing tests pass.

---

I worked on adding an index and MySQL FORCE INDEX for listLastTestCaseResultsForTestSuite because the query was very slow (~20–42s) on MySQL and caused slow test suite pipeline completion and Hikari connection warnings. With the index and hint, the query runs in ~0.5s.

Type of change:

• Bug fix
• Improvement
• New feature
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation

Checklist:

• I have read the CONTRIBUTING document.
• My PR title is Fixes <issue-number>: <short explanation>
• I have commented on my code, particularly in hard-to-understand areas.
• For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

• I have added tests around the new logic.
• For connector/ingestion changes: I updated the documentation.

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[gitar-bot]

🔍 CI failure analysis for 16ce08a: All CI failures are unrelated to this PR's SQL migration and DAO changes. They are pre-existing infrastructure, flakiness, or external repository issues.

Issue

None of the CI failures are related to this PR. The PR only modifies:

• bootstrap/sql/migrations/native/1.12.2/mysql/schemaChanges.sql
• bootstrap/sql/migrations/native/1.12.2/postgres/schemaChanges.sql
• openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/CollectionDAO.java

Root Cause

All failures are pre-existing infrastructure issues, test environment flakiness, failures in external repositories, or unrelated connector/E2E regressions.

Details

1. integration-tests-mysql-elasticsearch / Test Report — DJL/NDManager infrastructure failure

Caused by: java.lang.IllegalStateException: NDManager has been closed already.
    at ai.djl.ndarray.BaseNDManager.attachUncappedInternal(BaseNDManager.java:327)
    at org.openmetadata.service.search.vector.client.DjlEmbeddingClient.embedBatch

Test environment teardown ordering issue with DJL vector embedding — unrelated to this PR. The Test Report job is the reporting step for this same failure.

2. maven-sonarcloud-ci — SonarCloud configuration error

[ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.10.0.2594:sonar
  on project openmetadata-java-client: Project not found. Please check the 'sonar.projectKey'
  and 'sonar.organization' properties, the 'SONAR_TOKEN' environment variable...

This is a CI environment/token configuration issue unrelated to this PR's SQL migration and DAO changes.

3. maven-collate-ci — External Collate workflow failure

Triggers a downstream workflow in open-metadata/openmetadata-collate and fails with conclusion=failure after ~2 minutes. Failure in an external repository, not caused by this PR.

4. playwright-ci-postgresql — E2E test flakiness

• 1 permanent failure: Lineage.spec.ts:466 — locator timeout for lineage node
• 7 flaky: ExplorePageRightPanel.spec.ts, Glossary.spec.ts, Tag.spec.ts, Users.spec.ts, EntityVersionPages.spec.ts (browser timeout/close or undefined access errors)

None involve data_quality_data_time_series, the new index, or listLastTestCaseResultsForTestSuite.

5. py-run-tests (3.11) — Oracle connector unit test regression

Two Oracle tests fail in ingestion/tests/unit/topology/database/test_oracle.py due to incorrect FQDN construction. Pre-existing regression unrelated to this PR.

Code Review 👍 Approved with suggestions 0 resolved / 2 findings

Adds FORCE INDEX directives to the listLastTestCaseResultsForTestSuite query for MySQL performance optimization. Consider also applying FORCE INDEX to the related listLastTestCaseResult query and adding blank lines between migration statements for consistency.

💡 Performance: listLastTestCaseResult also missing FORCE INDEX for MySQL

📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/CollectionDAO.java:7966 📄 bootstrap/sql/migrations/native/1.12.2/mysql/schemaChanges.sql:14

The migration comment (line 14 of mysql/schemaChanges.sql) explicitly mentions both listLastTestCaseResultsForTestSuite and listLastTestCaseResult as beneficiaries of the new index. However, listLastTestCaseResult (the single-entity variant) still uses a plain @SqlQuery with the same anti-join pattern on data_quality_data_time_series but without FORCE INDEX on MySQL.

While this query filters on a single entityFQNHash (so it's less likely to be slow), the same index would still help MySQL choose the optimal plan. For consistency with the stated intent and to prevent the optimizer from choosing a suboptimal plan on large tables, consider converting this to @ConnectionAwareSqlQuery as well, with FORCE INDEX (idx_entity_timestamp_desc) on the MySQL variant.

💡 Quality: Missing blank line between migration statements

📄 bootstrap/sql/migrations/native/1.12.2/mysql/schemaChanges.sql:16 📄 bootstrap/sql/migrations/native/1.12.2/postgres/schemaChanges.sql:20

In both mysql/schemaChanges.sql (after line 16) and postgres/schemaChanges.sql (after line 20), the new ADD KEY / CREATE INDEX statement runs directly into the next comment block with no blank separator line. The rest of the migration file uses blank lines between logical blocks for readability. Adding a blank line would keep the file consistent.

🤖 Prompt for agents

Code Review: Adds FORCE INDEX directives to the listLastTestCaseResultsForTestSuite query for MySQL performance optimization. Consider also applying FORCE INDEX to the related listLastTestCaseResult query and adding blank lines between migration statements for consistency.

1. 💡 Performance: `listLastTestCaseResult` also missing FORCE INDEX for MySQL
   Files: openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/CollectionDAO.java:7966, bootstrap/sql/migrations/native/1.12.2/mysql/schemaChanges.sql:14

   The migration comment (line 14 of `mysql/schemaChanges.sql`) explicitly mentions both `listLastTestCaseResultsForTestSuite` and `listLastTestCaseResult` as beneficiaries of the new index. However, `listLastTestCaseResult` (the single-entity variant) still uses a plain `@SqlQuery` with the same anti-join pattern on `data_quality_data_time_series` but without `FORCE INDEX` on MySQL.
   
   While this query filters on a single `entityFQNHash` (so it's less likely to be slow), the same index would still help MySQL choose the optimal plan. For consistency with the stated intent and to prevent the optimizer from choosing a suboptimal plan on large tables, consider converting this to `@ConnectionAwareSqlQuery` as well, with `FORCE INDEX (idx_entity_timestamp_desc)` on the MySQL variant.

2. 💡 Quality: Missing blank line between migration statements
   Files: bootstrap/sql/migrations/native/1.12.2/mysql/schemaChanges.sql:16, bootstrap/sql/migrations/native/1.12.2/postgres/schemaChanges.sql:20

   In both `mysql/schemaChanges.sql` (after line 16) and `postgres/schemaChanges.sql` (after line 20), the new `ADD KEY` / `CREATE INDEX` statement runs directly into the next comment block with no blank separator line. The rest of the migration file uses blank lines between logical blocks for readability. Adding a blank line would keep the file consistent.

Tip

Comment Gitar fix CI or enable auto-apply: gitar auto-apply:on

Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing less information.

Comment with these commands to change:

Auto-apply	Compact
gitar auto-apply:on	gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud