fix(bulk-ops): exclude soft-deleted entities from column grid (#28653) [1.13]

[sonika-shah]

Cherry-pick of #28653 into 1.13.

The original fix merged to main and was picked into the 1.12.x line, but 1.12.11 was cut after 1.13.0 branched out — so 1.13 never received it. Confirmed the deleted:false filter was absent from both the ES and OS ColumnAggregator on 1.13.

Fixes #28437 for the 1.13 line.

Cherry-picked commit: 1079291 (clean, no conflicts).

Changes

• ElasticSearchColumnAggregator.java — deleted:false filter in buildFilters() and buildTagFilterQuery()
• OpenSearchColumnAggregator.java — same
• ColumnBulkUpdateIT.java — regression test test_columnGrid_excludesSoftDeletedTables

Greptile Summary

Adds consistent soft-deletion filtering to the column grid.

• Applies deleted:false to both standard and tag-filtered Elasticsearch column queries.
• Applies the equivalent filters to OpenSearch.
• Adds an integration test covering column visibility before and after a table is soft-deleted.

Confidence Score: 5/5

The PR appears safe to merge, with matching Elasticsearch and OpenSearch behavior and regression coverage for the reported failure.

Both aggregation paths now consistently require active parent documents, and the integration test exercises the observable column-grid result across the asynchronous soft-delete index update.

Important Files Changed

Filename	Overview
openmetadata-service/src/main/java/org/openmetadata/service/search/elasticsearch/ElasticSearchColumnAggregator.java	Excludes soft-deleted parent entities from both normal and tag-filtered Elasticsearch column aggregation paths.
openmetadata-service/src/main/java/org/openmetadata/service/search/opensearch/OpenSearchColumnAggregator.java	Mirrors the Elasticsearch soft-deletion filters across both OpenSearch query paths.
openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/ColumnBulkUpdateIT.java	Adds an isolated, condition-waiting regression test that verifies a soft-deleted table's column disappears from the grid.

_{Reviews (1): Last reviewed commit: "fix(bulk-ops): exclude soft-deleted enti..." | Re-trigger Greptile}

Context used:

• Knowledge Base — Search Indexing and Event/Notification Sync

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• No GitHub issue is linked. Link an issue in the Development section of the PR (or add Fixes #12345 to the description). For a same-org cross-repo issue, add Fixes open-metadata/<repo>#123 to the description.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[gitar-bot]

Code Review ✅ Approved

Cherry-pick of the fix excluding soft-deleted entities from the Elasticsearch and OpenSearch column aggregators, complete with a regression test. No issues found.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}

[github-actions]

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[github-actions]

🔴 Playwright Results — 6 test failure(s), 2 pipeline/setup failure(s)

✅ 3965 passed · ❌ 6 failed · 🟡 14 flaky · ⏭️ 30 skipped

Pipeline and setup failures

• Shard 2 reported zero tests.
• Shard 2 test execution finished with status failure without a reported test failure.

Shard	Passed	Failed	Flaky	Skipped
✅ Shard 1	291	0	0	4
✅ Shard 2	0	0	0	0
🟡 Shard 3	735	0	2	7
🟡 Shard 4	782	0	1	2
🟡 Shard 5	752	0	5	9
✅ Shard 6	693	0	0	0
🔴 Shard 7	712	6	6	8

Genuine Failures (failed on all attempts)

❌ Pages/GlossaryTermRelationSettings.spec.ts › creates a custom relation type via the drawer (shard 7)

TypeError: (0 , _admin.authenticateAdminPage) is not a function

❌ Pages/GlossaryTermRelationSettings.spec.ts › edits a custom relation type and keeps the name immutable (shard 7)

TypeError: (0 , _admin.authenticateAdminPage) is not a function

❌ Pages/GlossaryTermRelationSettings.spec.ts › rejects duplicate relation-type names with an inline error (shard 7)

TypeError: (0 , _admin.authenticateAdminPage) is not a function

❌ Pages/GlossaryTermRelationSettings.spec.ts › deletes a custom relation type (shard 7)

TypeError: (0 , _admin.authenticateAdminPage) is not a function

❌ Pages/GlossaryTermRelationSettings.spec.ts › locks system-defined relation types from edit and delete (shard 7)

TypeError: (0 , _admin.authenticateAdminPage) is not a function

❌ Pages/GlossaryTermRelationSettings.spec.ts › paginates relation types when they exceed a page (shard 7)

TypeError: (0 , _admin.authenticateAdminPage) is not a function

🟡 14 flaky test(s) (passed on retry)

• Features/BulkEditEntity.spec.ts › Database Schema (shard 3, 1 retry)
• Features/ColumnBulkOperations.spec.ts › should show no results when searching for nonexistent column (shard 3, 1 retry)
• Features/Permissions/GlossaryPermissions.spec.ts › Team-based permissions work correctly (shard 4, 1 retry)
• Pages/CustomProperties.spec.ts › Enum (shard 5, 1 retry)
• Pages/CustomProperties.spec.ts › Entity Reference (shard 5, 1 retry)
• Pages/DataContracts.spec.ts › Create Data Contract and validate for MlModel (shard 5, 1 retry)
• Pages/DataContracts.spec.ts › Contract Status badge should be visible on condition if Contract Tab is present/hidden by Persona (shard 5, 1 retry)
• Pages/DataContractsSemanticRules.spec.ts › Validate Description Rule Is_Set (shard 5, 1 retry)
• Pages/Glossary.spec.ts › Glossary Term Update in Glossary Page should persist tree (shard 7, 1 retry)
• Pages/InputOutputPorts.spec.ts › Output port drawer quick filter - behaviour matrix (shard 7, 2 retries)
• Pages/Lineage/LineageFilters.spec.ts › Verify lineage schema filter selection (shard 7, 1 retry)
• Pages/TestSuite.spec.ts › Logical TestSuite (shard 7, 1 retry)
• Pages/UserDetails.spec.ts › Create team with domain and verify visibility of inherited domain in user profile after team removal (shard 7, 1 retry)
• Pages/Users.spec.ts › Permissions for table details page for Data Consumer (shard 7, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace