Fixes #30466: ingest S3 bucket-level and root-level object tags

[akashverma0786]

Fixes #30466

What

The S3 connector was dropping two categories of S3 tags. This adds both.

• Root-level object tags — _get_bucket_name_and_key used len(parts) > 2, so a file at the bucket root (s3://bucket/file.csv, 2 segments) returned (None, None) and its object tags were skipped. Changed to >= 2.
• Bucket-level tags — added get_bucket_tagging for the bucket container (routed via a new _fetch_s3_tags helper: object tags for leaf files, bucket tags for the bucket). The bucket container was also missing its container_fqn, which is now set so its tags can attach.

Both confirmed against DataHub's S3 source (get_key_prefix, use_s3_bucket_tags).

Not changed

Nested-object tagging, folder containers (no tag leak), exception handling, and the includeTags=false default are all byte-identical to before.

Testing

• Unit: 293 passed (added root-level path test + bucket container_fqn assertions)
• Integration (real MinIO): 24 passed in tests/integration/s3/
• Live end-to-end against MinIO: bucket tags, root-object tags, and nested-object tags all land; includeTags: true required.

Greptile Summary

This PR extends S3 tag ingestion to bucket containers and root-level objects.

• Routes leaf containers to S3 object-tag retrieval and bucket-root containers to bucket-tag retrieval.
• Assigns the bucket container FQN required to associate fetched tags with the correct OpenMetadata container.
• Adds unit coverage for S3 path parsing, tag routing, FQN generation, empty tag sets, and tagging API failures.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failures remain.

Important Files Changed

Filename	Overview
ingestion/src/metadata/ingestion/source/storage/s3/metadata.py	Adds root-object path parsing, bucket/object tag routing, and bucket-container FQN assignment without an eligible blocking failure.
ingestion/tests/unit/topology/storage/test_s3_methods.py	Adds focused coverage for path parsing, tag routing, FQN generation, and exception behavior.
ingestion/tests/unit/topology/storage/test_s3_storage.py	Updates the unstructured bucket-container expectation to include its generated FQN.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[S3 container details] --> B{Container has FQN?}
  B -- No --> Z[Skip tag retrieval]
  B -- Yes --> C{Leaf with bucket and object key?}
  C -- Yes --> D[get_object_tagging]
  C -- No --> E{Bucket-root path?}
  E -- Yes --> F[get_bucket_tagging]
  E -- No --> Z
  D --> G[Convert S3 tags to OpenMetadata tags]
  F --> G

Loading

_{Reviews (3): Last reviewed commit: "fix(s3): correct type annotations for op..." | Re-trigger Greptile}

Context used:

• Knowledge Base — Ingestion Framework

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• Linked issue S3 connector improvement: ingest bucket-level tags and root-level object tags #30466 is not added to the Shipping project.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[gitar-bot]

Code Review ✅ Approved 1 resolved / 1 findings

Adds S3 bucket-level and root-level object tag ingestion with associated routing and FQN fixes, addressing the untested new bucket-tag branch finding. No issues found.

✅ 1 resolved

✅ Quality: New bucket-tag branch of _fetch_s3_tags is untested

📄 ingestion/src/metadata/ingestion/source/storage/s3/metadata.py:242-254
The added tests cover the root-level object path (_get_bucket_name_and_key) and the bucket container_fqn, but nothing exercises _fetch_s3_tags or its new get_bucket_tagging else-branch. The bucket-level tag ingestion — the core behavior this PR adds — has no unit coverage, so a regression (e.g. wrong path routing or a broken get_bucket_tagging call) would pass CI. Add a unit test that mocks s3_client.get_bucket_tagging and asserts bucket-container tags are returned, plus a case confirming a folder container yields no tags.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}

[github-actions]

🔴 Playwright Results — workflow failed

Validated commit 4e992ce567c4418b4c6f9141b50749c9e1d1e8cf in Playwright run 30090541573, attempt 1.

✅ 104 passed · ❌ 2 failed · 🟡 1 flaky · ⏭️ 3 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ❌ unmet · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 50m 30s

⏱️ Max setup 3m 3s · max shard execution 14m 36s · max shard-job elapsed before upload 21m 2s · reporting 7s

🌐 204.66 requests/attempt · 1.79 app boots/UI scenario · 0.00% common-shard skew

Optimization targets still in progress:

• Browser traffic was 204.66 requests per attempt (convergence target: fewer than 200).
• Application boot ratio was 1.79 per UI scenario (227 boots / 127 scenarios; convergence target: at most 1).

Shard	Passed	Failed	Flaky	Skipped	Lifecycle failed	Lifecycle flaky
✅ Shard chromium-01	46	0	0	3	0	0
✅ Shard ingestion-01	27	0	0	0	0	0
🔴 Shard ingestion-02	31	2	1	0	0	0

Genuine Failures (failed on all attempts)

❌ Features/TestSuiteMultiPipeline.spec.ts › TestSuite multi pipeline support (shard ingestion-02)

Error: locator.click: Error: strict mode violation: getByText('Data Observability') resolved to 2 elements:     1) <div class="d-flex justify-between gap-1">Data Observability</div> aka getByTestId('profiler').getByText('Data Observability')     2) <span class="tw:truncate tw:text-xs tw:font-medium tw:text-primary">Data Observability</span> aka getByTestId('asset-health-row-dataObservability').getByText('Data Observability')  Call log: �[2m  - waiting for getByText('Data Observability')�[22m 

❌ Features/TestSuiteMultiPipeline.spec.ts › Edit the pipeline's test case (shard ingestion-02)

Error: locator.click: Error: strict mode violation: getByText('Data Observability') resolved to 2 elements:     1) <div class="d-flex justify-between gap-1">Data Observability</div> aka getByTestId('profiler').getByText('Data Observability')     2) <span class="tw:truncate tw:text-xs tw:font-medium tw:text-primary">Data Observability</span> aka getByTestId('asset-health-row-dataObservability').getByText('Data Observability')  Call log: �[2m  - waiting for getByText('Data Observability')�[22m 

🟡 1 flaky test(s) (passed on retry)

• Features/IncidentManager.spec.ts › Resolving incident & re-run pipeline (shard ingestion-02, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud