fix(dbt): handle nanosecond-precision timestamps from dbt-fusion in test results by zak-nuccio · Pull Request #30481 · open-metadata/OpenMetadata

zak-nuccio · 2026-07-24T17:22:13Z

dbt-fusion outputs 9-digit nanosecond timestamps (e.g. 2026-07-22T09:27:12.979492347Z) which do not match Python %f (6-digit microseconds). Two sites were affected:

add_dbt_test_result — uncaught ValueError crashed test case ingestion for any dbt-fusion run result.
_execute_completed_at — silently returned None instead of parsing the timestamp, causing _get_latest_result to fall back to the wrong result.

Both now strip the extra nanosecond digits to 6-digit precision on ValueError, preserving the correct timestamp.

Greptile Summary

This PR adds dbt-fusion nanosecond timestamp normalization.

Truncates nine-digit fractional seconds to Python-supported microsecond precision when selecting the latest dbt result and persisting test results.
Adds unit coverage for nanosecond timestamp inputs.

Confidence Score: 4/5

The timestamp fallback should be fixed before merging because an unsupported timestamp can now discard an otherwise usable dbt test node.

The retry correctly handles dbt-fusion nanoseconds, but its unguarded second parse regresses the established invalid-timestamp fallback, while the new latest-result test bypasses the parsing branch.

Files Needing Attention: ingestion/src/metadata/ingestion/source/database/dbt/metadata.py, ingestion/tests/unit/test_dbt.py

Important Files Changed

Filename	Overview
ingestion/src/metadata/ingestion/source/database/dbt/metadata.py	Adds nanosecond normalization, but the latest-result parser now raises for unsupported timestamps instead of ignoring them.
ingestion/tests/unit/test_dbt.py	Adds nanosecond cases, although the latest-result case uses one match and therefore bypasses the parsing and ordering branch.

_{Reviews (1): Last reviewed commit: "fix(dbt): handle nanosecond-precision ti..." | Re-trigger Greptile}

Greptile also left 2 inline comments on this PR.

…est results dbt-fusion outputs 9-digit nanosecond timestamps (e.g. 2026-07-22T09:27:12.979492347Z) which don't match Python's %f (6-digit microseconds). Two sites affected: - add_dbt_test_result(strptime line): uncaught ValueError crashed test case ingestion for any dbt-fusion run result. - _execute_completed_at: silently returned None instead of parsing the timestamp, causing _get_latest_result to fall back to the wrong result. Both now strip the extra nanosecond digits to 6-digit precision on ValueError, preserving the correct timestamp. Adds test cases for both code paths.

github-actions · 2026-07-24T17:22:21Z

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

Linked issue fix(dbt): nanosecond-precision timestamps from dbt-fusion crash test result ingestion #30480 is not added to the Shipping project.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

github-actions · 2026-07-24T17:22:33Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

gitar-bot · 2026-07-24T17:23:50Z

                        try:
                            return datetime.strptime(completed, DBT_RUN_RESULT_DATE_FORMAT)
                        except ValueError:
-                            return None
+                            dot = completed.rfind(".")
+                            if dot != -1:
+                                completed = completed[: dot + 7] + "Z"
+                            return datetime.strptime(completed, DBT_RUN_RESULT_DATE_FORMAT)


⚠️ Bug: Fallback reparse in _execute_completed_at can crash ingestion

Previously the except ValueError in _execute_completed_at returned None, so any unparseable completed_at degraded gracefully and _get_latest_result fell back to matches[0]. Now the fallback path calls datetime.strptime(...) again with no guard: for any timestamp that fails to parse for a reason other than nanosecond precision (e.g. no fractional part, a +00:00 offset instead of Z, or a space instead of T), or where the reparse still fails, this second call raises an uncaught ValueError that propagates and crashes _get_latest_result. Wrap the fallback strptime in try/except and return None on failure to preserve the prior graceful behavior.

Return None if the truncated timestamp still fails to parse.:

try: return datetime.strptime(completed, DBT_RUN_RESULT_DATE_FORMAT) except ValueError: dot = completed.rfind(".") if dot != -1: completed = completed[: dot + 7] + "Z" try: return datetime.strptime(completed, DBT_RUN_RESULT_DATE_FORMAT) except ValueError: return None

Apply fix

_{Check the box to apply the fix or reply for a change | Was this helpful? React with 👍 / 👎}

gitar-bot · 2026-07-24T17:23:51Z

+                            dot = completed.rfind(".")
+                            if dot != -1:
+                                completed = completed[: dot + 7] + "Z"
+                            return datetime.strptime(completed, DBT_RUN_RESULT_DATE_FORMAT)


💡 Edge Case: Nanosecond fix assumes trailing 'Z' and drops timezone offsets

Both fallbacks slice to 6 fractional digits and unconditionally append a literal "Z" (completed[: dot + 7] + "Z"). This is correct for dbt-fusion's ...347Z form, but a timestamp with an explicit offset such as 2026-07-22T09:27:12.979492347+00:00 would be rewritten to ...492Z, silently discarding the offset and misrepresenting the instant. It also relies on there being at least 6 digits after the dot; fewer-than-6-digit fractions never reach this branch since %f accepts 1-6 digits, so the slice assumption holds only for the >6-digit case. Consider matching only the fractional-digit run rather than blindly appending Z.

_{Was this helpful? React with 👍 / 👎}

gitar-bot · 2026-07-24T17:23:51Z

Code Review ⚠️ Changes requested 0 resolved / 2 findings

Adds nanosecond-precision timestamp handling for dbt-fusion test results, but the fallback reparse in _execute_completed_at can crash ingestion and unconditionally strips timezone offsets.

⚠️

Bug: Fallback reparse in _execute_completed_at can crash ingestion

📄 ingestion/src/metadata/ingestion/source/database/dbt/metadata.py:637-643

Previously the except ValueError in _execute_completed_at returned None, so any unparseable completed_at degraded gracefully and _get_latest_result fell back to matches[0]. Now the fallback path calls datetime.strptime(...) again with no guard: for any timestamp that fails to parse for a reason other than nanosecond precision (e.g. no fractional part, a +00:00 offset instead of Z, or a space instead of T), or where the reparse still fails, this second call raises an uncaught ValueError that propagates and crashes _get_latest_result. Wrap the fallback strptime in try/except and return None on failure to preserve the prior graceful behavior.

Return None if the truncated timestamp still fails to parse.

try:
    return datetime.strptime(completed, DBT_RUN_RESULT_DATE_FORMAT)
except ValueError:
    dot = completed.rfind(".")
    if dot != -1:
        completed = completed[: dot + 7] + "Z"
    try:
        return datetime.strptime(completed, DBT_RUN_RESULT_DATE_FORMAT)
    except ValueError:
        return None

💡 Edge Case: Nanosecond fix assumes trailing 'Z' and drops timezone offsets

📄 ingestion/src/metadata/ingestion/source/database/dbt/metadata.py:640-643 📄 ingestion/src/metadata/ingestion/source/database/dbt/metadata.py:1971-1974

Both fallbacks slice to 6 fractional digits and unconditionally append a literal "Z" (completed[: dot + 7] + "Z"). This is correct for dbt-fusion's ...347Z form, but a timestamp with an explicit offset such as 2026-07-22T09:27:12.979492347+00:00 would be rewritten to ...492Z, silently discarding the offset and misrepresenting the instant. It also relies on there being at least 6 digits after the dot; fewer-than-6-digit fractions never reach this branch since %f accepts 1-6 digits, so the slice assumption holds only for the >6-digit case. Consider matching only the fractional-digit run rather than blindly appending Z.

🤖 Prompt for agents

Code Review: Adds nanosecond-precision timestamp handling for dbt-fusion test results, but the fallback reparse in _execute_completed_at can crash ingestion and unconditionally strips timezone offsets.

1. ⚠️ Bug: Fallback reparse in _execute_completed_at can crash ingestion
   Files: ingestion/src/metadata/ingestion/source/database/dbt/metadata.py:637-643

   Previously the `except ValueError` in `_execute_completed_at` returned `None`, so any unparseable `completed_at` degraded gracefully and `_get_latest_result` fell back to `matches[0]`. Now the fallback path calls `datetime.strptime(...)` again with no guard: for any timestamp that fails to parse for a reason other than nanosecond precision (e.g. no fractional part, a `+00:00` offset instead of `Z`, or a space instead of `T`), or where the reparse still fails, this second call raises an uncaught `ValueError` that propagates and crashes `_get_latest_result`. Wrap the fallback strptime in try/except and return `None` on failure to preserve the prior graceful behavior.

   Fix (Return None if the truncated timestamp still fails to parse.):
   try:
       return datetime.strptime(completed, DBT_RUN_RESULT_DATE_FORMAT)
   except ValueError:
       dot = completed.rfind(".")
       if dot != -1:
           completed = completed[: dot + 7] + "Z"
       try:
           return datetime.strptime(completed, DBT_RUN_RESULT_DATE_FORMAT)
       except ValueError:
           return None

2. 💡 Edge Case: Nanosecond fix assumes trailing 'Z' and drops timezone offsets
   Files: ingestion/src/metadata/ingestion/source/database/dbt/metadata.py:640-643, ingestion/src/metadata/ingestion/source/database/dbt/metadata.py:1971-1974

   Both fallbacks slice to 6 fractional digits and unconditionally append a literal `"Z"` (`completed[: dot + 7] + "Z"`). This is correct for dbt-fusion's `...347Z` form, but a timestamp with an explicit offset such as `2026-07-22T09:27:12.979492347+00:00` would be rewritten to `...492Z`, silently discarding the offset and misrepresenting the instant. It also relies on there being at least 6 digits after the dot; fewer-than-6-digit fractions never reach this branch since `%f` accepts 1-6 digits, so the slice assumption holds only for the >6-digit case. Consider matching only the fractional-digit run rather than blindly appending `Z`.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

`Compact`
`gitar display:verbose`

_{Was this helpful? React with 👍 / 👎 | Gitar}

Copilot

Pull request overview

This PR updates the dbt ingestion logic to correctly parse dbt-fusion completed_at timestamps that include nanosecond precision (9 fractional digits), avoiding crashes and incorrect “latest result” selection when multiple run_results files contain the same unique_id.

Changes:

Add nanosecond→microsecond truncation fallback when parsing dbt run result timestamps.
Fix _get_latest_result selection by ensuring execute.completed_at timestamps can be parsed consistently.
Add unit tests covering nanosecond timestamp handling in both _get_latest_result and add_dbt_test_result.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
ingestion/src/metadata/ingestion/source/database/dbt/metadata.py	Adds a fallback parse path to truncate nanosecond timestamps to microsecond precision in two timestamp parsing sites.
ingestion/tests/unit/test_dbt.py	Introduces unit tests to validate nanosecond timestamp parsing behavior for latest-result selection and test result ingestion.

Comments suppressed due to low confidence (1)

ingestion/tests/unit/test_dbt.py:3585

test_nanosecond_timestamp_is_parsed only asserts that add_test_case_results was called, but it doesn’t verify that the nanosecond timestamp was actually parsed/truncated and propagated into the emitted TestCaseResult.timestamp. Adding an assertion on the computed timestamp makes the test validate the intended behavior (and not just “no crash”).

        source.metadata.add_test_case_results.assert_called_once()

                        except ValueError:
-                            return None
+                            dot = completed.rfind(".")
+                            if dot != -1:
+                                completed = completed[: dot + 7] + "Z"
+                            return datetime.strptime(completed, DBT_RUN_RESULT_DATE_FORMAT)


+    def test_nanosecond_timestamp_parsed(self):
+        from metadata.ingestion.source.database.dbt.metadata import DbtSource
+
+        nano_result = self._make_result("test.pkg.my_test", "2026-07-22T09:27:12.979492347Z", "pass")
+        dbt_objects = self._make_dbt_objects([[nano_result]])
+
+        got = DbtSource._get_latest_result(dbt_objects, "test.pkg.my_test")
+        self.assertIs(got, nano_result)


greptile-apps · 2026-07-24T17:26:28Z

+                            dot = completed.rfind(".")
+                            if dot != -1:
+                                completed = completed[: dot + 7] + "Z"
+                            return datetime.strptime(completed, DBT_RUN_RESULT_DATE_FORMAT)


Invalid timestamps now skip tests

When multiple matching results include an unsupported timestamp, the unconditional second strptime raises instead of returning None, causing the entire test node to be skipped even when another matching result has a valid timestamp.

Knowledge Base Used: Ingestion Framework

greptile-apps · 2026-07-24T17:26:29Z

+        nano_result = self._make_result("test.pkg.my_test", "2026-07-22T09:27:12.979492347Z", "pass")
+        dbt_objects = self._make_dbt_objects([[nano_result]])


Nanosecond parser remains untested

This fixture contains only one matching result, so _get_latest_result returns it before parsing the timestamp. Add at least two matches with nanosecond timestamps so the new normalization and latest-result ordering are actually exercised.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

zak-nuccio requested a review from a team as a code owner July 24, 2026 17:22

Copilot AI review requested due to automatic review settings July 24, 2026 17:22

Copilot started reviewing on behalf of zak-nuccio July 24, 2026 17:22 View session

gitar-bot Bot reviewed Jul 24, 2026

View reviewed changes

Copilot AI reviewed Jul 24, 2026

View reviewed changes

greptile-apps Bot reviewed Jul 24, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(dbt): handle nanosecond-precision timestamps from dbt-fusion in test results#30481

fix(dbt): handle nanosecond-precision timestamps from dbt-fusion in test results#30481
zak-nuccio wants to merge 1 commit into
open-metadata:mainfrom
zak-nuccio:fix/dbt-nanosecond-timestamps

zak-nuccio commented Jul 24, 2026 •

edited by greptile-apps Bot

Loading

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

gitar-bot Bot Jul 24, 2026

Uh oh!

gitar-bot Bot Jul 24, 2026

Uh oh!

gitar-bot Bot commented Jul 24, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

greptile-apps Bot Jul 24, 2026

Uh oh!

greptile-apps Bot Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		nano_result = self._make_result("test.pkg.my_test", "2026-07-22T09:27:12.979492347Z", "pass")
		dbt_objects = self._make_dbt_objects([[nano_result]])

Uh oh!

Conversation

zak-nuccio commented Jul 24, 2026 • edited by greptile-apps Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Uh oh!

github-actions Bot commented Jul 24, 2026

❌ PR checklist incomplete

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

gitar-bot Bot Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

gitar-bot Bot Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

gitar-bot Bot commented Jul 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

greptile-apps Bot Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zak-nuccio commented Jul 24, 2026 •

edited by greptile-apps Bot

Loading