Skip to content

fix(pandas): Support pandas v3 timestamp semantics while preserving v2 behavior (DM-3534) - #2780

Open
haakonvt wants to merge 12 commits into
masterfrom
fix-pandas-v3-part2
Open

fix(pandas): Support pandas v3 timestamp semantics while preserving v2 behavior (DM-3534)#2780
haakonvt wants to merge 12 commits into
masterfrom
fix-pandas-v3-part2

Conversation

@haakonvt

Copy link
Copy Markdown
Contributor

Enable pandas v3 compatibility by making SDK timestamp resolution version-conditional ("ns" on v2, "ms" on v3), remove the pandas upper bound, update migration/docs and tests, and cache version/unit helper lookups.

@haakonvt
haakonvt requested review from a team as code owners August 21, 2026 13:02

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the SDK to support pandas v3 by dynamically adjusting timestamp precision (nanoseconds for pandas v2 and lower, and milliseconds for pandas v3+). Feedback on the changes highlights a bug in _pandas_helpers.py where np.ndarray timestamps are parsed without specifying unit="ms", and suggests using the PANDAS_TS_UNIT constant in integration tests to avoid hardcoding the timestamp unit.

Comment thread cognite/client/utils/_pandas_helpers.py
Comment thread tests/tests_integration/test_api/test_datapoints.py Outdated
Comment thread pyproject.toml
] # poetry 2.4.1 does not correctly limit numpy 2.2 to only python 3.10
sympy = ["sympy"]
pandas = ["pandas (>=2.1, <3)"]
pandas = ["pandas >=2.1"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't really reviewed anything yet, but shouldn't we have a strict upper bound still? Like, >= 2.1, <4

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, but it sure is up for discussion - and has been in the past.

Both numpy and pandas are very mature and responsible packages that we used to have open upper bounds for. It was only when v3 of pandas came out with what I deemed "a potentially breaking change for us" that we temporarily forced major version == 2 until it was addressed by us (aka as of this PR).

I didn't find the original discussion, but I found this; best summarised as "lock out future major releases just in case":
https://cognitedata.slack.com/archives/CG10VQPFX/p1773826110266419?thread_ts=1773760929.634279&cid=CG10VQPFX

@haakonvt haakonvt changed the title fix(pandas): Support pandas v3 timestamp semantics while preserving v2 behavior fix(pandas): Support pandas v3 timestamp semantics while preserving v2 behavior (DM-3534) Aug 21, 2026
@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.15254% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 93.04%. Comparing base (a72fee9) to head (c88a5cf).

Files with missing lines Patch % Lines
tests/tests_unit/test_utils/test_pandas_helpers.py 98.59% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##           master    #2780   +/-   ##
=======================================
  Coverage   93.04%   93.04%           
=======================================
  Files         515      516    +1     
  Lines       53006    53097   +91     
=======================================
+ Hits        49317    49404   +87     
- Misses       3689     3693    +4     
Files with missing lines Coverage Δ
cognite/client/_api/datapoints.py 97.66% <100.00%> (+<0.01%) ⬆️
cognite/client/_api/raw/rows.py 91.07% <100.00%> (+0.05%) ⬆️
cognite/client/_sync_api/datapoints.py 100.00% <100.00%> (ø)
cognite/client/_sync_api/raw/rows.py 100.00% <ø> (ø)
cognite/client/data_classes/datapoints.py 90.13% <100.00%> (ø)
cognite/client/utils/_pandas_helpers.py 92.18% <100.00%> (+1.12%) ⬆️
...tegration/test_api/test_datapoint_subscriptions.py 98.45% <100.00%> (ø)
...ests/tests_integration/test_api/test_datapoints.py 99.31% <100.00%> (+<0.01%) ⬆️
tests/tests_unit/test_api/test_datapoints.py 99.42% <100.00%> (ø)
tests/tests_unit/test_api/test_raw.py 100.00% <ø> (ø)
... and 7 more

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@haakonvt

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the SDK to support Pandas v3 by dynamically adjusting the precision of generated timestamp columns and indices (nanoseconds for Pandas v2 and milliseconds for Pandas v3) and updates dependency constraints to allow pandas >=2.1. Feedback on the changes suggests broadening the type hint of to_pandas_timestamp to int | float | None to prevent type-checking errors with optional timestamps, and adding a pytest.importorskip guard to the new test file to avoid failures during core-only test runs.

Comment thread cognite/client/utils/_pandas_helpers.py
Comment thread tests/tests_unit/test_utils/test_pandas_helpers.py
@haakonvt
haakonvt force-pushed the fix-pandas-v3-part2 branch from aa7934d to c88a5cf Compare August 21, 2026 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants