fix(pandas): Support pandas v3 timestamp semantics while preserving v2 behavior (DM-3534) - #2780
fix(pandas): Support pandas v3 timestamp semantics while preserving v2 behavior (DM-3534)#2780haakonvt wants to merge 12 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request updates the SDK to support pandas v3 by dynamically adjusting timestamp precision (nanoseconds for pandas v2 and lower, and milliseconds for pandas v3+). Feedback on the changes highlights a bug in _pandas_helpers.py where np.ndarray timestamps are parsed without specifying unit="ms", and suggests using the PANDAS_TS_UNIT constant in integration tests to avoid hardcoding the timestamp unit.
| ] # poetry 2.4.1 does not correctly limit numpy 2.2 to only python 3.10 | ||
| sympy = ["sympy"] | ||
| pandas = ["pandas (>=2.1, <3)"] | ||
| pandas = ["pandas >=2.1"] |
There was a problem hiding this comment.
Haven't really reviewed anything yet, but shouldn't we have a strict upper bound still? Like, >= 2.1, <4
There was a problem hiding this comment.
No, but it sure is up for discussion - and has been in the past.
Both numpy and pandas are very mature and responsible packages that we used to have open upper bounds for. It was only when v3 of pandas came out with what I deemed "a potentially breaking change for us" that we temporarily forced major version == 2 until it was addressed by us (aka as of this PR).
I didn't find the original discussion, but I found this; best summarised as "lock out future major releases just in case":
https://cognitedata.slack.com/archives/CG10VQPFX/p1773826110266419?thread_ts=1773760929.634279&cid=CG10VQPFX
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #2780 +/- ##
=======================================
Coverage 93.04% 93.04%
=======================================
Files 515 516 +1
Lines 53006 53097 +91
=======================================
+ Hits 49317 49404 +87
- Misses 3689 3693 +4
🚀 New features to boost your workflow:
|
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request updates the SDK to support Pandas v3 by dynamically adjusting the precision of generated timestamp columns and indices (nanoseconds for Pandas v2 and milliseconds for Pandas v3) and updates dependency constraints to allow pandas >=2.1. Feedback on the changes suggests broadening the type hint of to_pandas_timestamp to int | float | None to prevent type-checking errors with optional timestamps, and adding a pytest.importorskip guard to the new test file to avoid failures during core-only test runs.
aa7934d to
c88a5cf
Compare
Enable pandas v3 compatibility by making SDK timestamp resolution version-conditional (
"ns"on v2,"ms"on v3), remove the pandas upper bound, update migration/docs and tests, and cache version/unit helper lookups.