fix(data_classes): invalidate CogniteResourceList lookups on mutation - #2774
fix(data_classes): invalidate CogniteResourceList lookups on mutation#2774jaideeppyne wants to merge 3 commits into
Conversation
`CogniteResourceList.get(id=.../external_id=.../instance_id=...)` reads three `@cached_property` lookup maps (`_id_to_item`, `_external_id_to_item`, `_instance_id_to_item`) that are built lazily on the first `get(...)`. Only `extend()` kept them in sync — every other membership-mutating operation (`append`, `insert`, `remove`, `pop`, `clear`, `__setitem__`, `__delitem__`, `__iadd__`, `__imul__`) was inherited from `UserList` and mutated `self.data` without invalidating the caches. So after any such mutation `get()` silently returned a removed item, or `None` for an item that is present. Add `_clear_identifier_lookups()` (drops the cached maps from `__dict__`) and override every mutator to call it after delegating to `super()`. Closes the gap noted by the existing `# TODO: We inherit a lot from UserList ...` comment. Adds a regression test that primes the lookup cache, then exercises each mutator and asserts `get()` reflects the new membership. It fails on the current code (`get(id=...)` returns a popped item) and passes with the fix. Fixes cognitedata#1188
There was a problem hiding this comment.
Code Review
This pull request ensures that cached identifier lookup maps in CogniteResourceList are properly invalidated upon mutation by overriding list-mutating methods and adding corresponding unit tests. The review feedback recommends delegating __iadd__ to extend to prevent bypassing duplicate checks, adding a test assertion for this duplicate validation, and replacing Any type hints with stronger types in __setitem__ and __delitem__ to comply with the repository's style guide.
| def __iadd__(self: T_CogniteResourceList, other: Iterable[Any]) -> T_CogniteResourceList: | ||
| super().__iadd__(other) | ||
| self._clear_identifier_lookups() | ||
| return self |
There was a problem hiding this comment.
In Python, the in-place addition operator += (__iadd__) is semantically equivalent to .extend(). Currently, __iadd__ bypasses the duplicate checks implemented in extend(), allowing duplicate resources to be silently added to the list. This can lead to unexpected behavior and inconsistencies in the lookup maps.\n\nTo ensure consistent validation, delegate __iadd__ directly to self.extend(other).
def __iadd__(self: T_CogniteResourceList, other: Iterable[Any]) -> T_CogniteResourceList:\n self.extend(other)\n return self| def __setitem__(self, i: Any, item: Any) -> None: | ||
| super().__setitem__(i, item) | ||
| self._clear_identifier_lookups() |
There was a problem hiding this comment.
According to the Cognite Python Style Guide, we should avoid using Any and prefer strong typing. The index i can be typed as SupportsIndex | slice, and item can be typed as T_CogniteResource | Iterable[T_CogniteResource] to be more precise and type-safe.
def __setitem__(self, i: SupportsIndex | slice, item: T_CogniteResource | Iterable[T_CogniteResource]) -> None:\n super().__setitem__(i, item)\n self._clear_identifier_lookups()References
- Strong Typing: Use type hints extensively with MyPy. Avoid Any when possible (link)
| def __delitem__(self, i: Any) -> None: | ||
| super().__delitem__(i) | ||
| self._clear_identifier_lookups() |
There was a problem hiding this comment.
According to the Cognite Python Style Guide, we should avoid using Any and prefer strong typing. The index i can be typed as SupportsIndex | slice to be more precise and type-safe.
def __delitem__(self, i: SupportsIndex | slice) -> None:\n super().__delitem__(i)\n self._clear_identifier_lookups()References
- Strong Typing: Use type hints extensively with MyPy. Avoid Any when possible (link)
| resource_list += MyResourceList([MyResource(id=5, external_id="5")]) | ||
| assert resource_list.get(id=5) == MyResource(id=5, external_id="5") |
There was a problem hiding this comment.
Add an assertion to verify that += (__iadd__) correctly raises a ValueError when attempting to add duplicate resources, ensuring consistency with extend() and preventing regressions.
resource_list += MyResourceList([MyResource(id=5, external_id=\"5\")])\n assert resource_list.get(id=5) == MyResource(id=5, external_id=\"5\")\n\n with pytest.raises(ValueError, match=\"introduce duplicates\"):\n resource_list += MyResourceList([MyResource(id=5, external_id=\"5\")])…up check - __iadd__ now delegates to extend() so += enforces the same duplicate-id check (and cache maintenance) instead of bypassing it via UserList. - __setitem__/__delitem__ use SupportsIndex | slice (and a precise item type) instead of Any, per the style guide. - Add a test asserting += raises ValueError on duplicate ids.
|
Thanks for the review — addressed all three in 3d5f563:
mypy/ruff clean and the mutation test still passes. |
warn_unused_ignores is enabled in mypy.ini, so the extra [assignment] code in the type: ignore comment triggers [unused-ignore] and fails the mypy CI check. Only [index] is required.
|
Small follow-up in 423372e: dropped an unused |
Description
Fixes #1188.
CogniteResourceListbuilds three identifier→item lookup maps as@cached_property—_id_to_item,_external_id_to_item,_instance_id_to_item— which.get(id=.../external_id=.../instance_id=...)reads. The maps are built lazily on the first.get(...)call.Only
extend()was written to keep those cached maps in sync. Every other membership-mutating operation —append,insert,remove,pop,clear,__setitem__,__delitem__,__iadd__(+=),__imul__(*=) — is inherited straight fromUserListand mutatesself.datawithout invalidating the caches. So once a map is cached:get(id=X)afterpop/remove/delreturns the removed item (should beNone);get(id=X)afterappend/insert/+=returnsNonefor an item that is present.This is exactly the gap the existing
# TODO: We inherit a lot from UserList that we don't actually support...comment flags.Fix
Add
_clear_identifier_lookups()— which drops the threecached_propertyentries fromself.__dict__so they rebuild on next access — and override each mutating method to call it after delegating tosuper(). The invalidate-then-rebuild approach is simpler and less error-prone than hand-patching the dicts.Test
test_get_is_invalidated_by_mutationsprimes the lookup cache with aget(), then runs each mutator (pop/remove/append/insert/+=/__setitem__/del/clear) and assertsget()reflects the new membership (removed →None, added → found). It fails on the current code (get(id=2)returns the popped item) and passes with the fix.I ran the
TestCogniteResourceListsuite: 22 passed. The one remaining failure (test_to_pandas) is a pre-existing pandas-version dtype mismatch unrelated to this change (tracked separately).ruffcheck/format clean;mypyclean on the changed file.