Skip to content

fix(data_classes): invalidate CogniteResourceList lookups on mutation - #2774

Open
jaideeppyne wants to merge 3 commits into
cognitedata:masterfrom
jaideeppyne:fix/resource-list-get-stale-after-mutation
Open

fix(data_classes): invalidate CogniteResourceList lookups on mutation#2774
jaideeppyne wants to merge 3 commits into
cognitedata:masterfrom
jaideeppyne:fix/resource-list-get-stale-after-mutation

Conversation

@jaideeppyne

Copy link
Copy Markdown

Description

Fixes #1188.

CogniteResourceList builds three identifier→item lookup maps as @cached_property_id_to_item, _external_id_to_item, _instance_id_to_item — which .get(id=.../external_id=.../instance_id=...) reads. The maps are built lazily on the first .get(...) call.

Only extend() was written to keep those cached maps in sync. Every other membership-mutating operationappend, insert, remove, pop, clear, __setitem__, __delitem__, __iadd__ (+=), __imul__ (*=) — is inherited straight from UserList and mutates self.data without invalidating the caches. So once a map is cached:

  • get(id=X) after pop/remove/del returns the removed item (should be None);
  • get(id=X) after append/insert/+= returns None for an item that is present.

This is exactly the gap the existing # TODO: We inherit a lot from UserList that we don't actually support... comment flags.

Fix

Add _clear_identifier_lookups() — which drops the three cached_property entries from self.__dict__ so they rebuild on next access — and override each mutating method to call it after delegating to super(). The invalidate-then-rebuild approach is simpler and less error-prone than hand-patching the dicts.

Test

test_get_is_invalidated_by_mutations primes the lookup cache with a get(), then runs each mutator (pop/remove/append/insert/+=/__setitem__/del/clear) and asserts get() reflects the new membership (removed → None, added → found). It fails on the current code (get(id=2) returns the popped item) and passes with the fix.

I ran the TestCogniteResourceList suite: 22 passed. The one remaining failure (test_to_pandas) is a pre-existing pandas-version dtype mismatch unrelated to this change (tracked separately). ruff check/format clean; mypy clean on the changed file.

`CogniteResourceList.get(id=.../external_id=.../instance_id=...)` reads three
`@cached_property` lookup maps (`_id_to_item`, `_external_id_to_item`,
`_instance_id_to_item`) that are built lazily on the first `get(...)`. Only
`extend()` kept them in sync — every other membership-mutating operation
(`append`, `insert`, `remove`, `pop`, `clear`, `__setitem__`, `__delitem__`,
`__iadd__`, `__imul__`) was inherited from `UserList` and mutated `self.data`
without invalidating the caches. So after any such mutation `get()` silently
returned a removed item, or `None` for an item that is present.

Add `_clear_identifier_lookups()` (drops the cached maps from `__dict__`) and
override every mutator to call it after delegating to `super()`. Closes the
gap noted by the existing `# TODO: We inherit a lot from UserList ...` comment.

Adds a regression test that primes the lookup cache, then exercises each mutator
and asserts `get()` reflects the new membership. It fails on the current code
(`get(id=...)` returns a popped item) and passes with the fix.

Fixes cognitedata#1188
@jaideeppyne
jaideeppyne requested review from a team as code owners August 18, 2026 06:26

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request ensures that cached identifier lookup maps in CogniteResourceList are properly invalidated upon mutation by overriding list-mutating methods and adding corresponding unit tests. The review feedback recommends delegating __iadd__ to extend to prevent bypassing duplicate checks, adding a test assertion for this duplicate validation, and replacing Any type hints with stronger types in __setitem__ and __delitem__ to comply with the repository's style guide.

Comment thread cognite/client/data_classes/_base.py Outdated
Comment on lines +314 to +317
def __iadd__(self: T_CogniteResourceList, other: Iterable[Any]) -> T_CogniteResourceList:
super().__iadd__(other)
self._clear_identifier_lookups()
return self

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

In Python, the in-place addition operator += (__iadd__) is semantically equivalent to .extend(). Currently, __iadd__ bypasses the duplicate checks implemented in extend(), allowing duplicate resources to be silently added to the list. This can lead to unexpected behavior and inconsistencies in the lookup maps.\n\nTo ensure consistent validation, delegate __iadd__ directly to self.extend(other).

    def __iadd__(self: T_CogniteResourceList, other: Iterable[Any]) -> T_CogniteResourceList:\n        self.extend(other)\n        return self

Comment thread cognite/client/data_classes/_base.py Outdated
Comment on lines +306 to +308
def __setitem__(self, i: Any, item: Any) -> None:
super().__setitem__(i, item)
self._clear_identifier_lookups()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

According to the Cognite Python Style Guide, we should avoid using Any and prefer strong typing. The index i can be typed as SupportsIndex | slice, and item can be typed as T_CogniteResource | Iterable[T_CogniteResource] to be more precise and type-safe.

    def __setitem__(self, i: SupportsIndex | slice, item: T_CogniteResource | Iterable[T_CogniteResource]) -> None:\n        super().__setitem__(i, item)\n        self._clear_identifier_lookups()
References
  1. Strong Typing: Use type hints extensively with MyPy. Avoid Any when possible (link)

Comment thread cognite/client/data_classes/_base.py Outdated
Comment on lines +310 to +312
def __delitem__(self, i: Any) -> None:
super().__delitem__(i)
self._clear_identifier_lookups()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

According to the Cognite Python Style Guide, we should avoid using Any and prefer strong typing. The index i can be typed as SupportsIndex | slice to be more precise and type-safe.

    def __delitem__(self, i: SupportsIndex | slice) -> None:\n        super().__delitem__(i)\n        self._clear_identifier_lookups()
References
  1. Strong Typing: Use type hints extensively with MyPy. Avoid Any when possible (link)

Comment on lines +791 to +792
resource_list += MyResourceList([MyResource(id=5, external_id="5")])
assert resource_list.get(id=5) == MyResource(id=5, external_id="5")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Add an assertion to verify that += (__iadd__) correctly raises a ValueError when attempting to add duplicate resources, ensuring consistency with extend() and preventing regressions.

        resource_list += MyResourceList([MyResource(id=5, external_id=\"5\")])\n        assert resource_list.get(id=5) == MyResource(id=5, external_id=\"5\")\n\n        with pytest.raises(ValueError, match=\"introduce duplicates\"):\n            resource_list += MyResourceList([MyResource(id=5, external_id=\"5\")])

…up check

- __iadd__ now delegates to extend() so += enforces the same duplicate-id
  check (and cache maintenance) instead of bypassing it via UserList.
- __setitem__/__delitem__ use SupportsIndex | slice (and a precise item type)
  instead of Any, per the style guide.
- Add a test asserting += raises ValueError on duplicate ids.
@jaideeppyne

Copy link
Copy Markdown
Author

Thanks for the review — addressed all three in 3d5f563:

  • __iadd__ now delegates to self.extend(other), so += enforces the same duplicate-id ValueError (and cache maintenance) as extend() instead of bypassing it via UserList.
  • __setitem__ / __delitem__ now use SupportsIndex | slice (and a precise item type) instead of Any.
  • Added an assertion that += with a duplicate id raises ValueError.

mypy/ruff clean and the mutation test still passes.

warn_unused_ignores is enabled in mypy.ini, so the extra [assignment]
code in the type: ignore comment triggers [unused-ignore] and fails the
mypy CI check. Only [index] is required.
@jaideeppyne

Copy link
Copy Markdown
Author

Small follow-up in 423372e: dropped an unused assignment code from the # type: ignore[index, assignment] on __setitem__. Since mypy.ini sets warn_unused_ignores = true, the redundant code tripped [unused-ignore]; # type: ignore[index] alone is sufficient. mypy and ruff are now clean on the file. Thanks again for the thorough review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CogniteResourceList lacks correct dunder method implementations

1 participant