Skip to content

fix(core): load existing Milvus collections after restart - #1174

Open
zc277584121 wants to merge 1 commit into
basicmachines-co:mainfrom
zc277584121:fix/milvus-load-existing-collection
Open

fix(core): load existing Milvus collections after restart#1174
zc277584121 wants to merge 1 commit into
basicmachines-co:mainfrom
zc277584121:fix/milvus-load-existing-collection

Conversation

@zc277584121

Copy link
Copy Markdown
Contributor

Summary

  • Load a compatible existing Milvus collection after validating its schema and COSINE index.
  • Restore Milvus Lite query, search, and orphan reconciliation after a CLI process restart.
  • Add repository-level coverage and a real two-process Milvus Lite regression test.

Root cause

Milvus Lite persists collection data when its owning process exits, but the collection is released. A fresh Basic Memory process validated the existing collection and returned its dimensions without loading it, so the first search or query failed with error code 101.

New collections remain on the existing create path, which already loads them automatically. Existing compatible collections are now loaded during repository initialization before they are returned for use.

Verification

  • uv run pytest -q tests/repository/test_milvus_config.py tests/repository/test_milvus_index.py tests/repository/test_milvus_repository.py tests/repository/test_semantic_vector_index.py test-int/semantic/test_milvus_lite.py --no-cov — 85 passed
  • uv run pytest -q tests/repository/test_milvus_repository.py test-int/semantic/test_milvus_lite.py — 45 passed
  • uv run ty check src tests test-int
  • uv run ruff check src/basic_memory/repository/milvus_repository.py tests/repository/test_milvus_repository.py test-int/semantic/test_milvus_lite.py
  • uv run ruff format --check src/basic_memory/repository/milvus_repository.py tests/repository/test_milvus_repository.py test-int/semantic/test_milvus_lite.py

Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3b2450f5c3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +245 to +248
self._client.load_collection(
collection_name=collection_name,
timeout=self._timeout,
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Defer loading incompatible Milvus collections

When an existing collection has a valid COSINE index but a different embedding dimension, this new load happens before collection_dimensions() returns to MilvusVectorIndex._initialize_blocking(), where the mismatch is rejected only afterward. In that migration/rollback scenario, simply starting Basic Memory can load the old collection into Milvus memory before failing with the intended “Refusing to replace shared vector storage” error, which is costly for large remote collections and contradicts the preserve-until-migration path. Move the load to the caller path after dimensions == self.scope.dimensions, or expose an explicit load step for compatible collections only.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant