Add int8 data type support to diskann-garnet/vectorset

[Copilot]

• Analyze existing SB8 test patterns (all use VectorQuantType::NoQuant)
• Add create_test_index_q8 helper in ffi_tests.rs
• Add basic Q8 insert/search tests in ffi_tests.rs
• Parameterize run_sb8_recall in ffi_recall_tests.rs to accept quant type
• Add Q8 recall grid tests in ffi_recall_tests.rs
• Verify tests compile and pass (all 53 garnet tests pass)
• Clippy clean

[harsha-simhadri]

@copilot start reviewing this draft PR

[Copilot]

@copilot start reviewing this draft PR

Here's a summary of this PR's changes:

Rust FFI (diskann-garnet/src/lib.rs)

• Added VectorValueType::SB8 (signed int8, discriminant = 3) to match the C# enum
• Added VectorQuantType::Q8 arm in create_index → dispatches to create_index_impl::<i8>
• Extended interpret_vector:
  • FP32 input + Q8 quant: converts f32→i8→u8 for storage
  • SB8 input + Q8 quant: zero-copy passthrough (raw bytes are already i8)
  • SB8 input + NoQuant: sign-extends each byte to f32
• Byte-length calculation handles SB8 same as XB8 (1 byte/dim)

vectorset CLI (vectorset/src/main.rs)

• Added DataType::Int8 variant, impl Element for i8
• Dispatches SB8 value type and Q8 quantizer tokens for ingest/query

Tests (diskann-garnet/src/ffi_tests.rs, ffi_recall_tests.rs)

• add_check_and_remove_sb8_vector: insert, ID validation, search, remove lifecycle
• sb8_search_after_delete: verifies deletion exclusion
• sb8_search_multi_results: multi-vector nearest-neighbor ordering
• sb8_grid_l2_recall_3d_7 and sb8_grid_l2_recall_4d_5: grid recall tests matching the existing XB8/FP32 pattern

Garnet companion patch (garnet-sb8-support.patch.md)

• Instructions for adding VectorValueType.SB8 and wiring it through parsing, DiskANNService, and VectorManager in the C# Garnet repo

All 45 tests pass (including 5 new SB8 tests). Build is clean.

[Copilot]

Pull request overview

Adds end-to-end int8 (SB8) vector support to the diskann-garnet FFI layer and the vectorset ingestion/query tool, parallel to the existing uint8 (XB8) path. A new VectorValueType::SB8 is wired through create_index/interpret_vector with a new VectorQuantType::Q8 storage path (i8 backing) and a SB8 → NoQuant conversion path (sign-extended to f32). The PR also includes a Markdown document describing the corresponding manual edits required in the external microsoft/garnet C# server, and an unrelated tweak to vectorset/config.toml.example.

Changes:

• Add VectorValueType::SB8 and a VectorQuantType::Q8 branch in diskann-garnet, including FP32↔Q8 and SB8↔NoQuant conversions.
• Add Int8/SB8/Q8 handling to the vectorset CLI so it can ingest and query int8 datasets.
• Add SB8 FFI unit tests and recall tests; add a Garnet-side patch instructions doc and update the README.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
diskann-garnet/src/lib.rs	New SB8 enum variant and Q8 storage / SB8 interpretation paths in create_index and interpret_vector.
diskann-garnet/src/ffi_tests.rs	Adds SB8 helpers and tests covering insert/remove/search, negative values, large external IDs, and bitmap filtering (all using a NoQuant index).
diskann-garnet/src/ffi_recall_tests.rs	Adds SB8 recall helpers and 1D–4D grid recall tests against an f32/NoQuant index.
diskann-garnet/README.md	Documents new SB8 value type and Q8 quantizer and recommended pairings.
vectorset/src/main.rs	Adds Int8 DataType, Element for i8, and SB8/Q8 arguments in ingest/query pipelines.
vectorset/config.toml.example	Replaces the commented scope placeholder with scope = "" (unrelated to int8 support).
garnet-sb8-support.patch.md	New document with manual edit instructions for the external microsoft/garnet repo to mirror the SB8/Q8 wiring.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.