diff --git a/CHANGELOG.md b/CHANGELOG.md index 0aad3b36..f88fe405 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,15 @@ ## Unreleased +- Remove the 2 GiB aggregate canonical-payload limit from current SQLite + graph-index snapshots. Large graphs are now admitted by their validated + manifest and stored as independently bounded, content-addressed tree objects + with bounded write batches; point and range query work remains explicitly + bounded. Store status, validation, backup, and restore now stream file + digests and validate every reachable tree object without materializing the + canonical graph. The legacy monolithic snapshot compatibility API and + whole-JSON readers retain their existing allocation limits. + - Prevent oversized universal-evidence collections from collapsing code graphs to file scaffolding. Default-low builds compact unused leaf declaration details, project retained declarations before resolution, and resolve safe diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index 33874b74..c865aede 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -384,7 +384,15 @@ surface. Backups are digest-bound directories and restores never overwrite an existing destination. Local publication retains two complete snapshots and performs bounded reachability GC; remote leases, service quotas, and distributed GC remain deferred. The local API enforces bounded values, scans, -transactions, graph sizes, and request work. +transactions, and request work. Current `compass.store.graph-index/2` +snapshots do not impose an aggregate canonical-payload or record-count limit: +their manifest uses `u64` byte and record counts, while each immutable tree +object, write batch, scan, and query remains independently bounded. The legacy +monolithic `compass.store.graph-snapshot/1` compatibility API still enforces +its 2 GiB materialized-payload limit. `compass store status|validate|backup|restore` +stream graph and database digests through fixed-size buffers and traverse the +reachable immutable tree objects with bounded cache and path memory; they do +not depend on the whole-JSON reader limit. The hard-cut boundary is the sidecar and all disposable indexes. When a physical format is invalid or outside the support window, preserve diff --git a/crates/compass-cli/src/store_commands.rs b/crates/compass-cli/src/store_commands.rs index e8e55a98..99c60bc0 100644 --- a/crates/compass-cli/src/store_commands.rs +++ b/crates/compass-cli/src/store_commands.rs @@ -1,11 +1,9 @@ -use std::fs; +use std::fs::{self, File}; +use std::io::Read; use std::path::{Path, PathBuf}; use compass_files::BuildGuard; -use compass_graph::{ - GRAPH_SNAPSHOT_SELECTOR_SCHEMA_V1, GraphSnapshotReader, SnapshotSelector, canonical_graph_json, -}; -use compass_model::code_graph::GraphDocument; +use compass_graph::{GRAPH_SNAPSHOT_SELECTOR_SCHEMA_V1, GraphSnapshotReader, SnapshotSelector}; use compass_store::{ STORE_FILE_NAME, STORE_REF_FILE_NAME, STORE_SCHEMA_V1, SqliteStore, StoreRef, local_sqlite_store_path, @@ -60,17 +58,15 @@ fn status(args: &[String]) -> Result { let graph_path = output.join("graph.json"); let store_path = local_sqlite_store_path(&graph_path); let reference_path = output.join(STORE_REF_FILE_NAME); - let graph = if graph_path.is_file() { - let bytes = fs::read(&graph_path).map_err(|error| format!("read graph.json: {error}"))?; - let document = GraphDocument::load(&graph_path).map_err(|error| error.to_string())?; - Some(graph_status(&bytes, &document)) + let mut graph = if graph_path.is_file() { + Some(graph_status(&graph_path)?) } else { None }; let store = if store_path.is_file() { match SqliteStore::open_read_only(&store_path) { - Ok(store) => match validate_store(&store, graph.as_ref(), &graph_path) { + Ok(store) => match validate_store(&store, graph.as_mut(), &graph_path) { Ok((reference, snapshot_id, manifest_digest)) => Some(json!({ "present": true, "valid": true, @@ -142,16 +138,14 @@ fn validate(args: &[String]) -> Result { store_path.display() )); } - let graph = if graph_path.is_file() { - let bytes = fs::read(&graph_path).map_err(|error| format!("read graph.json: {error}"))?; - let document = GraphDocument::load(&graph_path).map_err(|error| error.to_string())?; - Some(graph_status(&bytes, &document)) + let mut graph = if graph_path.is_file() { + Some(graph_status(&graph_path)?) } else { None }; let store = SqliteStore::open_read_only(&store_path).map_err(|error| error.to_string())?; let (reference, snapshot_id, manifest_digest) = - validate_store(&store, graph.as_ref(), &graph_path)?; + validate_store(&store, graph.as_mut(), &graph_path)?; let reference_path = output.join(STORE_REF_FILE_NAME); if !reference_path.is_file() { return Err(format!( @@ -194,17 +188,15 @@ fn backup(args: &[String]) -> Result { let graph_path = output.join("graph.json"); let store_path = local_sqlite_store_path(&graph_path); let reference_path = output.join(STORE_REF_FILE_NAME); - let graph_bytes = fs::read(&graph_path).map_err(|error| format!("read graph.json: {error}"))?; - let graph = GraphDocument::load(&graph_path).map_err(|error| error.to_string())?; + let mut graph_value = graph_status(&graph_path)?; let reference_bytes = fs::read(&reference_path).map_err(|error| format!("read store.ref: {error}"))?; let reference: StoreRef = serde_json::from_slice(&reference_bytes) .map_err(|error| format!("decode store.ref: {error}"))?; reference.validate().map_err(|error| error.to_string())?; let store = SqliteStore::open(&store_path).map_err(|error| error.to_string())?; - let graph_value = graph_status(&graph_bytes, &graph); let (_, snapshot_id, manifest_digest) = - validate_store(&store, Some(&graph_value), &graph_path)?; + validate_store(&store, Some(&mut graph_value), &graph_path)?; if reference.snapshot_id != snapshot_id || reference.manifest_digest != manifest_digest { return Err("store.ref does not match the active snapshot".to_owned()); } @@ -223,7 +215,11 @@ fn backup(args: &[String]) -> Result { schema: BACKUP_SCHEMA_V1.to_owned(), store_schema: STORE_SCHEMA_V1.to_owned(), adapter: "sqlite".to_owned(), - graph_digest: digest(&graph_bytes), + graph_digest: graph_value + .get("sha256") + .and_then(Value::as_str) + .ok_or_else(|| "graph status is missing its digest".to_owned())? + .to_owned(), store_digest: digest_file(&destination.join(STORE_FILE_NAME))?, snapshot_id, manifest_digest, @@ -282,20 +278,14 @@ fn restore(args: &[String]) -> Result { .store_reference .validate() .map_err(|error| error.to_string())?; - let graph_bytes = fs::read(source.join("graph.json")) - .map_err(|error| format!("read backup graph.json: {error}"))?; - if digest(&graph_bytes) != manifest.graph_digest { + let graph_path = source.join("graph.json"); + let mut graph_value = graph_status(&graph_path)?; + if graph_value.get("sha256").and_then(Value::as_str) != Some(&manifest.graph_digest) { return Err("backup graph digest does not match manifest".to_owned()); } - let graph_path = source.join("graph.json"); - let graph = GraphDocument::load(&graph_path).map_err(|error| error.to_string())?; let backup_store = source.join(STORE_FILE_NAME); let store = SqliteStore::open_read_only(&backup_store).map_err(|error| error.to_string())?; - validate_store( - &store, - Some(&graph_status(&graph_bytes, &graph)), - &graph_path, - )?; + validate_store(&store, Some(&mut graph_value), &graph_path)?; if digest_file(&backup_store)? != manifest.store_digest { return Err("backup store digest does not match manifest".to_owned()); } @@ -313,7 +303,7 @@ fn restore(args: &[String]) -> Result { let result = (|| { SqliteStore::restore_from(&backup_store, destination.join(STORE_FILE_NAME)) .map_err(|error| error.to_string())?; - fs::write(destination.join("graph.json"), &graph_bytes) + fs::copy(&graph_path, destination.join("graph.json")) .map_err(|error| format!("restore graph.json: {error}"))?; fs::copy( source.join(STORE_REF_FILE_NAME), @@ -337,7 +327,7 @@ fn restore(args: &[String]) -> Result { fn validate_store( store: &SqliteStore, - graph: Option<&Value>, + graph: Option<&mut Value>, graph_path: &Path, ) -> Result<(StoreRef, String, String), String> { let reference_path = graph_path @@ -360,16 +350,26 @@ fn validate_store( ) .map_err(|error| error.to_string())?; let manifest = reader.manifest(); - let exported = reader - .export_json_bytes() + reader + .validate_integrity() .map_err(|error| error.to_string())?; if let Some(graph) = graph { - let graph_bytes = - fs::read(graph_path).map_err(|error| format!("read graph.json: {error}"))?; - let expected = graph_bytes_from_status(graph, &graph_bytes)?; - if exported != expected { - return Err("store export is not byte-identical to graph.json".to_owned()); + let graph_bytes = graph + .get("bytes") + .and_then(Value::as_u64) + .ok_or_else(|| "graph status is missing its byte count".to_owned())?; + let graph_digest = graph + .get("sha256") + .and_then(Value::as_str) + .ok_or_else(|| "graph status is missing its digest".to_owned())?; + if graph_bytes != manifest.graph_bytes || graph_digest != manifest.graph_digest { + return Err(format!( + "store manifest does not match {}", + graph_path.display() + )); } + graph["nodes"] = json!(manifest.node_count); + graph["edges"] = json!(manifest.edge_count); } let actual = store .graph_snapshot_reference_for(&reference.snapshot_id, &reference.manifest_digest) @@ -381,27 +381,15 @@ fn validate_store( Ok((reference, manifest.snapshot_id.clone(), manifest_digest)) } -fn graph_status(bytes: &[u8], graph: &GraphDocument) -> Value { - json!({ +fn graph_status(path: &Path) -> Result { + let bytes = fs::metadata(path) + .map_err(|error| format!("inspect {}: {error}", path.display()))? + .len(); + Ok(json!({ "present": true, - "bytes": bytes.len(), - "sha256": digest(bytes), - "nodes": graph.nodes.len(), - "edges": graph.links.len(), - }) -} - -fn graph_bytes_from_status(graph: &Value, bytes: &[u8]) -> Result, String> { - let expected = graph - .get("sha256") - .and_then(Value::as_str) - .ok_or_else(|| "graph status is missing its digest".to_owned())?; - if expected != digest(bytes) { - return Err("graph status digest changed during validation".to_owned()); - } - let document = serde_json::from_slice::(bytes) - .map_err(|error| format!("decode graph.json: {error}"))?; - canonical_graph_json(&document).map_err(|error| error.to_string()) + "bytes": bytes, + "sha256": digest_file(path)?, + })) } fn output_root(args: &[String]) -> Result { @@ -460,14 +448,21 @@ fn option<'a>(args: &'a [String], name: &str) -> Option<&'a str> { }) } -fn digest(bytes: &[u8]) -> String { - format!("{:x}", Sha256::digest(bytes)) -} - fn digest_file(path: &Path) -> Result { - Ok(digest(&fs::read(path).map_err(|error| { - format!("read {}: {error}", path.display()) - })?)) + let mut reader = + File::open(path).map_err(|error| format!("read {}: {error}", path.display()))?; + let mut hasher = Sha256::new(); + let mut buffer = vec![0_u8; 1024 * 1024]; + loop { + let read = reader + .read(&mut buffer) + .map_err(|error| format!("read {}: {error}", path.display()))?; + if read == 0 { + break; + } + hasher.update(&buffer[..read]); + } + Ok(format!("{:x}", hasher.finalize())) } fn render_text(value: &Value) -> String { diff --git a/crates/compass-cli/tests/store_cli.rs b/crates/compass-cli/tests/store_cli.rs index 1a68e2ec..1d07df9a 100644 --- a/crates/compass-cli/tests/store_cli.rs +++ b/crates/compass-cli/tests/store_cli.rs @@ -124,3 +124,36 @@ fn store_validate_rejects_a_corrupt_sidecar_without_touching_graph_json() assert_eq!(fs::read(active_output.join("graph.json"))?, graph); Ok(()) } + +#[test] +fn store_status_and_validation_do_not_use_the_whole_json_reader_limit() -> Result<(), Box> +{ + let root = tempfile::tempdir()?; + fs::write(root.path().join("main.rs"), "fn main() {}\n")?; + let init = Command::new(env!("CARGO_BIN_EXE_compass")) + .args(["init", ".", "--yes", "--store", "sqlite"]) + .current_dir(root.path()) + .env_remove("COMPASS_OUT") + .output()?; + assert!(init.status.success()); + let output = root.path().join("compass-out"); + + for operation in ["status", "validate"] { + let result = Command::new(env!("CARGO_BIN_EXE_compass")) + .args([ + "store", + operation, + output.to_str().ok_or("output path")?, + "--format", + "json", + ]) + .env("COMPASS_MAX_GRAPH_BYTES", "1") + .output()?; + assert!( + result.status.success(), + "{operation} stderr: {}", + String::from_utf8_lossy(&result.stderr) + ); + } + Ok(()) +} diff --git a/crates/compass-graph/src/snapshot.rs b/crates/compass-graph/src/snapshot.rs index b012a893..01d02058 100644 --- a/crates/compass-graph/src/snapshot.rs +++ b/crates/compass-graph/src/snapshot.rs @@ -19,9 +19,9 @@ use compass_model::code_graph::{ }; use compass_model::validate_code_graph; use compass_store::{ - ImmutableWrite, Key, MAX_GRAPH_BYTES, MAX_IMMUTABLE_BATCH_BYTES, MAX_IMMUTABLE_BATCH_ITEMS, - MAX_KEY_SEGMENTS, MAX_SCAN_BYTES, MAX_SCAN_ITEMS, MAX_VALUE_BYTES, NamespaceId, PartitionKey, - Store, StoreError, WriteCondition, decode_key_segments, encode_key_segments, + ImmutableWrite, Key, MAX_IMMUTABLE_BATCH_BYTES, MAX_IMMUTABLE_BATCH_ITEMS, MAX_KEY_SEGMENTS, + MAX_SCAN_BYTES, MAX_SCAN_ITEMS, MAX_VALUE_BYTES, NamespaceId, PartitionKey, Store, StoreError, + WriteCondition, decode_key_segments, encode_key_segments, }; use rayon::prelude::*; use serde::{Deserialize, Serialize}; @@ -43,6 +43,12 @@ pub const GRAPH_SNAPSHOT_CATALOG_PARTITION: &str = "graph-snapshot/catalog"; pub const GRAPH_SNAPSHOT_ACTIVE_KEY: &str = "active"; pub const GRAPH_SNAPSHOT_MAX_DEPTH: usize = 64; pub const GRAPH_SNAPSHOT_MAX_OBJECTS: usize = 100_000; +/// Maximum records materialized by one snapshot read/export request. +/// +/// This is deliberately not a limit on the logical graph stored in the +/// content-addressed tree. Point and range queries remain independently +/// bounded even when a snapshot contains more records than one materialized +/// response may return. pub const GRAPH_SNAPSHOT_MAX_ITEMS: usize = 5_000_000; pub const GRAPH_SNAPSHOT_MAX_FANOUT: usize = 32; pub const GRAPH_SNAPSHOT_MAX_LEAF_ENTRIES: usize = 128; @@ -169,17 +175,10 @@ impl GraphSnapshotManifest { SnapshotError::Corrupt(format!("{name} is not a SHA-256 digest: {error}")) })?; } - if self.graph_bytes == 0 || self.graph_bytes > MAX_GRAPH_BYTES as u64 { - return Err(SnapshotError::Corrupt(format!( - "graph byte count exceeds the {MAX_GRAPH_BYTES}-byte limit" - ))); - } - if self.node_count > GRAPH_SNAPSHOT_MAX_ITEMS as u64 - || self.edge_count > GRAPH_SNAPSHOT_MAX_ITEMS as u64 - { - return Err(SnapshotError::Limit(format!( - "graph record count exceeds the {GRAPH_SNAPSHOT_MAX_ITEMS}-item snapshot limit" - ))); + if self.graph_bytes == 0 { + return Err(SnapshotError::Corrupt( + "graph byte count must be nonzero".to_owned(), + )); } if self.roots.len() != IndexKind::ALL.len() { return Err(SnapshotError::Corrupt(format!( @@ -212,6 +211,27 @@ impl GraphSnapshotManifest { "manifest contains duplicate or missing index roots".to_owned(), )); } + for (index, expected) in [ + (IndexKind::Nodes, self.node_count), + (IndexKind::Edges, self.edge_count), + (IndexKind::Outgoing, self.edge_count), + (IndexKind::Incoming, self.edge_count), + ] { + let actual = self + .roots + .iter() + .find(|root| root.index == index) + .map(|root| root.entry_count) + .ok_or_else(|| { + SnapshotError::Corrupt(format!("{} root is missing", index.as_str())) + })?; + if actual != expected { + return Err(SnapshotError::Corrupt(format!( + "{} root count {actual} does not match manifest count {expected}", + index.as_str() + ))); + } + } Ok(()) } } @@ -1016,10 +1036,7 @@ fn write_fact_neutral_graph_json_delta_inner( validate_records: bool, writer: &mut W, ) -> io::Result { - if previous_bytes.is_empty() - || previous_bytes.len() > MAX_GRAPH_BYTES - || previous_bytes.len() > GRAPH_JSON_DELTA_MAX_SOURCE_BYTES - { + if previous_bytes.is_empty() || previous_bytes.len() > GRAPH_JSON_DELTA_MAX_SOURCE_BYTES { return Ok(false); } let Some(nodes_range) = top_level_member_range(previous_bytes, "nodes") else { @@ -1765,6 +1782,29 @@ impl<'a, S: Store + ?Sized> GraphSnapshotReader<'a, S> { }) } + /// Verify every independently bounded immutable object reachable from the + /// selected manifest without materializing the graph. + /// + /// The traversal validates content addresses, object schemas, index keys, + /// branch separators, global key ordering, tree depth, and root entry + /// counts. Memory remains bounded by the decoded-object cache and one + /// branch path even when the logical graph exceeds whole-document reader + /// budgets. + pub fn validate_integrity(&self) -> Result<(), SnapshotError> { + for root in &self.manifest.roots { + let integrity = validate_tree_integrity(self, root.index, &root.digest, 0)?; + if integrity.entries != root.entry_count { + return Err(SnapshotError::Corrupt(format!( + "{} tree contains {} entries but its root declares {}", + root.index.as_str(), + integrity.entries, + root.entry_count + ))); + } + } + Ok(()) + } + /// Read graph-level metadata without materializing file, coverage, or /// diagnostic supplements. pub fn metadata_summary(&self) -> Result { @@ -2845,6 +2885,74 @@ impl<'a, S: Store + ?Sized> GraphSnapshotReader<'a, S> { } } +struct TreeIntegrity { + entries: u64, + first_key: Option>, + last_key: Option>, +} + +fn validate_tree_integrity( + reader: &GraphSnapshotReader<'_, S>, + index: IndexKind, + digest: &str, + depth: usize, +) -> Result { + if depth >= GRAPH_SNAPSHOT_MAX_DEPTH { + return Err(SnapshotError::Limit( + "tree integrity validation exceeded the depth limit".to_owned(), + )); + } + let object = reader.load_tree_object_cached(index, digest)?; + match object.as_ref() { + TreeObject::Leaf { entries, .. } => Ok(TreeIntegrity { + entries: u64::try_from(entries.len()).map_err(|_| { + SnapshotError::Limit("tree leaf entry count does not fit u64".to_owned()) + })?, + first_key: entries.first().map(|entry| entry.key.clone()), + last_key: entries.last().map(|entry| entry.key.clone()), + }), + TreeObject::Branch { children, .. } => { + let mut entry_count = 0_u64; + let mut first_key = None; + let mut last_key: Option> = None; + for child in children { + let child_integrity = + validate_tree_integrity(reader, index, &child.digest, depth.saturating_add(1))?; + let child_first = child_integrity.first_key.ok_or_else(|| { + SnapshotError::Corrupt("tree branch references an empty child".to_owned()) + })?; + if child.first_key != child_first { + return Err(SnapshotError::Corrupt(format!( + "{} tree branch separator does not match its child", + index.as_str() + ))); + } + if last_key + .as_ref() + .is_some_and(|previous| previous >= &child_first) + { + return Err(SnapshotError::Corrupt(format!( + "{} tree child ranges are not strictly ordered", + index.as_str() + ))); + } + first_key.get_or_insert_with(|| child_first.clone()); + last_key = child_integrity.last_key; + entry_count = entry_count + .checked_add(child_integrity.entries) + .ok_or_else(|| { + SnapshotError::Limit("tree entry count exceeds u64".to_owned()) + })?; + } + Ok(TreeIntegrity { + entries: entry_count, + first_key, + last_key, + }) + } + } +} + fn index_entry_id(entry: &TreeEntry, label: &str) -> Result { let segments = decode_key_segments(&entry.key).map_err(SnapshotError::from)?; let id = segments @@ -2930,10 +3038,10 @@ fn digest_canonical_graph( graph: &GraphDocument, clear_generation: bool, ) -> Result<(String, u64), SnapshotError> { - digest_json( - &canonical_graph_document_with_generation(graph, clear_generation), - MAX_GRAPH_BYTES, - ) + digest_json(&canonical_graph_document_with_generation( + graph, + clear_generation, + )) } fn canonical_graph_document_with_generation( @@ -4270,31 +4378,28 @@ fn encode_json(value: &T) -> Result, SnapshotError> { struct DigestWriter { hasher: Sha256, - bytes: usize, - maximum: usize, - exceeded: bool, + bytes: u64, + overflowed: bool, } impl DigestWriter { - fn new(maximum: usize) -> Self { + fn new() -> Self { Self { hasher: Sha256::new(), bytes: 0, - maximum, - exceeded: false, + overflowed: false, } } } impl Write for DigestWriter { fn write(&mut self, buffer: &[u8]) -> std::io::Result { - let next = self.bytes.saturating_add(buffer.len()); - if next > self.maximum { - self.exceeded = true; - return Err(std::io::Error::other( - "serialized value exceeds its byte limit", - )); - } + let buffer_len = u64::try_from(buffer.len()) + .map_err(|_| std::io::Error::other("serialized byte count does not fit u64"))?; + let Some(next) = self.bytes.checked_add(buffer_len) else { + self.overflowed = true; + return Err(std::io::Error::other("serialized byte count exceeds u64")); + }; self.hasher.update(buffer); self.bytes = next; Ok(buffer.len()) @@ -4305,13 +4410,13 @@ impl Write for DigestWriter { } } -fn digest_json(value: &T, maximum: usize) -> Result<(String, u64), SnapshotError> { - let mut writer = DigestWriter::new(maximum); +fn digest_json(value: &T) -> Result<(String, u64), SnapshotError> { + let mut writer = DigestWriter::new(); if let Err(error) = serde_json::to_writer(&mut writer, value) { - if writer.exceeded { - return Err(SnapshotError::Limit(format!( - "canonical graph exceeds the {maximum}-byte limit" - ))); + if writer.overflowed { + return Err(SnapshotError::Limit( + "canonical graph byte count exceeds u64".to_owned(), + )); } return Err(SnapshotError::Encode(error.to_string())); } @@ -4320,8 +4425,7 @@ fn digest_json(value: &T, maximum: usize) -> Result<(String, u64), "canonical graph serialization is empty".to_owned(), )); } - let bytes = writer.bytes as u64; - Ok((format!("{:x}", writer.hasher.finalize()), bytes)) + Ok((format!("{:x}", writer.hasher.finalize()), writer.bytes)) } fn encode_tree_object(value: &TreeObject) -> Result, SnapshotError> { @@ -4481,6 +4585,20 @@ mod tests { use std::sync::Barrier; use std::sync::atomic::{AtomicUsize, Ordering}; + #[test] + fn canonical_digest_counter_has_no_two_gibibyte_cutoff() -> Result<(), std::io::Error> { + let mut writer = DigestWriter::new(); + writer.bytes = (2_u64 * 1024 * 1024 * 1024) + 1; + + writer.write_all(b"x")?; + + assert_eq!(writer.bytes, (2_u64 * 1024 * 1024 * 1024) + 2); + writer.bytes = u64::MAX; + assert!(writer.write_all(b"x").is_err()); + assert!(writer.overflowed); + Ok(()) + } + #[derive(Default)] struct CountingStore { inner: MemoryStore, diff --git a/crates/compass-graph/tests/store_snapshot.rs b/crates/compass-graph/tests/store_snapshot.rs index 11050cb6..224eeb37 100644 --- a/crates/compass-graph/tests/store_snapshot.rs +++ b/crates/compass-graph/tests/store_snapshot.rs @@ -15,7 +15,10 @@ use compass_model::provenance::{ EvidenceConfidence, EvidenceOrigin, OccurrenceRule, Provenance, SourceAnchor, }; use compass_store::SqliteStore; -use compass_store::{Key, MemoryStore, NamespaceId, PartitionKey, Store, WriteCondition}; +use compass_store::{ + Key, KeyRange, MAX_GRAPH_BYTES, MAX_VALUE_BYTES, MemoryStore, NamespaceId, PartitionKey, + ScanLimits, Store, WriteCondition, +}; use sha2::Digest; use tempfile::tempdir; @@ -160,9 +163,12 @@ fn snapshot_is_deterministic_and_reuses_immutable_objects() -> Result<(), Box 0); assert_eq!(first.write_transactions, 2); @@ -246,6 +252,61 @@ fn snapshot_is_deterministic_and_reuses_immutable_objects() -> Result<(), Box Result<(), Box> { + let store = MemoryStore::default(); + let builder = GraphSnapshotBuilder::new(); + let content = builder.prepare_content(&store, &graph())?; + let logical_graph_bytes = (MAX_GRAPH_BYTES as u64).saturating_add(1); + + let prepared = builder.finish_content(&store, content, "a".repeat(64), logical_graph_bytes)?; + builder.activate(&store, &prepared)?; + let reader = GraphSnapshotReader::open_active(&store)?.ok_or("active snapshot missing")?; + + assert_eq!(reader.manifest().graph_bytes, logical_graph_bytes); + assert_eq!( + reader.get_node("a")?.map(|node| node.id), + Some("a".to_owned()) + ); + let object_page = store.scan( + &NamespaceId::graph(), + &PartitionKey::new("graph-snapshot/objects")?, + &KeyRange::default(), + ScanLimits::default(), + None, + )?; + assert!(object_page.next.is_none()); + assert!(!object_page.entries.is_empty()); + assert!( + object_page + .entries + .iter() + .all(|entry| entry.value.len() <= MAX_VALUE_BYTES) + ); + Ok(()) +} + +#[test] +fn segmented_manifest_counts_are_not_limited_by_materialized_read_budgets() +-> Result<(), Box> { + let store = MemoryStore::default(); + let mut manifest = GraphSnapshotBuilder::new() + .prepare(&store, &graph())? + .manifest; + let logical_node_count = (compass_graph::GRAPH_SNAPSHOT_MAX_ITEMS as u64).saturating_add(1); + manifest.node_count = logical_node_count; + manifest + .roots + .iter_mut() + .find(|root| root.index == IndexKind::Nodes) + .ok_or("node root missing")? + .entry_count = logical_node_count; + + manifest.validate()?; + Ok(()) +} + #[test] fn nodes_for_terms_matches_diacritic_normalized_queries() -> Result<(), Box> { let store = MemoryStore::default(); @@ -845,6 +906,7 @@ fn missing_or_tampered_objects_fail_closed() -> Result<(), Box> { let prepared = builder.prepare(&store, &graph())?; let selector = builder.activate(&store, &prepared)?; let reader = GraphSnapshotReader::open_selector(&store, selector)?; + reader.validate_integrity()?; let root = reader .manifest() .roots @@ -861,6 +923,11 @@ fn missing_or_tampered_objects_fail_closed() -> Result<(), Box> { b"corrupt", WriteCondition::Any, )?; + let reader = GraphSnapshotReader::open_active(&store)?.ok_or("active snapshot missing")?; + assert!(matches!( + reader.validate_integrity(), + Err(SnapshotError::Corrupt(_)) + )); assert!(matches!( reader.get_node("a"), Err(SnapshotError::Corrupt(_)) diff --git a/crates/compass-store/src/lib.rs b/crates/compass-store/src/lib.rs index 937a8388..cd5e700a 100644 --- a/crates/compass-store/src/lib.rs +++ b/crates/compass-store/src/lib.rs @@ -41,13 +41,12 @@ pub const MAX_SCAN_ITEMS: usize = 1_000; pub const MAX_SCAN_BYTES: usize = 1024 * 1024; pub const MAX_IMMUTABLE_BATCH_ITEMS: usize = 1_024; pub const MAX_IMMUTABLE_BATCH_BYTES: usize = 16 * 1024 * 1024; -/// Maximum canonical graph payload accepted by the bounded store snapshot. +/// Maximum payload accepted by the legacy monolithic snapshot compatibility API. /// -/// The portable in-memory JSON readers intentionally keep their independent -/// 1 GiB cap. The local store is the bounded large-graph path and therefore -/// accepts up to 2 GiB while serving records through indexed scans instead of -/// materializing the whole document. The limit is still finite so malformed -/// or hostile snapshots cannot request unbounded allocation. +/// The current graph-index snapshot stores independently bounded immutable tree +/// objects and has no aggregate payload limit. This cap remains on the legacy +/// `publish_snapshot`/`read_snapshot` path because that API materializes the +/// canonical payload in one allocation. pub const MAX_GRAPH_BYTES: usize = 2 * 1024 * 1024 * 1024; const GRAPH_NAMESPACE: &[u8] = b"compass.current.graph.v1"; const CATALOG_PARTITION: &[u8] = b"catalog"; diff --git a/docs/design/compass-store.md b/docs/design/compass-store.md index b086dad5..383cca91 100644 --- a/docs/design/compass-store.md +++ b/docs/design/compass-store.md @@ -669,7 +669,12 @@ children. Completed leaves are encoded once and oversized leaves split deterministically; construction does not repeatedly clone and serialize partial leaves. Content-addressed writes use bounded `put_immutable_batch`, so repeated builds reuse byte-identical leaves and branches without one existence read or -durable transaction per object. The reader verifies the object +durable transaction per object. There is no aggregate canonical-payload or +record-count limit on this graph-index layout: manifests use `u64` counts and +cross-check node and edge totals against their tree roots. Every immutable +object remains at most 256 KiB, every immutable write batch remains at most +16 MiB, and reads retain independent item, byte, object, and depth budgets. +The reader verifies the object digest, schema, index kind, ordering, root selection, and graph validation before returning a record. Point reads and scans enforce item, byte, object, and depth limits; corruption is never converted into an empty result. @@ -690,6 +695,14 @@ entries in bounded transactions. A malformed or stale store fails the unpublished build and leaves the previous snapshot and its JSON engine readable. +Operational status, validation, backup, and restore do not reopen the portable +JSON artifact through its whole-document reader. They stream graph and store +digests through a fixed 1 MiB buffer, bind the graph byte count and digest to +the selected manifest, and traverse every reachable tree object. That +traversal validates content addresses, schemas, key ordering, branch +separators, depth, and root entry counts while retaining only the bounded +decoded-object cache and current branch path. + ### Structural sharing and incremental update An update compares the new canonical index streams with the prior roots. Tree diff --git a/docs/reference/outputs.md b/docs/reference/outputs.md index c45da914..14fedd86 100644 --- a/docs/reference/outputs.md +++ b/docs/reference/outputs.md @@ -192,6 +192,19 @@ Operators can set `COMPASS_MAX_GRAPH_BYTES` to an explicit byte count or `MB`/`GB`; raising it also raises the memory exposure of JSON decoding and indexing. +That whole-JSON reader cap is separate from the current SQLite graph-index +snapshot used by `--store sqlite`. The graph-index has no aggregate canonical +payload or record-count limit: it publishes content-addressed tree objects of +at most 256 KiB through write batches of at most 16 MiB, and bounds each point +or range query independently. Consumers that request a whole-graph export can +still encounter the materialized-read record budget and should use indexed +queries for substantially larger repositories. + +`compass store status`, `validate`, `backup`, and `restore` also remain on the +large-graph path. They stream file digests through fixed-size buffers and +validate the selected manifest plus every reachable immutable tree object; +they do not require a `COMPASS_MAX_GRAPH_BYTES` override. + ### Partial publication diagnostics A successful build can publish a strictly valid partial graph after