feat(theta): compact sketch serde v3/v4

[ZENOTME]

This refers #30, including:

• implement CompactThetaSketch
• serialize CompactThetaSketch for uncompress/compress
• deserialize CompactThetaSketch for v1,v2,v3,v4

[ZENOTME]

cc @tisonkun @PsiACE @notfilippo @Xuanwo

[notfilippo]

I plan to take a look at it during the weekend! Thanks

[ZENOTME]

Have rebase with #91 and it's ready to review. cc @notfilippo @tisonkun

[Copilot]

Pull request overview

This pull request implements serialization and deserialization support for Theta Sketch's compact representation, adding support for Apache DataSketches binary format versions 1-4. This enables interoperability with Java and C++ implementations and supports Iceberg's Puffin statistics format for storing NDV (number of distinct values) metadata.

Changes:

• Adds CompactThetaSketch struct for immutable, serialization-friendly theta sketch representation
• Implements uncompressed (v3) and compressed (v4) serialization with delta encoding and bit-packing
• Implements deserialization for legacy versions v1, v2, and modern versions v3, v4 with comprehensive validation

Reviewed changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
datasketches/src/theta/sketch.rs	Adds CompactThetaSketch struct with serialization/deserialization methods, compact() conversion, and versioned deserializers
datasketches/src/theta/serialization.rs	Defines binary format constants for flags and version numbers
datasketches/src/theta/mod.rs	Exports CompactThetaSketch and DEFAULT_UPDATE_SEED for public API
datasketches/src/theta/hash_table.rs	Adds hash_seed() accessor method for seed hash computation
datasketches/src/hash/mod.rs	Changes DEFAULT_UPDATE_SEED visibility from pub(crate) to pub for deserialization API
datasketches/src/error.rs	Updates invalid_preamble_longs to accept multiple expected values via AsRef<[u8]>
datasketches/src/codec/family.rs	Adds Family::THETA constant with id=3 and preamble size range 1-3
datasketches/src/tdigest/sketch.rs	Updates invalid_preamble_longs call site to use std::slice::from_ref
datasketches/src/frequencies/sketch.rs	Updates invalid_preamble_longs call sites to use std::slice::from_ref
datasketches/src/cpc/sketch.rs	Updates invalid_preamble_longs call site to use std::slice::from_ref
datasketches/src/countmin/sketch.rs	Updates invalid_preamble_longs call site to use std::slice::from_ref
datasketches/tests/theta_serialization_test.rs	Adds Java/C++ compatibility tests for both compressed and uncompressed formats

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ZENOTME]

Let me change it to bit packer for more natural use and readability.

[tisonkun]

cc @ZENOTME I pushed 0db7843 to fine tune some methods and reduce duplicate.

[tisonkun]

@ZENOTME We'd better define constants for what 1,2,3 here mean.

[ZENOTME]

I'm thinking maybe make it as a type is a clearer way:

struct PremableV2(u8);

impl PremableV2 {
  bool is_estimate()
  ...
}

struct PremableV3(u8);

impl PremableV3 {
  bool is_estimate()
  ...
}

[ZENOTME]

Actually, I noticed that it's different when checking for serialization/deserialization of the same field, e.g., theta, as follows:

// serialize
if (this->is_estimation_mode()) write(os, this->theta_); // means that preamble = 3

// deserialize
if (preamble_longs > 2) theta = read<uint64_t>(is);

Change like the following may be better and more consistent:

// serialize
if (this->is_estimation_mode()) write(os, this->theta_); // means that preamble = 3

// deserialize
if (preamble_longs.is_estimate()) theta = read<uint64_t>(is);

[tisonkun]

Thanks for your contribution! LGTM.

I'm waiting a few days for @notfilippo and @leerho review. But I'd merge this patch next week if there are no more concerns.