Skip to content

Commit d090eb3

Browse files
committed
Add Table Properties for Encryption Configuration
This PR introduces table-level encryption properties to enable configuration of encryption settings for Iceberg tables. These properties lay the groundwork for future encryption implementation while maintaining compatibility with the Java implementation's property names and structure. Table-level encryption is a critical security feature in Apache Iceberg's Java implementation. To support encryption in iceberg-rust and ensure interoperability between Java and Rust implementations, we need to start by adding the configuration properties that control encryption behavior. This PR adds the property definitions and parsing logic without implementing the actual encryption, keeping the change focused and reviewable. **Modified:** `crates/iceberg/src/spec/table_properties.rs` Added encryption-related properties to the `TableProperties` struct: - `PROPERTY_ENCRYPTION_KEY_ID` (`"encryption.key-id"`) - Master key ID for encrypting data encryption keys - `PROPERTY_ENCRYPTION_DEK_LENGTH` (`"encryption.data-key-length"`) - Data encryption key length (default: 16 bytes) - `PROPERTY_ENCRYPTION_AAD_LENGTH` (`"encryption.aad-length"`) - AAD prefix length for GCM (default: 16 bytes) - `PROPERTY_ENCRYPTION_KMS_TYPE` (`"encryption.kms-type"`) - KMS type (e.g., "aws", "gcp", "azure") All `Option<T>` as encryption is optional: - `encryption_key_id: Option<String>` - `encryption_dek_length: Option<usize>` - `encryption_aad_length: Option<usize>` - `encryption_kms_type: Option<String>` Extended `TryFrom<&HashMap<String, String>>` implementation to parse encryption properties Property names match exactly with Java's implementation: - Java: `TableProperties.ENCRYPTION_TABLE_KEY` → Rust: `PROPERTY_ENCRYPTION_KEY_ID` - Java: `TableProperties.ENCRYPTION_DEK_LENGTH` → Rust: `PROPERTY_ENCRYPTION_DEK_LENGTH` - Java: `CatalogProperties.ENCRYPTION_KMS_TYPE` → Rust: `PROPERTY_ENCRYPTION_KMS_TYPE` **Note:** Java's `ENCRYPTION_KMS_IMPL` property (for custom KMS implementations via reflection) is intentionally not included since Rust doesn't support runtime reflection. KMS implementations will be selected based on the `encryption.kms-type` property with compiled-in implementations. Added comprehensive test coverage: 1. `test_table_properties_default`: Verifies encryption properties are None by default 2. `test_encryption_properties_valid`: Tests parsing all encryption properties with valid values 3. `test_encryption_properties_partial`: Tests partial encryption configuration 4. `test_encryption_properties_invalid_numeric`: Verifies invalid numeric values are handled gracefully (parsed as None) 5. `test_encryption_properties_with_other_properties`: Tests encryption properties alongside existing table properties All tests pass: ``` running 7 tests test spec::table_properties::tests::test_table_properties_default ... ok test spec::table_properties::tests::test_encryption_properties_partial ... ok test spec::table_properties::tests::test_encryption_properties_invalid_numeric ... ok test spec::table_properties::tests::test_encryption_properties_valid ... ok test spec::table_properties::tests::test_encryption_properties_with_other_properties ... ok test spec::table_properties::tests::test_table_properties_valid ... ok test spec::table_properties::tests::test_table_properties_invalid ... ok ``` 1. **Optional Fields**: All encryption properties are `Option<T>` since encryption is an optional feature 2. **Silent Failure for Invalid Numbers**: Invalid numeric values for `dek_length` and `aad_length` are parsed as None rather than failing, matching the pattern for optional properties 3. **No Validation**: This PR doesn't validate property values (e.g., valid key lengths), leaving that for the encryption implementation 4. **No Custom KMS**: Omitted `encryption.kms-impl` property since Rust lacks reflection - KMS type selection will use `encryption.kms-type` with a factory pattern 5. **Independent PR**: No dependencies on other encryption code, can be merged independently This PR is part of a series to implement encryption support: - ✅ PR 1: Core encryption primitives (AES-GCM operations) - ✅ PR 2: Table properties for encryption (this PR) - PR 3: Key management interfaces - PR 4: EncryptionManager implementation - PR 5: Native Parquet encryption support - PR 6: Integration with Table and FileIO
1 parent b05a675 commit d090eb3

1 file changed

Lines changed: 127 additions & 0 deletions

File tree

crates/iceberg/src/spec/table_properties.rs

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,10 @@ pub struct TableProperties {
5151
pub write_target_file_size_bytes: usize,
5252
/// Whether to use `FanoutWriter` for partitioned tables.
5353
pub write_datafusion_fanout_enabled: bool,
54+
/// Master key ID for encryption. When set, all data and manifest files will be encrypted.
55+
pub encryption_key_id: Option<String>,
56+
/// Length of data encryption keys in bytes.
57+
pub encryption_dek_length: Option<usize>,
5458
}
5559

5660
impl TableProperties {
@@ -144,6 +148,29 @@ impl TableProperties {
144148
pub const PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED: &str = "write.datafusion.fanout.enabled";
145149
/// Default value for fanout writer enabled
146150
pub const PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED_DEFAULT: bool = true;
151+
152+
// Encryption properties
153+
154+
/// Master key ID for encrypting data encryption keys.
155+
///
156+
/// When set, enables table-level encryption where all data and manifest
157+
/// files are encrypted using data encryption keys (DEKs) that are
158+
/// themselves encrypted with this master key.
159+
pub const PROPERTY_ENCRYPTION_KEY_ID: &str = "encryption.key-id";
160+
161+
/// Length of data encryption keys in bytes.
162+
///
163+
/// Controls the key size for AES encryption. Common values are 16 (AES-128)
164+
/// which is the only encryption method currently supported in the parquet
165+
pub const PROPERTY_ENCRYPTION_DEK_LENGTH: &str = "encryption.data-key-length";
166+
/// Default length for data encryption keys (16 bytes = AES-128).
167+
pub const PROPERTY_ENCRYPTION_DEK_LENGTH_DEFAULT: usize = 16;
168+
169+
/// Default AAD (Additional Authenticated Data) length for GCM encryption.
170+
///
171+
/// AAD provides additional context for authenticated encryption modes like AES-GCM.
172+
/// This is hardcoded to 16 bytes for Java compatibility and is not configurable.
173+
pub const PROPERTY_ENCRYPTION_AAD_LENGTH_DEFAULT: usize = 16;
147174
}
148175

149176
impl TryFrom<&HashMap<String, String>> for TableProperties {
@@ -187,6 +214,13 @@ impl TryFrom<&HashMap<String, String>> for TableProperties {
187214
TableProperties::PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED,
188215
TableProperties::PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED_DEFAULT,
189216
)?,
217+
// Encryption properties - all optional
218+
encryption_key_id: props
219+
.get(TableProperties::PROPERTY_ENCRYPTION_KEY_ID)
220+
.cloned(),
221+
encryption_dek_length: props
222+
.get(TableProperties::PROPERTY_ENCRYPTION_DEK_LENGTH)
223+
.and_then(|v| v.parse().ok()),
190224
})
191225
}
192226
}
@@ -219,6 +253,9 @@ mod tests {
219253
table_properties.write_target_file_size_bytes,
220254
TableProperties::PROPERTY_WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT
221255
);
256+
// Encryption properties should be None by default
257+
assert_eq!(table_properties.encryption_key_id, None);
258+
assert_eq!(table_properties.encryption_dek_length, None);
222259
}
223260

224261
#[test]
@@ -293,4 +330,94 @@ mod tests {
293330
"Invalid value for write.target-file-size-bytes: invalid digit found in string"
294331
));
295332
}
333+
334+
#[test]
335+
fn test_encryption_properties_valid() {
336+
let props = HashMap::from([
337+
(
338+
TableProperties::PROPERTY_ENCRYPTION_KEY_ID.to_string(),
339+
"test-key-123".to_string(),
340+
),
341+
(
342+
TableProperties::PROPERTY_ENCRYPTION_DEK_LENGTH.to_string(),
343+
"32".to_string(),
344+
),
345+
]);
346+
let table_properties = TableProperties::try_from(&props).unwrap();
347+
assert_eq!(
348+
table_properties.encryption_key_id,
349+
Some("test-key-123".to_string())
350+
);
351+
assert_eq!(table_properties.encryption_dek_length, Some(32));
352+
}
353+
354+
#[test]
355+
fn test_encryption_properties_partial() {
356+
// Test with only the key ID set, not the DEK length
357+
let props = HashMap::from([(
358+
TableProperties::PROPERTY_ENCRYPTION_KEY_ID.to_string(),
359+
"my-master-key".to_string(),
360+
)]);
361+
let table_properties = TableProperties::try_from(&props).unwrap();
362+
assert_eq!(
363+
table_properties.encryption_key_id,
364+
Some("my-master-key".to_string())
365+
);
366+
assert_eq!(table_properties.encryption_dek_length, None);
367+
}
368+
369+
#[test]
370+
fn test_encryption_properties_invalid_numeric() {
371+
// Test that invalid numeric values are silently ignored (parsed as None)
372+
let props = HashMap::from([
373+
(
374+
TableProperties::PROPERTY_ENCRYPTION_KEY_ID.to_string(),
375+
"key-456".to_string(),
376+
),
377+
(
378+
TableProperties::PROPERTY_ENCRYPTION_DEK_LENGTH.to_string(),
379+
"not-a-number".to_string(),
380+
),
381+
]);
382+
let table_properties = TableProperties::try_from(&props).unwrap();
383+
assert_eq!(
384+
table_properties.encryption_key_id,
385+
Some("key-456".to_string())
386+
);
387+
// Invalid numeric values should be parsed as None
388+
assert_eq!(table_properties.encryption_dek_length, None);
389+
}
390+
391+
#[test]
392+
fn test_encryption_properties_with_other_properties() {
393+
// Test encryption properties alongside other table properties
394+
let props = HashMap::from([
395+
(
396+
TableProperties::PROPERTY_COMMIT_NUM_RETRIES.to_string(),
397+
"8".to_string(),
398+
),
399+
(
400+
TableProperties::PROPERTY_DEFAULT_FILE_FORMAT.to_string(),
401+
"orc".to_string(),
402+
),
403+
(
404+
TableProperties::PROPERTY_ENCRYPTION_KEY_ID.to_string(),
405+
"combined-test-key".to_string(),
406+
),
407+
(
408+
TableProperties::PROPERTY_ENCRYPTION_DEK_LENGTH.to_string(),
409+
"16".to_string(),
410+
),
411+
]);
412+
let table_properties = TableProperties::try_from(&props).unwrap();
413+
// Check regular properties
414+
assert_eq!(table_properties.commit_num_retries, 8);
415+
assert_eq!(table_properties.write_format_default, "orc".to_string());
416+
// Check encryption properties
417+
assert_eq!(
418+
table_properties.encryption_key_id,
419+
Some("combined-test-key".to_string())
420+
);
421+
assert_eq!(table_properties.encryption_dek_length, Some(16));
422+
}
296423
}

0 commit comments

Comments
 (0)