Skip to content

CLAUDE.md

Latest commit

 

History

History
90 lines (65 loc) · 4.46 KB

CLAUDE.md

File metadata and controls

90 lines (65 loc) · 4.46 KB

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. Most of the commands are for stuff inside iceberg_rust_ffi folder, with the exception of the RustyIceberg.jl section below.

Common Development Commands

Building (in the iceberg_rust_ffi folder)

  • cargo build - Build the Rust FFI library (debug)
  • cargo build --release - Build the Rust FFI library (production)

Testing (in the root folder)

  • make run-containers - Start Docker containers for S3 testing
  • make test - Run the Julia tests (requires build and containers)
  • make stop-containers - Stop Docker containers

Code Quality (in the iceberg_rust_ffi folder)

  • cargo fmt - Format Rust code
  • cargo clippy - Run Rust linter
  • cargo check - Quick check for Rust compilation errors

Cleanup

  • make clean - Clean build artifacts (but keep target directory)
  • make clean-all - Clean everything including target directory

Architecture Overview

This project provides a Foreign Function Interface (FFI) for Apache Iceberg, allowing C programs (and other languages through C bindings) to access Iceberg tables stored in object storage systems like S3. The majority of the infrastucture relies on object_store_ffi crate. If you don't have access to that crate's code locally, access it at this URL.

Key Components

Rust Library (src/lib.rs)

  • Core FFI Implementation: Exposes Iceberg functionality through C-compatible functions
  • Async Runtime Integration: Uses Tokio for async operations with object_store_ffi for callback handling. Async operations rely on export_runtime_op! macro, which has a sync block, which is a builder function, where all deserialization and conversion is done. Then the result of that is passed to an async block. Each parameter has to implement Send trait, in order to be passed to the async block
  • Julia Integration: Conditional compilation features for Julia interop (julia feature flag)
  • Memory Management: Safe FFI patterns with proper cleanup functions

FFI Design Patterns

Async Operations with Callbacks

The FFI uses an async callback pattern where:

  1. C calls an async function with a response structure
  2. Rust spawns the operation and returns immediately
  3. When complete, Rust invokes a callback to signal completion
  4. C polls or waits for completion, then checks the response structure

Memory Management

  • Owned Pointers: Rust allocates, C receives opaque pointers
  • Cleanup Functions: Every allocated resource has a corresponding _free function
  • Error Handling: Errors are returned via response structures with allocated error strings

Context and Cancellation

  • Operations return a context pointer that can be used for cancellation
  • iceberg_cancel_context and iceberg_destroy_context functions manage operation lifecycle

S3 Configuration

The test expects AWS S3 credentials through environment variables:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • AWS_REGION or AWS_DEFAULT_REGION
  • AWS_ENDPOINT_URL (for MinIO or custom S3-compatible storage)

Use the .env file or export variables directly. The test is designed to fail with permission errors when S3 paths are inaccessible, which confirms the API is working correctly.

Build System

Cargo Features

  • Default features: ["julia"]
  • julia feature: Enables Julia thread adoption and GC integration

RustyIceberg.jl

Whenever making API changes in the iceberg_rust_ffi, the corresponding changes should be made in its parent folder. The parent folder is home for Julia package, which provides Julia bindings on top of the FFI. Once changes are made there, they should be tested by running make test. If error is related to missing minio/s3 or Iceberg REST catalog server etc., suggest to user to run make run-containers.

Development Notes

Working with FFI

  • Always check for null pointers in C code before dereferencing
  • Use the provided _free functions to avoid memory leaks
  • Error messages are allocated strings that must be freed with iceberg_destroy_cstring

Testing Changes

Run make test after making changes to verify the FFI still works.

Object Store Integration

This crate depends on object_store_ffi for async runtime management and callback handling. The integration provides:

  • Cross-platform async runtime setup
  • Callback infrastructure for async operations
  • Context management for cancellation support