feat: Add InfluxDB connector implementation by ryerraguntla · Pull Request #2801 · apache/iggy

ryerraguntla · 2026-02-24T02:47:23Z

Overview

This PR introduces an initial implementation of an InfluxDB connector for Apache Iggy.

Changes

Initial version of the InfluxDB connector implementation
Integration with the existing Redshift connector developed by another contributor

Notes

Extends Iggy's streaming capabilities to support InfluxDB as a data destination

Implements Issue #2540 - Redshift Sink Connector with S3 staging support. Features: - S3 staging with automatic CSV file upload - Redshift COPY command execution via PostgreSQL wire protocol - IAM role authentication (recommended) or access key credentials - Configurable batch size and compression (gzip, lzop, bzip2, zstd) - Automatic table creation with customizable schema - Retry logic with exponential backoff for transient failures - Automatic cleanup of staged S3 files Configuration options: - connection_string: Redshift cluster connection URL - target_table: Destination table name - iam_role: IAM role ARN for S3 access (recommended) - s3_bucket/s3_region/s3_prefix: S3 staging location - batch_size: Messages per batch (default: 10000) - compression: COPY compression format - delete_staged_files: Auto-cleanup toggle (default: true) - auto_create_table: Create table if missing (default: true) Closes #2540

- Fix markdown lint issues in README.md (table formatting, blank lines, code fence language) - Fix trailing newline in Cargo.toml - Apply TOML formatting via taplo - Add missing dependencies to DEPENDENCIES.md (rust-s3, rxml, rxml_validation, static_assertions)

- Add Redshift sink integration test using PostgreSQL (Redshift-compatible) and LocalStack for S3 - Add s3_endpoint config option to support custom endpoints (LocalStack, MinIO) - Add path-style S3 access for custom endpoints - Add localstack feature to testcontainers-modules - Create test configuration files for Redshift connector

- Add s3_endpoint: None to test_config() in lib.rs (fixes E0063) - Add endpoint parameter to S3Uploader tests in s3.rs - Fix formatting for long line in init_s3_uploader() - Add iggy_connector_redshift_sink to DEPENDENCIES.md - Add maybe-async, md5, minidom to DEPENDENCIES.md

Critical fixes: - Change Rust edition from 2024 to 2021 in Cargo.toml - Fix S3 cleanup to happen regardless of COPY result (prevents orphaned files) Moderate fixes: - Remove zstd from valid compression options (not supported by Redshift) - Update README to remove zstd from compression list - Handle bucket creation error in integration tests with expect() - Log JSON serialization errors instead of silent unwrap_or_default() Performance: - Cache escaped quote string to avoid repeated format! allocations Windows compatibility (for local testing): - Add #[cfg(unix)] conditionals for Unix-specific code in sender/mod.rs

Fixes clippy warning about unused 'runtime' field in test setup struct. The runtime field is kept for future test expansion.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

- Changed CONFIG_ to PLUGIN_CONFIG_ for plugin configuration fields - Changed TOPICS_0 to TOPICS with proper JSON array format - Added CONSUMER_GROUP environment variable

… into ryerraguntla/issue2540

Update to the latest

Updated with the master versions

with master versions

Syncing with master versions

Merging with redshift changes

added new mods and made the fn pub

Merged the dependencies

Removed duplicates.

Merging the dependencies and removing unused imports

feat(connectors): issue2540 - implement Redshift Sink Connector with S3 staging.

feat(connectors): issue2540 - implement Redshift Sink Connector with S3 staging updating the iggy_common crates version