Skip to content

feat(cpp): refactor dependency sourcing and extend codec compatibility - #905

Merged
ColinLeeo merged 22 commits into
apache:developfrom
ColinLeeo:feature/cpp-dependency-source
Aug 18, 2026
Merged

feat(cpp): refactor dependency sourcing and extend codec compatibility#905
ColinLeeo merged 22 commits into
apache:developfrom
ColinLeeo:feature/cpp-dependency-source

Conversation

@ColinLeeo

@ColinLeeo ColinLeeo commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Background

This PR implements the community-approved plan for managing TsFile C++ third-party dependencies:

Previously, several complete or trimmed third-party source trees were committed directly under cpp/third_party. Their origins, versions, local modifications, and license boundaries were difficult to audit and maintain.

What changes are proposed?

Dependency source management

This PR introduces the global CMake option TSFILE_DEPENDENCY_SOURCE, which accepts the following values:

  • AUTO: Prefer a compatible system package and fall back to a verified upstream source archive.
  • SYSTEM: Require compatible externally installed dependencies and fail clearly when they are unavailable.
  • BUNDLED: Download or reuse pinned upstream source archives and build them inside the build tree.

BUNDLED does not mean that third-party source code is committed to the TsFile repository.

Offline builds are supported through TSFILE_DEPENDENCY_OFFLINE=ON and TSFILE_DEPENDENCY_CACHE=/path/to/cache.

All downloaded archives are pinned by version or commit and verified with SHA-256 before extraction.

Migrated dependencies

Dependency Version License
ANTLR4 C++ Runtime 4.9.3 BSD-3-Clause with MIT notices
utf8cpp 3.1.1 Boost Software License 1.0
Snappy 1.2.2 BSD-3-Clause
LZ4 1.9.4 BSD-2-Clause
lzokay commit 5cb18da MIT
SIMDe 0.8.4-rc3 MIT
zlib 1.3.1 zlib License
Zstandard 1.5.7 BSD-3-Clause option
liblzma / XZ Utils 5.8.3 0BSD for the selected liblzma sources

The corresponding committed source trees are removed from cpp/third_party.

ANTLR4 compatibility changes are applied as a reviewed and hash-verified extraction-time patch. Other downloaded upstream sources are not locally modified.

XZ Utils is a mixed-license archive. The build selects only the 0BSD-licensed liblzma sources and verifies that GPL/LGPL components are not compiled, linked, installed, or redistributed.

Codec support

This PR also adds C++ support for existing TsFile codecs:

  • Zstandard compression
  • LZMA2 compression
  • CHIMP encoding
  • RLBE encoding
  • CAMEL encoding

The codec identifiers are exposed through the C wrapper and Python bindings where applicable.

Licensing and documentation

  • Update the root LICENSE and NOTICE.
  • Document dependency origins, versions, archive hashes, trimming scope, local modifications, and license boundaries.
  • Document AUTO, SYSTEM, BUNDLED, and offline build usage.
  • Validate the GoogleTest archive with SHA-256.

CI and compatibility coverage

A dedicated dependency-source workflow now validates:

  • A fresh BUNDLED build from verified archives.
  • A BUNDLED build using a populated cache with network downloads disabled.
  • A SYSTEM build using independently installed dependencies.

The dependency CMake tests cover compatible, missing, outdated, and incompatible system packages. Nested tests inherit the active CMake generator, platform, and toolset across MinGW, MSVC, Ninja, and Makefile builds.

Cross-language compatibility tests generate fixtures with both Java and C++ and verify that the other implementation can read them, including the newly supported encodings and compression methods.

Validation

The following checks pass on PR #905:

  • Linux, macOS, MinGW, and MSVC C++ unit tests
  • Python tests on Linux, macOS, Windows, and MSVC
  • Java and C++ codec compatibility tests
  • AUTO, SYSTEM, BUNDLED, and offline dependency builds
  • MSVC default and v141 toolsets, with and without ASan
  • CodeQL and cppcheck
  • Spotless and Apache RAT

@ColinLeeo ColinLeeo changed the title Feature/cpp dependency source feat(cpp): refactor dependency sourcing and extend codec compatibility Aug 17, 2026
@ColinLeeo
ColinLeeo requested a lite review from Copilot August 17, 2026 02:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

@ColinLeeo
ColinLeeo requested a lite review from Copilot August 17, 2026 02:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.


if (ENABLE_LZMA2)
if (TSFILE_LIBLZMA_SOURCE STREQUAL "BUNDLED")
if (CMAKE_VERSION VERSION_LESS "3.20")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmake_minimum_required is 3.11, but default ENABLE_LZMA2=ON + BUNDLED liblzma fatals on CMake < 3.20. Please bump the minimum to 3.20, or disable BUNDLED LZMA2 below 3.20.

Comment thread cpp/README.md Outdated
`antlr4-4.9.3.tar.gz`, `utfcpp-v3.1.1.tar.gz`, `snappy-1.2.2.tar.gz`,
`lz4-v1.9.4.tar.gz`,
`lzokay-5cb18da508cc4d3ec41bc04dccdeef9c5ffedfb2.tar.gz`,
`simde-v0.8.4-rc3.tar.gz`, and `zlib-v1.3.1.tar.gz` in a persistent cache,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Offline/BUNDLED docs omit zstd-v1.5.7.tar.gz and xz-5.8.3.tar.gz (and the matching -DTSFILE_ZSTD_ARCHIVE / -DTSFILE_LIBLZMA_ARCHIVE flags), but ENABLE_ZSTD and ENABLE_LZMA2 default to ON. Following this README with TSFILE_DEPENDENCY_OFFLINE=ON will fail at configure. Please align with cpp/third_party/README.md.

@ColinLeeo
ColinLeeo merged commit d4c3c94 into apache:develop Aug 18, 2026
62 of 64 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants