fix(release): backport reproducible source archives to 0.3

.github/workflows/release_candidate.yaml

@@ -78,9 +78,13 @@ jobs:
             --output-dir release/ci
 
       - name: Audit source archive
+        env:
+          # The creator must ignore caller-provided gzip defaults.
+          GZIP: "-9"
         run: |
           scripts/releasing/verify_release_candidate.sh \
             --allow-unsigned \
+            --git-ref HEAD \
             --skip-build \
             "release/ci/apache-paimon-cpp-${RELEASE_VERSION}-src.tgz"
 

scripts/releasing/README.md

@@ -34,16 +34,25 @@ Before starting a release:
 - obtain an ASF code-signing key, publish it through the ASF account system,
   and make sure it is present in
   [Paimon KEYS](https://downloads.apache.org/paimon/KEYS);
-- install `git`, `gpg`, `svn`, `gh`, `python3`, `curl` or `wget`, Java, CMake,
-  Ninja, and the toolchain needed by `ci/scripts/build_paimon.sh` (Java is
-  required by Apache RAT);
+- install `git`, GNU `gzip`, `gpg`, `svn`, `gh`, `python3`, `curl` or `wget`,
+  Java, CMake, Ninja, and the toolchain needed by
+  `ci/scripts/build_paimon.sh` (Java is required by Apache RAT);
 - authenticate `gh` with access to read GitHub Actions runs in
   `apache/paimon-cpp`;
 - make sure the Apache Git remote points directly to
   `apache/paimon-cpp`;
 - prepare and merge a release-preparation PR that updates the release notes and
   all version metadata, and passes the normal and release-candidate workflows.
 
+The source archive uses GNU gzip with fixed options so that macOS and Linux
+produce the same bytes. On macOS, install Homebrew gzip and either put it first
+on `PATH` or select it explicitly:
+
+```bash
+brew install gzip
+export PAIMON_GZIP="$(brew --prefix gzip)/bin/gzip"
+```
+
 For example, update all version locations and review the diff:
 
 ```bash
@@ -80,10 +89,13 @@ The release scripts use `vVERSION-rcRC` for release-candidate tags and
 
 Start from the exact clean commit approved for the candidate. Before publishing,
 the wrapper fetches the release branch and requires `HEAD` to be contained in
-its current history. It then creates and verifies a signed RC tag, creates the
-source archive and its checksum/signature, performs the full source-release
-verification, pushes the tag, waits for the tag-triggered release-candidate
-workflow to succeed, and imports the artifacts into ASF `dist/dev`:
+its current history. It then creates and verifies a signed RC tag, pushes the
+tag, and waits for the tag-triggered release-candidate workflow. That workflow
+creates the canonical source archive and checksum, builds and tests the same
+archive with GCC and Clang, and uploads it as a workflow artifact. The wrapper
+downloads those exact bytes, confirms that they are reproducible from the tag,
+signs the archive locally, performs the full source-release verification, and
+imports the three files into ASF `dist/dev`:
 
 ```bash
 scripts/releasing/release_rc.sh \
@@ -96,12 +108,13 @@ scripts/releasing/release_rc.sh \
 The release branch defaults to `main`; use `--release-branch NAME` for a
 maintenance release from another Apache branch.
 
-Use `--prepare-only` to create and verify artifacts without pushing the tag or
-uploading to ASF infrastructure. This local-only mode does not require `HEAD`
-to match the remote release branch. Use `--dry-run` to print identifiers
-without making changes. A resumed run reuses an existing local tag or complete
-artifact set only after validating it. A prepare-only run does not print a vote
-email and must not be used to start a vote.
+Use `--prepare-only` to create and verify preview artifacts without pushing the
+tag or uploading to ASF infrastructure. This local-only mode does not require
+`HEAD` to match the remote release branch. A preview is not authoritative: a
+published run downloads the workflow artifact and rejects an existing local
+archive if its bytes differ. Use `--dry-run` to print identifiers without
+making changes. A prepare-only run does not print a vote email and must not be
+used to start a vote.
 
 The candidate directory contains:
 
@@ -164,13 +177,16 @@ The verifier checks:
 - installation plus compilation and execution of an external CMake consumer.
 
 Pass `--git-ref v0.3.0-rc1` when the Git repository is available to regenerate
-the archive from the signed tag and compare it byte-for-byte.
+the archive from the signed tag with GNU gzip and compare it byte-for-byte.
+This check requires GNU gzip on every platform; it intentionally rejects the
+macOS system gzip instead of treating different compressed bytes as equivalent.
 
 `--allow-unsigned`, `--skip-rat`, `--skip-build`, and `--skip-install` exist for
 CI or local development of the release process. They are not a substitute for
 the corresponding checks when voting. The release-candidate workflow creates
-an unsigned archive for deterministic CI validation; official artifacts must
-always be signed by the release manager.
+the unsigned canonical archive for deterministic CI validation. The release
+manager downloads and signs that exact archive; the private signing key remains
+only on the release manager's machine.
 
 ## Publish an approved release
 
@@ -207,8 +223,9 @@ than ASF's general one-hour minimum.
 
 - `bump_version.py`: consistently check or update CMake and documentation
   version metadata.
-- `create_source_release.sh`: deterministically create an archive, SHA-512
-  checksum, and optional detached signature from an immutable Git ref.
+- `create_source_release.sh`: deterministically create an archive with GNU
+  gzip, a SHA-512 checksum, and an optional detached signature from an immutable
+  Git ref.
 - `validate_source_archive.py`: reject unsafe or non-portable tar members and
   compiled files.
 - `verify_release_candidate.sh`: perform voter-facing integrity, license,

scripts/releasing/create_source_release.sh

@@ -49,6 +49,9 @@ The script creates:
   apache-paimon-cpp-VERSION-src.tgz.asc  (when --signing-key is provided)
 
 Existing artifacts are never overwritten.
+
+GNU gzip is required so macOS and Linux produce the same compressed bytes.
+Set PAIMON_GZIP to an explicit GNU gzip executable when it is not on PATH.
 EOF
 }
 
@@ -68,6 +71,36 @@ calculate_sha512() {
     fi
 }
 
+find_gnu_gzip() {
+    local candidate
+    local resolved
+    local version_line
+    local -a candidates
+
+    if [[ -n "${PAIMON_GZIP:-}" ]]; then
+        candidates=("${PAIMON_GZIP}")
+    else
+        candidates=(gzip ggzip)
+    fi
+
+    for candidate in "${candidates[@]}"; do
+        if [[ -x "${candidate}" ]]; then
+            resolved=${candidate}
+        elif resolved=$(command -v "${candidate}" 2>/dev/null); then
+            :
+        else
+            continue
+        fi
+        version_line=$("${resolved}" --version 2>/dev/null | sed -n '1p' || true)
+        if [[ "${version_line}" =~ ^gzip[[:space:]][0-9] ]]; then
+            printf '%s\n' "${resolved}"
+            return 0
+        fi
+    done
+
+    fail "GNU gzip is required for reproducible source archives; on macOS run 'brew install gzip' and set PAIMON_GZIP to the Homebrew gzip executable"
+}
+
 while [[ $# -gt 0 ]]; do
     case "$1" in
         --version)
@@ -128,6 +161,8 @@ DOCS_VERSION=$(
 [[ "${DOCS_VERSION}" == "${RELEASE_VERSION}" ]] ||
     fail "documentation version ${DOCS_VERSION:-<missing>} does not match ${RELEASE_VERSION}"
 
+GZIP_BIN=$(find_gnu_gzip)
+
 ARTIFACT_NAME="apache-paimon-cpp-${RELEASE_VERSION}-src.tgz"
 ARCHIVE_ROOT="paimon-cpp-${RELEASE_VERSION}"
 
@@ -147,7 +182,10 @@ git -C "${SOURCE_ROOT}" -c tar.umask=0022 archive \
     --format=tar \
     --prefix="${ARCHIVE_ROOT}/" \
     "${GIT_REF}" |
-    gzip -n >"${TEMP_DIR}/${ARTIFACT_NAME}"
+    (
+        unset GZIP
+        "${GZIP_BIN}" --no-name --stdout -6
+    ) >"${TEMP_DIR}/${ARTIFACT_NAME}"
 
 SHA512=$(calculate_sha512 "${TEMP_DIR}/${ARTIFACT_NAME}")
 printf '%s  %s\n' "${SHA512}" "${ARTIFACT_NAME}" \

scripts/releasing/release_rc.sh

@@ -31,6 +31,8 @@ DIST_DEV_BASE_URL="https://dist.apache.org/repos/dist/dev/paimon"
 PREPARE_ONLY=false
 DRY_RUN=false
 WORKFLOW_DISCOVERY_TIMEOUT_SECONDS=600
+WORKFLOW_RUN_ID=""
+TEMP_DIR=""
 
 usage() {
     cat <<'EOF'
@@ -53,8 +55,9 @@ Options:
   --dry-run               Print the planned release identifiers and exit
   -h, --help              Show this help
 
-The script is resumable when the local signed tag or artifacts already exist,
-provided that they match HEAD and pass all verification checks.
+For a published RC, GitHub Actions creates and tests the canonical source
+archive. This script downloads those exact bytes, signs them locally, and
+uploads them to ASF dist/dev. Prepare-only mode creates a local preview.
 EOF
 }
 
@@ -117,6 +120,74 @@ wait_for_release_candidate_workflow() {
         --exit-status \
         --interval 30 ||
         fail "Release Candidate workflow run ${run_id} failed"
+    WORKFLOW_RUN_ID=${run_id}
+}
+
+validate_workflow_artifact_directory() {
+    local directory=$1
+    local -a entries
+    local entry
+    local name
+
+    shopt -s dotglob nullglob
+    entries=("${directory}"/*)
+    shopt -u dotglob nullglob
+    [[ ${#entries[@]} -eq 2 ]] ||
+        fail "workflow artifact must contain exactly the archive and checksum"
+    for entry in "${entries[@]}"; do
+        [[ -f "${entry}" ]] ||
+            fail "workflow artifact contains a non-file entry: ${entry}"
+        name=$(basename "${entry}")
+        case "${name}" in
+            "${ARTIFACT_NAME}" | "${ARTIFACT_NAME}.sha512")
+                ;;
+            *)
+                fail "workflow artifact contains an unexpected file: ${name}"
+                ;;
+        esac
+    done
+}
+
+download_and_sign_workflow_artifact() {
+    local workflow_dir="${TEMP_DIR}/source-archive"
+    local source
+    local target
+    local suffix
+
+    [[ -n "${WORKFLOW_RUN_ID}" ]] || fail "release workflow run ID is missing"
+    mkdir -p "${workflow_dir}"
+    gh run download "${WORKFLOW_RUN_ID}" \
+        --repo apache/paimon-cpp \
+        --name source-archive \
+        --dir "${workflow_dir}"
+    validate_workflow_artifact_directory "${workflow_dir}"
+
+    mkdir -p "${OUTPUT_DIR}"
+    for suffix in "" ".sha512"; do
+        source="${workflow_dir}/${ARTIFACT_NAME}${suffix}"
+        target="${OUTPUT_DIR}/${ARTIFACT_NAME}${suffix}"
+        if [[ -e "${target}" ]]; then
+            [[ -f "${target}" ]] || fail "artifact path is not a file: ${target}"
+            cmp "${source}" "${target}" >/dev/null ||
+                fail "existing ${target} differs from workflow run ${WORKFLOW_RUN_ID}"
+        else
+            cp -p "${source}" "${target}"
+        fi
+    done
+
+    ARTIFACT="${OUTPUT_DIR}/${ARTIFACT_NAME}"
+    if [[ -e "${ARTIFACT}.asc" ]]; then
+        [[ -f "${ARTIFACT}.asc" ]] ||
+            fail "artifact signature path is not a file: ${ARTIFACT}.asc"
+        echo "Reusing existing source artifact signature."
+    else
+        echo "Signing workflow source artifact with ${SIGNING_KEY}."
+        gpg --armor \
+            --local-user "${SIGNING_KEY}" \
+            --detach-sign \
+            --output "${ARTIFACT}.asc" \
+            "${ARTIFACT}"
+    fi
 }
 
 validate_artifact_directory() {
@@ -224,6 +295,9 @@ EOF
     exit 0
 fi
 
+TEMP_DIR=$(mktemp -d)
+trap 'rm -rf "${TEMP_DIR}"' EXIT
+
 for command in git gpg python3; do
     require_command "${command}"
 done
@@ -269,33 +343,33 @@ else
 fi
 
 ARTIFACT="${OUTPUT_DIR}/${ARTIFACT_NAME}"
-if [[ -e "${ARTIFACT}" || -e "${ARTIFACT}.asc" || -e "${ARTIFACT}.sha512" ]]; then
-    [[ -f "${ARTIFACT}" && -f "${ARTIFACT}.asc" && -f "${ARTIFACT}.sha512" ]] ||
-        fail "artifact directory contains an incomplete release candidate"
-    echo "Reusing existing artifacts in ${OUTPUT_DIR}."
-else
-    "${SCRIPT_DIR}/create_source_release.sh" \
-        --version "${VERSION}" \
+if [[ "${PREPARE_ONLY}" == true ]]; then
+    if [[ -e "${ARTIFACT}" || -e "${ARTIFACT}.asc" || -e "${ARTIFACT}.sha512" ]]; then
+        [[ -f "${ARTIFACT}" && -f "${ARTIFACT}.asc" && -f "${ARTIFACT}.sha512" ]] ||
+            fail "artifact directory contains an incomplete release candidate"
+        echo "Reusing existing preview artifacts in ${OUTPUT_DIR}."
+    else
+        "${SCRIPT_DIR}/create_source_release.sh" \
+            --version "${VERSION}" \
+            --git-ref "${RC_TAG}" \
+            --output-dir "${OUTPUT_DIR}" \
+            --signing-key "${SIGNING_KEY}"
+    fi
+
+    "${SCRIPT_DIR}/verify_release_candidate.sh" \
         --git-ref "${RC_TAG}" \
-        --output-dir "${OUTPUT_DIR}" \
-        --signing-key "${SIGNING_KEY}"
-fi
-
-"${SCRIPT_DIR}/verify_release_candidate.sh" \
-    --git-ref "${RC_TAG}" \
-    --keys-url "https://downloads.apache.org/paimon/KEYS" \
-    "${ARTIFACT}"
+        --keys-url "https://downloads.apache.org/paimon/KEYS" \
+        "${ARTIFACT}"
+    validate_artifact_directory
 
-validate_artifact_directory
-
-if [[ "${PREPARE_ONLY}" == true ]]; then
     cat <<EOF
 
 Local release preparation completed successfully.
 
-The signed tag and source artifacts were created and verified locally.
+The signed tag and preview source artifacts were created and verified locally.
 No tag was pushed and no artifacts were uploaded to ASF dist/dev.
-Do not start a release vote from this prepare-only run.
+The published workflow artifact is authoritative; do not start a release vote
+from this prepare-only run.
 EOF
     exit 0
 fi
@@ -305,6 +379,14 @@ if svn info "${RC_URL}" >/dev/null 2>&1; then
 fi
 git push "${REMOTE}" "${RC_TAG}"
 wait_for_release_candidate_workflow
+download_and_sign_workflow_artifact
+
+"${SCRIPT_DIR}/verify_release_candidate.sh" \
+    --git-ref "${RC_TAG}" \
+    --keys-url "https://downloads.apache.org/paimon/KEYS" \
+    "${ARTIFACT}"
+validate_artifact_directory
+
 svn import "${OUTPUT_DIR}" "${RC_URL}" \
     -m "Add Apache Paimon C++ ${VERSION} RC${RC}"