Skip to content
This repository was archived by the owner on Sep 12, 2025. It is now read-only.
Closed
Show file tree
Hide file tree
Changes from 31 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
03ed1c0
Add scripts for create ubuntu jammy docker file
leborchuk Mar 26, 2025
b09962e
Remove internal yandex dependencies
leborchuk Mar 26, 2025
779dafd
Deb version always should start with a number
leborchuk Mar 26, 2025
f2ab2ed
Install sudo - need for various scripts
leborchuk Mar 26, 2025
212980c
Resolve review notes
leborchuk Mar 27, 2025
044362f
Update images/docker/cbdb/test/ubuntu22.04/Dockerfile
leborchuk Apr 3, 2025
181f398
Update images/docker/cbdb/test/ubuntu22.04/Dockerfile
leborchuk Apr 3, 2025
843e667
Update images/docker/cbdb/test/ubuntu22.04/Dockerfile
leborchuk Apr 3, 2025
4d44425
Merge branch 'apache:main' into AddJammyBuild
leborchuk Apr 3, 2025
0e777df
Combed the Dockerfile based on the comments in PR
leborchuk Apr 4, 2025
9dd4b6f
install into /usr/cloudberry-db
leborchuk Apr 4, 2025
38b9d5b
Move debian package creation changes from https://github.com/apache/c…
leborchuk Apr 7, 2025
60bbbb7
Move some changes to https://github.com/apache/cloudberry-devops-rele…
leborchuk Apr 7, 2025
0e5dd66
Do not touch build_automation/cloudberry/scripts in this PR
leborchuk Apr 7, 2025
b30da1f
Add test_cloudberry_db_env
leborchuk Apr 7, 2025
1114b87
Add gpinitsystem.conf
leborchuk Apr 7, 2025
aa2f271
Use python3 in test container
leborchuk Apr 8, 2025
ff80ee0
Fix test debian build
leborchuk May 9, 2025
484a546
Merge branch 'apache:main' into AddJammyBuild
leborchuk Jun 2, 2025
c133b60
Merge branch 'apache:main' into AddJammyBuild
leborchuk Jun 5, 2025
70a7ef8
Cope edespino files
Jun 6, 2025
55c129d
Add go to path
Jun 6, 2025
19a6754
Change docker key
Jun 6, 2025
bfd7d3c
Push to ghcr.io registry
Jun 6, 2025
08af24e
Merge branch 'AddJammyBuild' into AddDeb
leborchuk Jun 10, 2025
d1673d4
Merge pull request #3 from leborchuk/AddDeb
leborchuk Jun 10, 2025
2db34e1
Remove excessive dependencies
Jun 10, 2025
8549e40
Tru to set correct repository in tag
Jun 10, 2025
a4bf288
Add libxerces to dependencies
Jun 13, 2025
f06fa77
Do not need python-setuptools to build deb
Jun 14, 2025
9d4400b
Move back to incubator-cloudberry docker
Jun 16, 2025
c49b173
Revert registry to apache incubator
Jun 18, 2025
90f06a6
Do not change rights to binary files
Jun 26, 2025
f506a9c
Add jammy build to cloudberry-devops-release
tuhaihe Jun 7, 2025
1475527
Merge branch 'main' into AddJammyBuild
leborchuk Jul 24, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions .github/workflows/docker-cbdb-build-containers.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ on:
paths:
- 'images/docker/cbdb/build/rocky8/**'
- 'images/docker/cbdb/build/rocky9/**'
- 'images/docker/cbdb/build/ubuntu22.04/**'
workflow_dispatch: # Manual trigger

# Prevent multiple workflow runs from interfering with each other
Expand All @@ -76,7 +77,7 @@ jobs:
# Matrix strategy to build for both Rocky Linux 8 and 9
strategy:
matrix:
platform: ['rocky8', 'rocky9']
platform: ['rocky8', 'rocky9', 'ubuntu22.04']

steps:
# Checkout repository code with full history
Expand All @@ -103,17 +104,17 @@ jobs:
- 'images/docker/cbdb/build/rocky8/**'
rocky9:
- 'images/docker/cbdb/build/rocky9/**'
ubuntu22.04:
- 'images/docker/cbdb/build/ubuntu22.04/**'

# Set up QEMU for multi-architecture support
# This allows building ARM64 images on AMD64 infrastructure and vice versa
- name: Set up QEMU
if: ${{ steps.platform-filter.outputs[matrix.platform] == 'true' }}
uses: docker/setup-qemu-action@v3

# Login to DockerHub for pushing images
# Requires DOCKERHUB_USER and DOCKERHUB_TOKEN secrets to be set
- name: Login to Docker Hub
if: ${{ steps.platform-filter.outputs[matrix.platform] == 'true' }}
# Login to GitHub Container Registry for pushing images
- name: Login to GitHub Container Registry
Comment thread
leborchuk marked this conversation as resolved.
Outdated
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USER }}
Expand Down
21 changes: 12 additions & 9 deletions .github/workflows/docker-cbdb-test-containers.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ on:
paths:
- 'images/docker/cbdb/test/rocky8/**'
- 'images/docker/cbdb/test/rocky9/**'
- 'images/docker/cbdb/test/ubuntu22.04/**'
workflow_dispatch: # Manual trigger

# Prevent multiple workflow runs from interfering with each other
Expand All @@ -62,8 +63,8 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
# Build for both Rocky Linux 8 and 9
platform: ['rocky8', 'rocky9']
# Build for both Rocky Linux 8 and 9, ubuntu 22.04
platform: ['rocky8', 'rocky9', 'ubuntu22.04']

steps:
# Checkout repository code
Expand All @@ -87,6 +88,8 @@ jobs:
- 'images/docker/cbdb/test/rocky8/**'
rocky9:
- 'images/docker/cbdb/test/rocky9/**'
ubuntu22.04:
- 'images/docker/cbdb/test/ubuntu22.04/**'

# Skip if no changes for current platform
- name: Skip if not relevant
Expand Down Expand Up @@ -144,8 +147,8 @@ jobs:
type=gha,scope=docker-cbdb-test-${{ matrix.platform }}
# Tag with both latest and version-specific tags
tags: |
apache/incubator-cloudberry:cbdb-test-${{ matrix.platform }}-latest
apache/incubator-cloudberry:cbdb-test-${{ matrix.platform }}-${{ steps.version.outputs.BUILD_DATE }}-${{ steps.version.outputs.SHA_SHORT }}
ghcr.io/${{ github.repository }}:cbdb-test-${{ matrix.platform }}-latest
ghcr.io/${{ github.repository }}:cbdb-test-${{ matrix.platform }}-${{ steps.version.outputs.BUILD_DATE }}-${{ steps.version.outputs.SHA_SHORT }}
Comment thread
leborchuk marked this conversation as resolved.
Outdated
# Add metadata labels for better image tracking
labels: |
org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}
Expand All @@ -168,15 +171,15 @@ jobs:
echo "- **Build Date**: ${{ steps.version.outputs.BUILD_DATE }}" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "#### 🐳 Docker Images" >> $GITHUB_STEP_SUMMARY
echo "- Latest tag: \`apache/incubator-cloudberry:cbdb-test-${{ matrix.platform }}-latest\`" >> $GITHUB_STEP_SUMMARY
echo "- Version tag: \`apache/incubator-cloudberry:cbdb-test-${{ matrix.platform }}-${{ steps.version.outputs.BUILD_DATE }}-${{ steps.version.outputs.SHA_SHORT }}\`" >> $GITHUB_STEP_SUMMARY
echo "- Latest tag: \`ghcr.io/${{ github.repository }}:cbdb-test-${{ matrix.platform }}-latest\`" >> $GITHUB_STEP_SUMMARY
echo "- Version tag: \`ghcr.io/${{ github.repository }}:cbdb-test-${{ matrix.platform }}-${{ steps.version.outputs.BUILD_DATE }}-${{ steps.version.outputs.SHA_SHORT }}\`" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "#### 📋 Quick Reference" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`bash" >> $GITHUB_STEP_SUMMARY
echo "# Pull the image (automatically selects correct architecture)" >> $GITHUB_STEP_SUMMARY
echo "docker pull apache/incubator-cloudberry:cbdb-test-${{ matrix.platform }}-latest" >> $GITHUB_STEP_SUMMARY
echo "docker pull ghcr.io/${{ github.repository }}:cbdb-test-${{ matrix.platform }}-latest" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "# Pull specific architecture if needed" >> $GITHUB_STEP_SUMMARY
echo "docker pull --platform linux/amd64 apache/incubator-cloudberry:cbdb-test-${{ matrix.platform }}-latest" >> $GITHUB_STEP_SUMMARY
echo "docker pull --platform linux/arm64 apache/incubator-cloudberry:cbdb-test-${{ matrix.platform }}-latest" >> $GITHUB_STEP_SUMMARY
echo "docker pull --platform linux/amd64 ghcr.io/${{ github.repository }}:cbdb-test-${{ matrix.platform }}-latest" >> $GITHUB_STEP_SUMMARY
echo "docker pull --platform linux/arm64 ghcr.io/${{ github.repository }}:cbdb-test-${{ matrix.platform }}-latest" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
6 changes: 3 additions & 3 deletions build_automation/cloudberry/scripts/build-cloudberry.sh
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
# Prerequisites:
# - configure-cloudberry.sh must be run first
# - Required build dependencies must be installed
# - /usr/local/cloudberry-db/lib must exist and be writable
# - ${BUILD_DESTINATION}/lib must exist and be writable
#
# Exit Codes:
# 0 - Build and installation completed successfully
Expand All @@ -67,11 +67,11 @@ export LOG_DIR="${SRC_DIR}/build-logs"
BUILD_LOG="${LOG_DIR}/build.log"

# Initialize environment
init_environment "Cloudberry Build Script" "${BUILD_LOG}"
init_environment "Cloudberry Build Script" "${BUILD_LOG}" "${BUILD_DESTINATION}"

# Set environment
log_section "Environment Setup"
export LD_LIBRARY_PATH=/usr/local/cloudberry-db/lib:LD_LIBRARY_PATH
export LD_LIBRARY_PATH=${BUILD_DESTINATION}/lib:LD_LIBRARY_PATH
log_section_end "Environment Setup"

# Build process
Expand Down
25 changes: 23 additions & 2 deletions build_automation/cloudberry/scripts/cloudberry-utils.sh
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,12 @@
# LOG_DIR - Directory for logs (defaults to ${SRC_DIR}/build-logs)
#
# Functions:
# init_environment "Script Name" "Log File"
# init_environment "Script Name" "Log File" "Build Destination"
# - Initialize logging and verify environment
# - Parameters:
# * script_name: Name of the calling script
# * log_file: Path to log file
# * build_destination: Path to Cloudberry destination, by default is /usr/local/cloudberry-db
# - Returns: 0 on success, 1 on failure
#
# execute_cmd command [args...]
Expand Down Expand Up @@ -70,7 +71,7 @@
#
# Example:
# source ./cloudberry-utils.sh
# init_environment "My Script" "${LOG_FILE}"
# init_environment "My Script" "${LOG_FILE}" "${BUILD_DESTINATION}"
# execute_cmd make clean
# log_section "Build Process"
# execute_cmd make -j$(nproc)
Expand All @@ -79,10 +80,18 @@
#
# --------------------------------------------------------------------

DEFAULT_BUILD_DESTINATION=/usr/local/cloudberry-db

# Initialize logging and environment
init_environment() {
local script_name=$1
local log_file=$2
local build_destination=$3

if [ -z "$build_destination" ]; then
build_destination=${DEFAULT_BUILD_DESTINATION}
fi
export BUILD_DESTINATION=$build_destination

echo "=== Initializing environment for ${script_name} ==="
echo "${script_name} executed at $(date)" | tee -a "${log_file}"
Expand All @@ -91,6 +100,7 @@ init_environment() {
echo "Working directory: $(pwd)" | tee -a "${log_file}"
echo "Source directory: ${SRC_DIR}" | tee -a "${log_file}"
echo "Log directory: ${LOG_DIR}" | tee -a "${log_file}"
echo "Build destination: ${BUILD_DESTINATION}" | tee -a "${log_file}"

if [ -z "${SRC_DIR:-}" ]; then
echo "Error: SRC_DIR environment variable is not set" | tee -a "${log_file}"
Expand Down Expand Up @@ -146,3 +156,14 @@ log_completion() {
local timestamp=$(date "+%Y.%m.%d-%H.%M.%S")
echo "${script_name} execution completed successfully at ${timestamp}" | tee -a "${log_file}"
}

detect_os() {
if [ -f /etc/os-release ]; then
. /etc/os-release
OS_ID=$ID
OS_VERSION=$VERSION_ID
else
echo "Unsupported system: cannot detect OS" >&2
exit 99
fi
}
35 changes: 21 additions & 14 deletions build_automation/cloudberry/scripts/configure-cloudberry.sh
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
# Description: Configures Apache Cloudberry build environment and runs
# ./configure with optimized settings. Performs the
# following:
# 1. Prepares /usr/local/cloudberry-db directory
# 1. Prepares ${BUILD_DESTINATION} directory
# 2. Sets up library dependencies
# 3. Configures build with required features enabled
#
Expand All @@ -33,7 +33,6 @@
# - MapReduce Processing
# - Oracle Compatibility (orafce)
# - ORCA Query Optimizer
# - PAX Access Method
# - PXF External Table Access
# - Test Automation Support (tap-tests)
#
Expand Down Expand Up @@ -92,49 +91,57 @@ set -euo pipefail
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
source "${SCRIPT_DIR}/cloudberry-utils.sh"

# Call it before conditional logic
detect_os

echo "Detected OS: $OS_ID $OS_VERSION"

# Define log directory and files
export LOG_DIR="${SRC_DIR}/build-logs"
CONFIGURE_LOG="${LOG_DIR}/configure.log"

# Initialize environment
init_environment "Cloudberry Configure Script" "${CONFIGURE_LOG}"
init_environment "Cloudberry Configure Script" "${CONFIGURE_LOG}" "${BUILD_DESTINATION}"

# Initial setup
log_section "Initial Setup"
execute_cmd sudo rm -rf /usr/local/cloudberry-db || exit 2
execute_cmd sudo rm -rf ${BUILD_DESTINATION}/* || exit 2
execute_cmd sudo chmod a+w /usr/local || exit 2
execute_cmd mkdir -p /usr/local/cloudberry-db/lib || exit 2
execute_cmd sudo cp /usr/local/xerces-c/lib/libxerces-c.so \
/usr/local/xerces-c/lib/libxerces-c-3.3.so \
/usr/local/cloudberry-db/lib || exit 3
execute_cmd sudo chown -R gpadmin:gpadmin /usr/local/cloudberry-db || exit 2

execute_cmd sudo mkdir -p ${BUILD_DESTINATION}/lib || exit 2
if [[ "$OS_ID" == "rocky" && "$OS_VERSION" =~ ^(8|9) ]]; then
execute_cmd sudo cp /usr/local/xerces-c/lib/libxerces-c.so \
/usr/local/xerces-c/lib/libxerces-c-3.3.so \
/usr/local/cloudberry-db/lib || exit 3
fi
execute_cmd sudo chown -R gpadmin:gpadmin ${BUILD_DESTINATION} || exit 2
Comment thread
leborchuk marked this conversation as resolved.

log_section_end "Initial Setup"

# Set environment
log_section "Environment Setup"
export LD_LIBRARY_PATH=/usr/local/cloudberry-db/lib:LD_LIBRARY_PATH
export LD_LIBRARY_PATH=${BUILD_DESTINATION}/lib:LD_LIBRARY_PATH
log_section_end "Environment Setup"

# Add debug options if ENABLE_DEBUG is set to "true"
CONFIGURE_DEBUG_OPTS=""

if [ "${ENABLE_DEBUG:-false}" = "true" ]; then
CONFIGURE_DEBUG_OPTS="--enable-debug \
--enable-profiling \
--enable-cassert \
--enable-debug-extensions"
fi

# Configure build
log_section "Configure"
execute_cmd ./configure --prefix=/usr/local/cloudberry-db \
execute_cmd ./configure --prefix=${BUILD_DESTINATION} \
--disable-external-fts \
--enable-gpcloud \
--enable-ic-proxy \
--enable-mapreduce \
--enable-orafce \
--enable-orca \
--enable-pax \
--enable-pax \
--enable-pxf \
--enable-tap-tests \
${CONFIGURE_DEBUG_OPTS} \
Expand All @@ -152,7 +159,7 @@ execute_cmd ./configure --prefix=/usr/local/cloudberry-db \
--with-openssl \
--with-uuid=e2fs \
--with-includes=/usr/local/xerces-c/include \
--with-libraries=/usr/local/cloudberry-db/lib || exit 4
--with-libraries=${BUILD_DESTINATION}/lib || exit 4
log_section_end "Configure"

# Capture version information
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
# LOG_DIR - Directory for logs (defaults to ${SRC_DIR}/build-logs)
#
# Prerequisites:
# - Apache Cloudberry must be installed (/usr/local/cloudberry-db)
# - Apache Cloudberry must be installed (${BUILD_DESTINATION})
# - SSH must be configured for passwordless access to localhost
# - User must have permissions to create cluster directories
# - PostgreSQL client tools (psql) must be available
Expand Down Expand Up @@ -75,11 +75,11 @@ export LOG_DIR="${SRC_DIR}/build-logs"
CLUSTER_LOG="${LOG_DIR}/cluster.log"

# Initialize environment
init_environment "Cloudberry Demo Cluster Script" "${CLUSTER_LOG}"
init_environment "Cloudberry Demo Cluster Script" "${CLUSTER_LOG}" "${BUILD_DESTINATION}"

# Setup environment
log_section "Environment Setup"
source /usr/local/cloudberry-db/greenplum_path.sh || exit 1
source ${BUILD_DESTINATION}/greenplum_path.sh || exit 1
log_section_end "Environment Setup"

# Verify SSH access
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ export LOG_DIR="${SRC_DIR}/build-logs"
CLUSTER_LOG="${LOG_DIR}/destroy-cluster.log"

# Initialize environment
init_environment "Destroy Cloudberry Demo Cluster Script" "${CLUSTER_LOG}"
init_environment "Destroy Cloudberry Demo Cluster Script" "${CLUSTER_LOG}" "${BUILD_DESTINATION}"

# Source Cloudberry environment
log_section "Environment Setup"
Expand Down
2 changes: 1 addition & 1 deletion build_automation/cloudberry/scripts/test-cloudberry.sh
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ export LOG_DIR="build-logs"
TEST_LOG="${LOG_DIR}/test.log"

# Initialize environment
init_environment "Cloudberry Test Script" "${TEST_LOG}"
init_environment "Cloudberry Test Script" "${TEST_LOG}" "${BUILD_DESTINATION}"

# Source Cloudberry environment
log_section "Environment Setup"
Expand Down
4 changes: 2 additions & 2 deletions build_automation/cloudberry/scripts/unittest-cloudberry.sh
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,11 @@ export LOG_DIR="${SRC_DIR}/build-logs"
UNITTEST_LOG="${LOG_DIR}/unittest.log"

# Initialize environment
init_environment "Cloudberry Unittest Script" "${UNITTEST_LOG}"
init_environment "Cloudberry Unittest Script" "${UNITTEST_LOG}" "${BUILD_DESTINATION}"

# Set environment
log_section "Environment Setup"
export LD_LIBRARY_PATH=/usr/local/cloudberry-db/lib:LD_LIBRARY_PATH
export LD_LIBRARY_PATH=${BUILD_DESTINATION}/lib:LD_LIBRARY_PATH
log_section_end "Environment Setup"

# Unittest process
Expand Down
Loading