Skip to content

GH-14: Add JNI CI test #441

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kou merged 6 commits into from
Dec 4, 2024
Merged

GH-14: Add JNI CI test #441

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8fd09cd
GH-14: Add JNI CI test
lidavidm Dec 2, 2024
cba8272
use 3.9.9
lidavidm Dec 2, 2024
5fcfca4
Update ci/docker/conda-java-jni.dockerfile
lidavidm Dec 3, 2024
99014d8
name?
lidavidm Dec 3, 2024
4b6e284
feedback
lidavidm Dec 4, 2024
49237d2
update
lidavidm Dec 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,4 +47,4 @@ ULIMIT_CORE=-1

# Default versions for various dependencies
JDK=11
MAVEN=3.9.6
MAVEN=3.9.9
11 changes: 8 additions & 3 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,16 +38,21 @@ env:

jobs:
ubuntu:
name: AMD64 Ubuntu 22.04 JDK ${{ matrix.jdk }} Maven ${{ matrix.maven }}
name: ${{ matrix.name }} AMD64 Ubuntu 22.04 JDK ${{ matrix.jdk }} Maven ${{ matrix.maven }}
Copy link
Copy Markdown
Member

@kou kou Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use "Ubuntu 24.04" for conda-java-jni-cdata?
(Or we may want to use just "Ubuntu".)

runs-on: ubuntu-latest
if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
jdk: [11, 17, 21, 22]
maven: [3.9.6]
image: [java]
maven: [3.9.9]
image: [java, conda-java-jni-cdata]
Comment thread
lidavidm marked this conversation as resolved.
Outdated
include:
- image: java
Copy link
Copy Markdown
Member

@kou kou Dec 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we rename this "java" too?
"ubuntu"? Because maven image uses Ubuntu.

Copy link
Copy Markdown
Member Author

@lidavidm lidavidm Dec 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, I also tweaked the CI job names a bit so that "Conda" is in the name.

name: ""
- image: conda-java-jni-cdata
name: JNI
env:
JDK: ${{ matrix.jdk }}
MAVEN: ${{ matrix.maven }}
Expand Down
30 changes: 30 additions & 0 deletions ci/docker/conda-java-jni.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

FROM ghcr.io/mamba-org/micromamba:ubuntu24.04
Copy link
Copy Markdown
Member

@wgtmac wgtmac Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm new to the process so this question might be dumb. Do we only build JNI libraries for ubuntu?

Copy link
Copy Markdown
Member Author

@lidavidm lidavidm Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the resulting libraries shouldn't be tied to Ubuntu. But perhaps we should use an older system as a base to avoid potential glibc incompatibility

Copy link
Copy Markdown
Member Author

@lidavidm lidavidm Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In any case, this is for testing and not for building the actual binaries, though

Copy link
Copy Markdown
Member

@wgtmac wgtmac Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation! Let me investigate how the actual binaries (for different platforms) are built for a release.


ARG jdk=11
ARG maven=3.9.9

RUN micromamba install -y \
-c conda-forge \
cmake \
compilers \
maven=${maven} \
ninja \
openjdk=${jdk} && \
micromamba clean --all
80 changes: 80 additions & 0 deletions ci/scripts/java_jni_build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

set -eo pipefail

arrow_dir=${1}
arrow_install_dir=${2}
build_dir=${3}/java_jni
# The directory where the final binaries will be stored when scripts finish
dist_dir=${4}
prefix_dir="${build_dir}/java-jni"

echo "=== Clear output directories and leftovers ==="
# Clear output directories and leftovers
rm -rf ${build_dir}

echo "=== Building Arrow Java C Data Interface native library ==="
mkdir -p "${build_dir}"
pushd "${build_dir}"

case "$(uname)" in
Linux)
n_jobs=$(nproc)
;;
Darwin)
n_jobs=$(sysctl -n hw.ncpu)
Copy link
Copy Markdown
Member

@kou kou Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, hw.logicalcpu is better than hw.ncpu:

Suggested change
n_jobs=$(sysctl -n hw.ncpu)
n_jobs=$(sysctl -n hw.logicalcpu)

https://developer.apple.com/documentation/kernel/1387446-sysctlbyname/determining_system_capabilities

hw.activecpu

    The number of enabled logical processor cores in the SoC. This is an alias of hw.logicalcpu.
hw.ncpu

    The number of logical processor cores in the SoC. This is an alias of hw.logicalcpu_max.
hw.logicalcpu

    The number of enabled logical processor cores in the SoC. This is an alias of hw.activecpu.
hw.logicalcpu_max

    The number of logical processor cores in the SoC. This is an alias of hw.ncpu.
hw.physicalcpu

    The number of enabled physical processor cores in the SoC.
hw.physicalcpu_max

    The number of physical processor cores in the SoC.

;;
*)
n_jobs=${NPROC:-1}
;;
esac

: ${ARROW_JAVA_BUILD_TESTS:=${ARROW_BUILD_TESTS:-OFF}}
: ${CMAKE_BUILD_TYPE:=release}
cmake \
-DARROW_JAVA_JNI_ENABLE_DATASET=${ARROW_DATASET:-OFF} \
-DARROW_JAVA_JNI_ENABLE_GANDIVA=${ARROW_GANDIVA:-OFF} \
-DARROW_JAVA_JNI_ENABLE_ORC=${ARROW_ORC:-OFF} \
-DBUILD_TESTING=${ARROW_JAVA_BUILD_TESTS} \
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
-DCMAKE_PREFIX_PATH=${arrow_install_dir} \
-DCMAKE_INSTALL_PREFIX=${prefix_dir} \
-DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD:-OFF} \
-DProtobuf_USE_STATIC_LIBS=ON \
-GNinja \
${JAVA_JNI_CMAKE_ARGS:-} \
${arrow_dir}
export CMAKE_BUILD_PARALLEL_LEVEL=${n_jobs}
cmake --build . --config ${CMAKE_BUILD_TYPE}
if [ "${ARROW_JAVA_BUILD_TESTS}" = "ON" ]; then
ctest \
--output-on-failure \
--parallel ${n_jobs} \
--timeout 300
fi
cmake --build . --config ${CMAKE_BUILD_TYPE} --target install
popd

mkdir -p ${dist_dir}
# For Windows. *.dll are installed into bin/ on Windows.
if [ -d "${prefix_dir}/bin" ]; then
mv ${prefix_dir}/bin/* ${dist_dir}/
else
mv ${prefix_dir}/lib/* ${dist_dir}/
fi
36 changes: 33 additions & 3 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,43 @@ services:
# docker compose build java
# docker compose run java
# Parameters:
# MAVEN: 3.9.6
# MAVEN: 3.9.9
# JDK: 11, 17, 21
image: ${ARCH}/maven:${MAVEN}-eclipse-temurin-${JDK}
volumes: &java-volumes
volumes:
- .:/arrow-java:delegated
- ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
command: &java-command >
command:
/bin/bash -c "
/arrow-java/ci/scripts/java_build.sh /arrow-java /build &&
/arrow-java/ci/scripts/java_test.sh /arrow-java /build"

conda-java-jni-cdata:
Copy link
Copy Markdown
Member

@kou kou Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may want to remove the -java part because this repository is only for the Java implementation.

# Usage:
# docker compose build java-jni-cdata
# docker compose run java-jni-cdata
Copy link
Copy Markdown
Member

@kou kou Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# docker compose build java-jni-cdata
# docker compose run java-jni-cdata
# docker compose build conda-java-jni-cdata
# docker compose run conda-java-jni-cdata

# Parameters:
# MAVEN: 3.9.9
# JDK: 11, 17, 21
image: ${REPO}:${ARCH}-conda-java-${JDK}-maven-${MAVEN}-jni-integration
build:
context: .
dockerfile: ci/docker/conda-java-jni.dockerfile
cache_from:
- ${REPO}:${ARCH}-conda-java-${JDK}-maven-${MAVEN}-jni-integration
args:
jdk: ${JDK}
maven: ${MAVEN}
# required to use micromamba with rootless docker
# https://github.com/mamba-org/micromamba-docker/issues/407#issuecomment-2088523507
user: root
volumes:
- .:/arrow-java:delegated
- ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
environment:
ARROW_JAVA_CDATA: "ON"
command:
/bin/bash -c "
/arrow-java/ci/scripts/java_jni_build.sh /arrow-java /build/jni /build /jni &&
/arrow-java/ci/scripts/java_build.sh /arrow-java /build /jni &&
/arrow-java/ci/scripts/java_test.sh /arrow-java /build /jni"