Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,23 +43,23 @@ Parquet-Java requires Java 17 or higher and uses Maven to build. It also depends
To build and install the thrift compiler, run:

```
wget -nv https://archive.apache.org/dist/thrift/0.23.0/thrift-0.23.0.tar.gz
tar xzf thrift-0.23.0.tar.gz
cd thrift-0.23.0
wget -nv https://archive.apache.org/dist/thrift/0.24.0/thrift-0.24.0.tar.gz
tar xzf thrift-0.24.0.tar.gz
cd thrift-0.24.0
chmod +x ./configure
./configure --disable-libs
sudo make install -j
```

Note: if you wish to verify the signature and checksum of a release:
1. The GPG and sha checksums can be found under https://archive.apache.org/dist/thrift/0.23.0/
1. The GPG and sha checksums can be found under https://archive.apache.org/dist/thrift/0.24.0/
2. Validate the signature of the artifact against the [Thrift committer KEYS](https://downloads.apache.org/thrift/KEYS).

If you're on OSX and use homebrew, you can instead install Thrift 0.23.0 with `brew` and ensure that it comes first in your `PATH`.
If you're on OSX and use homebrew, you can instead install Thrift 0.24.0 with `brew` and ensure that it comes first in your `PATH`.

```
brew install thrift
export PATH="/usr/local/opt/thrift@0.23.0/bin:$PATH"
export PATH="$(brew --prefix thrift)/bin:$PATH"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunally, this is an improvement since brew does not keep the old Thrift versions

```

### Build Parquet with Maven
Expand Down
2 changes: 1 addition & 1 deletion dev/ci-before_install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
# This script gets invoked by the CI system in a "before install" step
################################################################################

export THRIFT_VERSION=0.23.0
export THRIFT_VERSION=0.24.0

set -e
set -o pipefail
Expand Down
64 changes: 64 additions & 0 deletions dev/docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

FROM maven:3.9.8-eclipse-temurin-17

ARG THRIFT_VERSION=0.24.0
ARG THRIFT_SHA256=e0fa5839a4c5c1d631b0931cf2c554ebbfa4e2fee3a9fb3ffd4f82ce4396c6e4

USER root

RUN set -eux; \
apt-get update; \
apt-get install --yes --no-install-recommends \
autoconf \
automake \
bison \
ca-certificates \
flex \
g++ \
libboost-dev \
libevent-dev \
libssl-dev \
libtool \
make \
pkg-config \
wget; \
wget --quiet --output-document=/tmp/thrift.tar.gz \
"https://archive.apache.org/dist/thrift/${THRIFT_VERSION}/thrift-${THRIFT_VERSION}.tar.gz"; \
echo "${THRIFT_SHA256} /tmp/thrift.tar.gz" | sha256sum --check --status; \
mkdir /tmp/thrift-src; \
tar --extract --gzip --file=/tmp/thrift.tar.gz --directory=/tmp/thrift-src --strip-components=1; \
cd /tmp/thrift-src; \
chmod +x ./configure; \
./configure --disable-libs; \
make -j"$(nproc)"; \
make install; \
thrift -version | grep --fixed-strings "Thrift version ${THRIFT_VERSION}"; \
rm -rf /tmp/thrift.tar.gz /tmp/thrift-src /var/lib/apt/lists/*

RUN useradd --create-home --user-group --shell /bin/bash parquet \
&& install --directory --owner=parquet --group=parquet /workspace /home/parquet/.m2

ENV MAVEN_CONFIG=/home/parquet/.m2
# TestLargeColumnChunk intentionally writes a 4 GiB row group.
ENV JAVA_TOOL_OPTIONS=-Xmx6g

USER parquet
WORKDIR /workspace

CMD ["bash"]
69 changes: 69 additions & 0 deletions dev/docker/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
<!--
~ Licensed to the Apache Software Foundation (ASF) under one
~ or more contributor license agreements. See the NOTICE file
~ distributed with this work for additional information
~ regarding copyright ownership. The ASF licenses this file
~ to you under the Apache License, Version 2.0 (the
~ "License"); you may not use this file except in compliance
~ with the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing,
~ software distributed under the License is distributed on an
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
~ KIND, either express or implied. See the License for the
~ specific language governing permissions and limitations
~ under the License.
-->

# Parquet developer container

**Status:** Experimental developer tooling. This Dockerfile is not officially supported.

This image provides the Java 17, Maven 3.9.8, and Thrift 0.24.0 toolchain used to build Parquet-Java. It deliberately does not copy project sources into the image: mount the checkout you are actively editing at `/workspace`.

## Build the image

From the repository root, run:

```bash
docker build --tag parquet-java-dev --file dev/docker/Dockerfile dev/docker
```

Verify the container's Thrift compiler:

```bash
docker run --rm parquet-java-dev thrift -version
```

## Work in the container

Start an interactive shell with the current checkout and a persistent Maven dependency cache:

```bash
docker run --rm --init --interactive --tty \
--mount "type=bind,src=$PWD,dst=/workspace" \
--mount type=volume,src=parquet-java-m2,dst=/home/parquet/.m2 \
parquet-java-dev
```

Build and test the local checkout without opening a shell:

```bash
docker run --rm --init \
--mount "type=bind,src=$PWD,dst=/workspace" \
--mount type=volume,src=parquet-java-m2,dst=/home/parquet/.m2 \
parquet-java-dev \
./mvnw --batch-mode test
```

The full suite includes a test that writes a 4 GiB row group. Configure Docker with at least 8 GiB of memory; the image gives JVM processes a 6 GiB maximum heap by default. Override that setting when needed with `--env JAVA_TOOL_OPTIONS=-Xmx<heap-size>`.

## Reuse the host Maven cache

The commands above use the persistent `parquet-java-m2` Docker volume. To reuse the local Maven repository instead, replace its volume mount with:

```bash
--mount "type=bind,src=$HOME/.m2,dst=/home/parquet/.m2"
```
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@
<thrift.executable>thrift</thrift.executable>
<format.thrift.executable>${thrift.executable}</format.thrift.executable>
<thrift-maven-plugin.version>0.10.0</thrift-maven-plugin.version>
<thrift.version>0.23.0</thrift.version>
<thrift.version>0.24.0</thrift.version>
<format.thrift.version>${thrift.version}</format.thrift.version>
<fastutil.version>8.5.19</fastutil.version>
<semver.api.version>0.9.33</semver.api.version>
Expand Down
Loading