archive:create-from and archive:create support for archives larger than 2 GB#2624
Open
vincentml wants to merge 8 commits intoBaseXdb:mainfrom
Open
archive:create-from and archive:create support for archives larger than 2 GB#2624vincentml wants to merge 8 commits intoBaseXdb:mainfrom
vincentml wants to merge 8 commits intoBaseXdb:mainfrom
Conversation
…tOfBoundsException: Maximum array size exceeded" when creating zip file larger than 2 GB
…tOfBoundsException: Maximum array size exceeded" when creating zip file larger than 2 GB, return lazy reference to temp file
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request makes it possible for
archive:createandarchive:create-fromto create zip files that are larger than 2 Gb.Using BaseX version 12.2, when attempting to create zip files using
archive:createorarchive:create-fromand the size of the files is larger than about 2 Gb I've run into error messages such as "java.lang.ArrayIndexOutOfBoundsException: Maximum array size exceeded (2147483640 > 2147483639)."For example, this error is produced if the total size of a folder being zipped is 3 Gb when passing the result of
archive:create-fromdirectly tofile:write-binary:and when using a variable to pass the result of
archive:create-fromtofile:write-binary:After the changes in this pull request, the above queries produce the expected zip file and the error does not occur.
The current limitation of ~ 2 Gb is due to the file contents being accumulated in memory and exceding the maximum array size set by Java's Integer.MAX_VALUE.
This pull request solves this problem by avoiding the use of an array, and instead accumulates data in memory up to a threshold then switches to a temporary file if the data exceeds the threshold. The threshold is determined from available memory capped at the maximum array size. The temporary file, if created, is deleted automatically. This approach attempts to optimize for the typical use cases of creating small or mid-size archives while making it possible to create very large archives.