Compile s3 --include/--exclude patterns once per transfer - #10552
Open
Adityaj0 wants to merge 1 commit into
Open
Compile s3 --include/--exclude patterns once per transfer#10552Adityaj0 wants to merge 1 commit into
Adityaj0 wants to merge 1 commit into
Conversation
``Filter._match_pattern`` ran ``pattern.replace('/', os.sep)`` for every
file for every pattern, and ``fnmatch.fnmatch`` normcased both the path
and the pattern on each call before its cache lookup. All of that is
constant across the transfer. Patterns are now separator-normalized and
translated to a compiled regex once per source type, and the path is
normcased once per file rather than once per pattern.
100k files, 2 patterns 3.30 -> 2.12 us/file (-36%)
100k files, 6 patterns 9.94 -> 5.99 us/file (-40%)
Behaviour is unchanged: a differential test comparing the old and new
matching over 400 randomized pattern/path combinations reports no
mismatches, and the two added tests pass against both implementations.
``self.patterns`` / ``self.dst_patterns`` keep their existing shape and
contents, since tests and external callers read them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split out of #10545 at @aemous's request — this is half 2 of 2. The other half is #10551 (deferring the docutils imports); the two are independent and touch disjoint files.
Filter._match_patternranpattern.replace('/', os.sep)for every file for every pattern, andfnmatch.fnmatchnormcased both the path and the pattern on every call before its internal cache lookup. All of that is constant across a transfer.Patterns are now separator-normalized and translated to a compiled regex once per source type, and the path is normcased once per file instead of once per pattern.
Profiling 50k files against 6 patterns showed only 0.19s of the original 1.68s was actual regex matching — the rest was repeated setup:
End to end:
Worth being clear about the scope of that number: this is client-side filter evaluation only. On a real
aws s3 syncthe network dominates, so this matters most for large listings where most files are filtered out — it is not a 40% reduction in overall sync time.self.patterns/self.dst_patternskeep their existing shape and contents, since tests and external callers read them.Testing
tests/unit/customizations/s3,tests/functional/s3: 996 passed.local/s3source types, spaces, case variation, glob metacharacters in filenames): 0 mismatches.ruffoutput on the touched files is unchanged from baseline.I did not run the full
tests/functionalsuite locally.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.