Support Git-style searchPaths with wildcards in AWS S3 buckets (#2812)#2958
Open
tomy8964 wants to merge 2 commits into
Open
Support Git-style searchPaths with wildcards in AWS S3 buckets (#2812)#2958tomy8964 wants to merge 2 commits into
tomy8964 wants to merge 2 commits into
Conversation
…g-cloud#2812) Fixes spring-cloud#2812 - ListObjectsV2 + AntPathMatcher based matching for *, **, ?, dot-wildcard - auto-ext lookup order (.properties → .json → .yml/.yaml) - directory scan, deduplication, prefix extraction - only active when searchPaths non-empty Signed-off-by: Geonwook Ham <tomy8964@naver.com> Signed-off-by: ham <tomy8964@naver.com>
Signed-off-by: Geonwook Ham <tomy8964@naver.com>
Author
|
Hi @ryanjbaxter! |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends the AWS S3 environment repository to support Git-style searchPaths (placeholders + wildcard matching), enabling S3-backed config layouts to match existing Git-backed repository conventions.
Changes:
- Add
searchPathsconfiguration to the AWS S3 backend and wire it through the factory. - Implement placeholder expansion and wildcard-based S3 key discovery (including directory scans) with deduplication.
- Add an extensive JUnit 5 test suite covering placeholder/wildcard resolution scenarios.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 10 comments.
| File | Description |
|---|---|
spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/environment/AwsS3EnvironmentRepository.java |
Adds searchPaths support, wildcard matching via AntPathMatcher, S3 list/head probing, and key→property source wrapping. |
spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/environment/AwsS3EnvironmentRepositoryFactory.java |
Passes searchPaths from properties into the repository constructor. |
spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/environment/AwsS3EnvironmentProperties.java |
Introduces searchPaths as a bindable configuration property. |
spring-cloud-config-server/src/test/java/org/springframework/cloud/config/server/environment/AwsS3EnvironmentRepositoryTests.java |
Adds coverage for placeholder/wildcard behavior, ordering, and special cases. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
121
to
127
| String[] profileArray = parseProfiles(profiles); | ||
| List<String> apps = Arrays.asList(StringUtils.commaDelimitedListToStringArray(application.replace(" ", ""))); | ||
| Collections.reverse(apps); | ||
| if (!apps.contains(serverProperties.getDefaultApplicationName())) { | ||
| if (searchPaths.isEmpty() && !apps.contains(serverProperties.getDefaultApplicationName())) { | ||
| Collections.reverse(apps); | ||
| apps = new ArrayList<>(apps); | ||
| apps.add(serverProperties.getDefaultApplicationName()); | ||
| } |
Comment on lines
150
to
+162
| private void addPropertySources(Environment environment, List<String> apps, String[] profiles, | ||
| List<String> labels) { | ||
| if (!this.searchPaths.isEmpty()) { | ||
| for (String label : labels) { | ||
| for (String profile : profiles) { | ||
| for (String app : apps) { | ||
| List<S3ConfigFile> s3ConfigFiles = getS3ConfigFileWithSearchPaths(app, profile, label); | ||
| addPropertySource(environment, s3ConfigFiles); | ||
| } | ||
| } | ||
| } | ||
| return; | ||
| } |
Comment on lines
+300
to
+305
| for (String template : this.searchPaths) { | ||
| String pattern = template | ||
| .replace("{application}", application) | ||
| .replace("{profile}", profile == null ? "" : profile) | ||
| .replace("{label}", label == null ? "" : label); | ||
|
|
Comment on lines
+308
to
+312
| for (String ext : List.of(".properties", ".json", ".yml", ".yaml")) { | ||
| String key = pattern.endsWith(ext) ? pattern : pattern + ext; | ||
| if (!seenKeys.add(key)) { | ||
| continue; | ||
| } |
Comment on lines
+1042
to
+1045
| server.setDefaultLabel("main"); | ||
| List<String> paths = List.of("{label}/foo-bar.yml"); | ||
| AwsS3EnvironmentRepository repo = | ||
| new AwsS3EnvironmentRepository(s3Client, "bucket1", false, server, paths); |
Comment on lines
121
to
127
| String[] profileArray = parseProfiles(profiles); | ||
| List<String> apps = Arrays.asList(StringUtils.commaDelimitedListToStringArray(application.replace(" ", ""))); | ||
| Collections.reverse(apps); | ||
| if (!apps.contains(serverProperties.getDefaultApplicationName())) { | ||
| if (searchPaths.isEmpty() && !apps.contains(serverProperties.getDefaultApplicationName())) { | ||
| Collections.reverse(apps); | ||
| apps = new ArrayList<>(apps); | ||
| apps.add(serverProperties.getDefaultApplicationName()); | ||
| } |
Comment on lines
150
to
+162
| private void addPropertySources(Environment environment, List<String> apps, String[] profiles, | ||
| List<String> labels) { | ||
| if (!this.searchPaths.isEmpty()) { | ||
| for (String label : labels) { | ||
| for (String profile : profiles) { | ||
| for (String app : apps) { | ||
| List<S3ConfigFile> s3ConfigFiles = getS3ConfigFileWithSearchPaths(app, profile, label); | ||
| addPropertySource(environment, s3ConfigFiles); | ||
| } | ||
| } | ||
| } | ||
| return; | ||
| } |
Comment on lines
+300
to
+305
| for (String template : this.searchPaths) { | ||
| String pattern = template | ||
| .replace("{application}", application) | ||
| .replace("{profile}", profile == null ? "" : profile) | ||
| .replace("{label}", label == null ? "" : label); | ||
|
|
Comment on lines
+308
to
+312
| for (String ext : List.of(".properties", ".json", ".yml", ".yaml")) { | ||
| String key = pattern.endsWith(ext) ? pattern : pattern + ext; | ||
| if (!seenKeys.add(key)) { | ||
| continue; | ||
| } |
Comment on lines
+1042
to
+1045
| server.setDefaultLabel("main"); | ||
| List<String> paths = List.of("{label}/foo-bar.yml"); | ||
| AwsS3EnvironmentRepository repo = | ||
| new AwsS3EnvironmentRepository(s3Client, "bucket1", false, server, paths); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add full Git-style
searchPathssupport (placeholders + wildcards) to the AWS S3 backend so that users can migrate existing Git-based configurations without renaming toapplication.*..*), single (?) and double (**) wildcard matching.properties → .json → .yml → .yamlwhen expanding literals or.*patternsThis issue (#2812)
resolves #2812
Changes
Core Logic
Extended
getS3ConfigFileWithSearchPaths(...)to treat{application}and{profile}placeholders identically to the Git backendAdded support for:
.*): expands against supported extensions in priority order?)**)/Ensured seenKeys set to avoid duplicate property sources
Tests
Added new JUnit 5 tests in
AwsS3EnvironmentRepositoryTeststo cover all scenarios:searchPaths_placeholderOnly_shouldResolveExactFilesearchPaths_wildcardOnly_shouldResolveAllPropertiessearchPaths_placeholderAndWildcard_shouldResolveMatchingKeyssearchPaths_orderMatters_forPropertySourceOrdersearchPath_extensionPreserved(dynamic tests for each extension)searchPaths_applicationAsDirectory_shouldStillHonorSearchPathsmultiDocumentYaml_withSearchPaths_shouldNotSplitDocumentsgetLocations_returnsCorrectsearchPaths_deduplication_shouldOnlyAddOncesearchPaths_singleCharacterWildcard_shouldMatchExactlyOneCharsearchPaths_withEmptyLabel_shouldUseDefaultLabelsearchPaths_multipleLabels_shouldApplyForEachLabelInReverseOrdersearchPaths_literalNotFound_shouldReturnEmpty…plus the additional edge cases for literal-stop, dot-wildcard priority, and nested directory patterns.
Example Usage
Additional Notes
Backward Compatibility
This update does not break any existing AWS S3 config setups. The default lookup behavior is unchanged if no placeholders or wildcards are used in
searchPaths.Migration
Existing users migrating from Git-backed config servers can now use their current
searchPaths(including wildcards and placeholders) on S3 with no renaming or convention change required.Documentation
Documentation and usage examples for the enhanced
searchPathswill be updated in the relevant documentation files after the merge.If there are specific locations that require documentation updates, please let me know—I will be happy to update them.
Performance and Cost
Using wildcards (such as
*or**) in searchPaths may result in additional AWS S3 API calls (e.g., ListObjects), especially for large buckets or deeply nested directory patterns.This could lead to increased latency and higher AWS costs.
Users should consider the structure and size of their buckets when designing searchPaths and monitor AWS S3 usage accordingly.