SOLR-17328: Add CycloneDX SBOMs to Solr binary distributions - #4690
Open
ppkarwasz wants to merge 11 commits into
Open
SOLR-17328: Add CycloneDX SBOMs to Solr binary distributions#4690ppkarwasz wants to merge 11 commits into
ppkarwasz wants to merge 11 commits into
Conversation
Each binary distribution (full and slim) now ships a CycloneDX 1.6 bom.json describing its actual contents. Two resolvable configurations in :solr:packaging (bomFull, bomSlim) mirror the distribution assembly and are rendered by the CycloneDX Gradle plugin, then a post-processing step adjusts the result: * The metadata declares the "build" lifecycle phase (a CISA Build SBOM) and lists the post-processing and cyclonedx-npm next to the plugin in the tools. * The main component identifies the binary release with a draft "sid" purl (purl-spec issue apache#516, pkg:sid/apache.org/solr/solr@<version> with an edition qualifier) and the Solr CPE used by the NVD. * Maven BOM/platform dependencies and the internal :platform project, which are not part of the distribution, are stripped. * Solr project components get valid Maven purls (artifactId instead of the Gradle project name, with the ASF snapshots repository for snapshot builds), a description and the Apache-2.0 license. * The UI artifacts inside the webapp are covered: the npm packages bundled by browserify into the OpenAPI JS client and the Maven artifacts compiled into the wasmJs UI are nested as subassemblies of first-party solr-js-client and solr-ui components, from child SBOMs generated in :solr:webapp:js-client (official cyclonedx-npm tool, runtime dependencies only) and :solr:ui (plugin task over wasmJsRuntimeClasspath). The vendored JavaScript libraries of the AngularJS admin UI are listed from a curated list. * The location of every JAR and JavaScript file in the distribution is recorded as evidence.occurrences by matching SHA-256 hashes against the assembled directories, which also proves the vendored libraries ship unmodified. * Only the SHA-256 hash of each component is kept; the plugin emits eight algorithms per artifact, which only adds bulk. The BOM configurations resolve strictly with the JVM runtime attributes (the packaging project applies jvm-ecosystem for the attribute schema), so BOM-managed and variant-aware dependencies resolve like a runtime classpath instead of silently disappearing. bomFull resolves consistently with bomSlim, matching the distribution layout where server libraries win over module-pulled versions. The child SBOMs degrade gracefully when the UI projects are disabled with -PdisableJsClient / -PdisableUiModule. Assisted-By: Claude Fable 5 <noreply@anthropic.com>
epugh
reviewed
Aug 1, 2026
epugh
left a comment
epugh
left a comment
Contributor
There was a problem hiding this comment.
It feels like there is a lot of boilerplate-ish comments. Do we need all of them? Maybe the more verbose pattern is something we want to adopt?
Review feedback on apache#4690: the CycloneDX BOM configurations, the two generator tasks and the post-processing filled roughly 480 of the 873 lines of solr/packaging/build.gradle, a file otherwise concerned with assembling the distribution archives. They now live in their own script plugin, applied from the root build next to gradle/solr/packaging.gradle. The move is verbatim except for the CycloneDX types. A script plugin is not compiled against the plugins block classpath of the script that applies it, so CyclonedxDirectTask and Component.Type are looked up by name through the buildscript class loader instead of being imported. What stays in :solr:packaging is the wiring that depends on tasks the distribution plugin only creates later: the bom.json entries in the distribution contents and the dependsOn 'cyclonedx' declarations. Both SBOMs are byte-identical to the ones generated before the move, apart from the per-run serial number and timestamp. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bomFull and bomSlim configurations exist only on this branch, so changes on main that alter what :solr:packaging resolves do not update their entries in solr/packaging/gradle.lockfile. Two such changes had accumulated and broke the build: * SOLR-18187 (apache#4259) added the anthropic, google-ai-gemini and ollama langchain4j providers to :solr:modules:language-models, which bomFull pulls in transitively. * apache#4594 bumped the OpenTelemetry and Prometheus stacks. Regenerated with "gradlew resolveAndLockAll collectJarInfos --write-locks"; only the packaging lockfile changed. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review feedback on apache#4690: the third-party JavaScript under solr/webapp/web/libs was described by a hard-coded list in the SBOM code, which nothing tied to the directory it described, so a file could be added, bumped or removed and the SBOM would keep reporting the old state. Three of the sixteen files also had no purl at all, leaving them invisible to vulnerability scanners. The components now live in solr/webapp/vendored-libs.json as CycloneDX component objects, merged into the distribution SBOM as they are, and :solr:webapp:validateVendoredLibs fails whenever the manifest and the directory disagree. The generator tasks depend on it, so a stale hash cannot reach a BOM, which also matters for the archive locations of step 8 since those are matched by hash. updateVendoredLibs refreshes the hashes; it cannot invent identity or pedigree, so a new file keeps failing validation until those are filled in by hand. Every component now identifies its upstream package by purl, even where the shipped bytes deviate, so that advisories still match, and records the deviation under "pedigree" with the original in pedigree.ancestors. Comparing each file against upstream established that: * eleven are the pristine upstream artifact with an Apache licence header prepended, and the ancestors carry the verified upstream SHA-256 next to a distribution reference; * ui-grid.min.js is byte-identical to npm angular-ui-grid 4.10.0 and so needs no pedigree at all; * ngtimeago.js is a fork of uttesh/ngtimeago pinned to the untagged commit 0b1e72785a6e, patched by SOLR-7780, LUCENE-6732 and SOLR-13343, which pedigree.commits now records. It is unrelated to the npm package ng-timeago; * jquery-ui.min.js, highlight.js, angular-chosen.min.js and jssha-3.3.1-sha256.min.js are custom or differently built artifacts that match no published file; they keep the upstream purl so that advisories are not missed, with the deviation in pedigree.notes. d3.js and angular-utf8-base64.min.js are served by neither unpkg nor cdnjs under those names; the former is d3.v2.js inside the d3 2.8.1 tarball and the latter exists only in the git repository, hence its pkg:github purl. Both BOMs validate against the CycloneDX 1.6 schema. Apart from the three components that gain a purl, and the pedigree entries, the generated SBOMs are unchanged. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to 53981c1, which described the blocks prepended to the files under solr/webapp/web/libs as Apache licence headers. They are not: each is the upstream project's own licence text carrying the upstream copyright holder, MIT for the AngularJS, jQuery, Chosen, jsTree and ngtimeago files and BSD-3-Clause for d3. The pedigree notes now say which licence text was prepended, without asserting anything about who wrote it. The one file that does carry Apache-2.0 text is angular-utf8-base64.min.js, and that is not the ASF's either: the library encapsulates Vassilis Petroulias's base64.js, released by its own author under Apache-2.0, whose notice the vendoring reproduces. Since the shipped code is therefore part MIT and part Apache-2.0, its licence becomes the SPDX expression "MIT AND Apache-2.0". Attribution moves out of prose and into the fields meant for it. Every component now carries a manufacturer, which is what satisfies the Component Producer element of the 2026 CISA SBOM Minimum Elements; that element replaced the 2021 NTIA Supplier Name and asks for the original project or maintaining organization. Individually authored libraries also list their authors. The copyright field is dropped: it is not a minimum element under either the 2021 or the 2026 guidance, and the attribution it held is now carried by manufacturer and authors. highlight.js records its version as "unknown" rather than omitting it. Component Version is a required element, and the guidance asks the SBOM author to state explicitly that information is unknown instead of leaving it out. The file carries no version marker and its API predates highlight.js 8.0. All producer URLs use https, and each was checked to resolve. Two could not simply change scheme: angular-ui.github.com serves no https, so ui-grid points at its GitHub project, and getharvest.com redirects. Both BOMs still validate against the CycloneDX 1.6 schema. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Member
Author
|
Yesterday's Apache Trusted Releases presentation made me reconsider the "SBOM location" decision above. Rather than shipping This does give up the two properties I argued for earlier (the SBOM travelling with the archive so it can't get detached, and being auto-discoverable via the recognized CycloneDX filename patterns inside the archive) in exchange for consistency with how ATR expects to publish and surface SBOMs alongside the rest of the release metadata. Open to keeping both if that's not considered redundant: sidecar for ATR/downloads.apache.org discovery, |
Each module's lib/ directory is assembled from a copyRecursive() of its runtimeLibs configuration. A copied Gradle configuration inherits its source's dependencies and excludes but not its shouldResolveConsistentlyWith alignment and not its lock state, so the alignWithRuntimeClasspath call a few lines above, whose comment promises "the distribution contains the same library versions that were tested", was silently discarded for the copy that gets packaged. The subtraction that follows removes the platform libraries by file, so an unaligned copy that resolved a different version of a library the platform also ships was not recognised as a duplicate and stayed. Five such jars reached the binary distribution: cross-dc-manager/lib/error_prone_annotations-2.49.0.jar modules/cuvs/lib/lucene-backward-codecs-10.2.0.jar modules/gcs-repository/lib/jackson-annotations-2.18.3.jar modules/gcs-repository/lib/jackson-core-2.18.3.jar modules/gcs-repository/lib/jackson-databind-2.18.3.jar They were not the versions the tests ran against, they were absent from the lock state, and they were the only five of the 188 third-party jars under the module lib directories with no solr/licenses entry, so validateJarChecksums never saw them either. The error_prone_annotations one was never a runtime dependency at all: 2.49.0 is locked only for the annotationProcessor and errorprone configurations. Aligning the copy makes both sides of the subtraction resolve to the same file, so all five are now recognised as duplicates of the platform's copies and dropped. Nothing is added: consistent resolution only raises versions of modules already in the graph. Runtime behaviour does not change. SolrResourceLoader appends module lib URLs after the existing ones and the WEB-INF/lib jars sit in the parent loader, so the platform version already won; the five jars were dead weight that only file-level scanners could see. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gap fixed in the previous commit was invisible to the build: the distribution shipped five jars that no BOM component described, and nothing noticed. Compare with a file-level scan of the tarball and they show up immediately. Step 8 of the post-processing already indexes every file in the assembled distribution by SHA-256 in order to record evidence.occurrences, so the leftovers of that match are free. Any jar whose hash matches no component now fails the generator task, naming the paths. The check is scoped to jars. The JavaScript client bundle and the wasm UI bundle are single files assembled from many components, so their occurrences are recorded by hand in steps 6 and 7 rather than matched by hash. Verified by reverting the packaging fix and rerunning cyclonedxFull, which fails with exactly those five paths. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Member
Author
|
e96844d introduces a change to the built package: it removes 5 JARs from |
Member
Author
|
I cross-checked the SBOM this PR generates against three external scanners, all on the same
Where they overlap they agree: 346 of Syft's maven coordinates and 416 of cdxgen's match ours exactly. The differences are scope, not disagreement. What the scanners add: shaded dependencies read from embedded What they miss: the entire Admin UI supply chain. Both cdxgen modes and Syft are Java-only here, so the 44 npm packages browserified into Caveats on the tools.
|
The full distribution builds on the slim one with "with(distributions.slim.getContents())", which brought in the slim BOM as bom.json, and the full BOM was then added under the same name with DuplicatesStrategy.INCLUDE. The archive therefore carried two bom.json entries, 588 KB and 938 KB, and which one landed on disk was up to the extraction tool: GNU tar overwrites and so happens to leave the correct full BOM, but a tool that stops at the first match would silently give the slim BOM for the full distribution. Lift the shared content into a "commonDistContents" copy spec that holds no BOM, so each distribution adds only its own. Verified to be a no-op otherwise: the full archive listing goes from 2042 to 2041 entries with the duplicate as the only difference, and the slim listing is unchanged down to the file modes and sizes. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each distribution already carries its BOM as bom.json in the archive root, which keeps the SBOM with the artifact it describes but means anyone who only wants the SBOM has to download the whole distribution to reach it. Publish the same file next to the archive as well, named after it: solr-<version>.tgz.cdx.json solr-<version>-slim.tgz.cdx.json with a checksum and, when signing is enabled, a signature, like every other release artifact. The two files are exposed as separate artifacts of a new "sbom" configuration rather than as a directory, so that computeChecksums and the signing task see each of them. No SBOM is published for the source release: assembleSourceTgz is a raw "git archive" export, so there is no resolved dependency set for a build-time SBOM to describe. smokeTestRelease.py needs a matching change. checkSigs() walked a sorted listing and treated every entry starting with "<artifact>." as one of its signatures, which the new files break twice over: solr-X.tgz.cdx.json merely starts with solr-X.tgz., and its own .sha512 sorts between the archive and the archive's .sha512, so the archive's signatures are no longer adjacent to it. An entry is now recognised as a signature only when stripping .asc or .sha512 leaves the name of another entry, which drops the dependency on listing order altogether. Verified against a locally assembled release folder, signed with a throwaway key: each sibling is byte-identical to the bom.json inside the corresponding archive, every checksum and signature verifies, checkSigs() passes end to end, and both files validate against the CycloneDX 1.6 schema. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ssing The closing parenthesis was misplaced, so the tuple was passed to RuntimeError as a second argument and the format string was left with one argument for two placeholders. Rather than reporting the missing file, testOpenApi raised "TypeError: not enough arguments for format string". Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
documentation
Member
Author
This was implemented in 89dd020. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/SOLR-17328
Generates precise SBOM for the Solr binary distributions (
solr-<version>.tgzandsolr-<version>-slim.tgz) that include all the informationavailable today to the build tool:
:solr:packaging,solr/webapp/web/libs,:solr:webapp:js-client, inserted as nested assemblies ofsolr-js-client.:solr:ui, inserted as nested assemblies ofsolr-ui.The components for 1 and 4 are generated by the CycloneDX Gradle Plugin, while components from 3 are generated by
@cyclonedx/cyclonedx-npm.Vendored libraries, of course, are handled as a static list that maintainers need to keep up to date.
This is the second attempt after #3929.
Known unknowns
This PR does not attempt to manually detect the components embedded in JARs through shading or binary library inclusion.
An incomplete list of artifacts with such hidden dependencies is:
(The Hadoop artifacts and
org.tallison.xmp:xmpcore-shadedfrom earlier iterations of this list no longer ship in the distribution.)SBOM location
Each archive contains its SBOM as
bom.jsonin the root of the distribution, next toLICENSE.txtandNOTICE.txt(see this comment on the previous PR for the full discussion):bom.jsonis one of the recognized CycloneDX filename patterns, so tools can discover it without configuration.syftorcdxgenremains a useful, complementary verification step.There are other publication models: a
solr-<version>.tgz.cdx.jsonsidecar next to the.ascand.sha512files ondownloads.apache.org, or embedding the SBOM in an in-toto attestation bundle together with SLSA provenance.SBOM postprocessing
The raw output of the CycloneDX Gradle Plugin and
@cyclonedx/cyclonedx-npmis postprocessed this way:buildlifecycle phase is recorded, and the post-processing and cyclonedx-npm are listed in the tools next to the plugin.sidpurl (purl-spec issue #516),pkg:sid/apache.org/solr/solr@<version>?edition=full|slim, and by the Solr CPE used by the NVD.:platformproject are stripped, they are not part of the distribution.Apache-2.0license.solr/webapp/web/libsare added.server/solr-webapp/webapp/libs/solr/index.jsare nested as subassemblies of a first-partysolr-js-clientcomponent.server/solr-webapp/webapp/uiare nested as subassemblies of a first-partysolr-uicomponent, from a child SBOM generated in:solr:uioverwasmJsRuntimeClasspath, together with the@js-joda/corenpm package bundled by the Kotlin toolchain.evidence.occurrences. This also proves that each vendored JavaScript library ships unmodified.externalReferences(SHA-512 of the registry tarballs) are preserved.This change was developed with significant use of an AI coding assistant (Claude), with human review of every step; commits carry an
Assisted-Bytrailer.Checklist
Please review the following and check all that apply:
mainbranch../gradlew precommit; the full./gradlew checktest suite was not run for this build-only change.bom.jsoncould be documented in a follow-up)