Skip to content

feat(python): add pyproject.toml provider support #358

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ruromero merged 1 commit into from
Mar 24, 2026
Merged

feat(python): add pyproject.toml provider support #358

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 25 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ public class TrustifyExample {
<li><a href="https://www.java.com/">Java</a> - <a href="https://maven.apache.org/">Maven</a></li>
<li><a href="https://www.javascript.com//">JavaScript</a> - <a href="https://www.npmjs.com//">Npm</a></li>
<li><a href="https://go.dev//">Golang</a> - <a href="https://go.dev/blog/using-go-modules//">Go Modules</a></li>
<li><a href="https://go.dev//">Python</a> - <a href="https://pypi.org/project/pip//">pip Installer</a></li>
<li><a href="https://www.python.org/">Python</a> - <a href="https://pypi.org/project/pip/">pip Installer</a> (<code>requirements.txt</code>, <code>pyproject.toml</code>)</li>
<li><a href="https://gradle.org//">Gradle</a> - <a href="https://gradle.org/install//">Gradle Installation</a></li>
<li><a href="https://www.rust-lang.org/">Rust</a> - <a href="https://doc.rust-lang.org/cargo/">Cargo</a></li>

Expand Down Expand Up @@ -264,7 +264,7 @@ require (
</li>

<li>
<em>Python pip</em> users can add in requirement text a comment with #trustify-da-ignore(or # trustify-da-ignore) to the right of the same artifact to be ignored, for example:
<em>Python pip</em> users can add in <code>requirements.txt</code> a comment with #trustify-da-ignore(or # trustify-da-ignore) to the right of the same artifact to be ignored, for example:

```properties
anyio==3.6.2
Expand Down Expand Up @@ -297,6 +297,20 @@ zipp==3.6.0
```
</li>

<li>
<em>Python pyproject.toml</em> users can add a comment with #trustify-da-ignore next to a dependency in <code>pyproject.toml</code>:

```toml
[project]
name = "my-project"
dependencies = [
"requests>=2.28.1",
"flask>=2.0", # trustify-da-ignore
"click>=8.0",
]
```
</li>

<li>
<em>Gradle</em> users can add in build.gradle a comment with //trustify-da-ignore next to the package to be ignored:
```build.gradle
Expand Down Expand Up @@ -538,7 +552,7 @@ System.setProperty("TRUSTIFY_DA_MVN_LOCAL_REPO", "/home/user/custom-maven-repo")
##### Background

In Python pip and in golang go modules package managers ( especially in Python pip) , There is a big chance that for a certain manifest and a given package inside it, the client machine environment has different version installed/resolved
for that package, which can lead to perform the analysis on the installed packages' versions , instead on the declared versions ( in manifests - that is requirements.txt/go.mod ), and this
for that package, which can lead to perform the analysis on the installed packages' versions , instead on the declared versions ( in manifests - that is requirements.txt/pyproject.toml/go.mod ), and this
can cause a confusion for the user in the client consuming the API and leads to inconsistent output ( in THE manifest there is version X For a given Package `A` , and in the analysis report there is another version for the same package `A` - Y).

##### Usage
Expand Down Expand Up @@ -579,16 +593,18 @@ TRUSTIFY_DA_GO_MVS_LOGIC_ENABLED=false

#### Python Support

Python support works with both `requirements.txt` and `pyproject.toml` manifest files. The `pyproject.toml` provider scans production dependencies from PEP 621 `[project.dependencies]` and Poetry `[tool.poetry.dependencies]` sections. Optional dependencies (`[project.optional-dependencies]`) and Poetry group dependencies (`[tool.poetry.group.*.dependencies]`) are intentionally excluded to focus on runtime dependencies.

By default, Python support assumes that the package is installed using the pip/pip3 binary on the system PATH, or of the customized
Binaries passed to environment variables. If the package is not installed , then an error will be thrown.

There is an experimental feature of installing the requirement.txt on a virtual env(only python3 or later is supported for this feature) - in this case,
There is an experimental feature of installing the dependencies on a virtual env(only python3 or later is supported for this feature) - in this case,
it's important to pass in a path to python3 binary as `TRUSTIFY_DA_PYTHON3_PATH` or instead make sure that python3 is on the system path.
in such case, You can use that feature by setting environment variable `TRUSTIFY_DA_PYTHON_VIRTUAL_ENV` to true
in such case, You can use that feature by setting environment variable `TRUSTIFY_DA_PYTHON_VIRTUAL_ENV` to true

##### "Best Efforts Installation"
Since Python pip packages are very sensitive/picky regarding python version changes( every small range of versions is only tailored for a certain python version), I'm introducing this feature, that
tries to install all packages in requirements.txt onto created virtual environment while **disregarding** versions declared for packages in requirements.txt
tries to install all packages in the manifest onto created virtual environment while **disregarding** versions declared for packages
This increasing the chances and the probability a lot that the automatic installation will succeed.

##### Usage
Expand Down Expand Up @@ -693,6 +709,9 @@ java -jar trustify-da-java-client-cli.jar stack /path/to/build.gradle --html
# Component analysis with JSON output (default)
java -jar trustify-da-java-client-cli.jar component /path/to/requirements.txt

# Component analysis for pyproject.toml
java -jar trustify-da-java-client-cli.jar component /path/to/pyproject.toml

# Component analysis with summary
java -jar trustify-da-java-client-cli.jar component /path/to/go.mod --summary

Expand Down
273 changes: 16 additions & 257 deletions src/main/java/io/github/guacsec/trustifyda/providers/PythonPipProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,196 +16,42 @@
*/
package io.github.guacsec.trustifyda.providers;

import static io.github.guacsec.trustifyda.impl.ExhortApi.debugLoggingIsNeeded;

import com.fasterxml.jackson.core.JsonProcessingException;
import com.github.packageurl.MalformedPackageURLException;
import com.github.packageurl.PackageURL;
import io.github.guacsec.trustifyda.Api;
import io.github.guacsec.trustifyda.Provider;
import io.github.guacsec.trustifyda.license.LicenseUtils;
import io.github.guacsec.trustifyda.logging.LoggersFactory;
import io.github.guacsec.trustifyda.sbom.Sbom;
import io.github.guacsec.trustifyda.sbom.SbomFactory;
import io.github.guacsec.trustifyda.tools.Ecosystem;
import io.github.guacsec.trustifyda.tools.Operations;
import io.github.guacsec.trustifyda.utils.Environment;
import io.github.guacsec.trustifyda.utils.IgnorePatternDetector;
import io.github.guacsec.trustifyda.utils.PythonControllerBase;
import io.github.guacsec.trustifyda.utils.PythonControllerRealEnv;
import io.github.guacsec.trustifyda.utils.PythonControllerVirtualEnv;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.logging.Logger;
import java.util.stream.Collectors;

public final class PythonPipProvider extends Provider {

private static final Logger log = LoggersFactory.getLogger(PythonPipProvider.class.getName());
private static final String DEFAULT_PIP_ROOT_COMPONENT_NAME = "default-pip-root";
private static final String DEFAULT_PIP_ROOT_COMPONENT_VERSION = "0.0.0";

public void setPythonController(PythonControllerBase pythonController) {
this.pythonController = pythonController;
}

private PythonControllerBase pythonController;
public final class PythonPipProvider extends PythonProvider {

public PythonPipProvider(Path manifest) {
super(Ecosystem.Type.PYTHON, manifest);
super(manifest);
}

@Override
public String readLicenseFromManifest() {
return LicenseUtils.readLicenseFile(manifest);
protected Path getRequirementsPath() {
return manifest;
}

@Override
public Content provideStack() throws IOException {
PythonControllerBase pythonController = getPythonController();
List<Map<String, Object>> dependencies =
pythonController.getDependencies(manifest.toString(), true);
printDependenciesTree(dependencies);
Sbom sbom = SbomFactory.newInstance(Sbom.BelongingCondition.PURL, "sensitive");
sbom.addRoot(
toPurl(DEFAULT_PIP_ROOT_COMPONENT_NAME, DEFAULT_PIP_ROOT_COMPONENT_VERSION),
readLicenseFromManifest());
for (Map<String, Object> component : dependencies) {
addAllDependencies(sbom.getRoot(), component, sbom);
}
byte[] requirementsFile = Files.readAllBytes(manifest);
handleIgnoredDependencies(new String(requirementsFile), sbom);
return new Content(
sbom.getAsJsonString().getBytes(StandardCharsets.UTF_8), Api.CYCLONEDX_MEDIA_TYPE);
}

private void addAllDependencies(PackageURL source, Map<String, Object> component, Sbom sbom) {

PackageURL packageURL =
toPurl((String) component.get("name"), (String) component.get("version"));
sbom.addDependency(source, packageURL, null);

List<Map<String, Object>> directDeps =
(List<Map<String, Object>>) component.get("dependencies");
if (directDeps != null) {
for (Map<String, Object> dep : directDeps) {
addAllDependencies(packageURL, dep, sbom);
}
}
protected void cleanupRequirementsPath(Path requirementsPath) {
// No cleanup needed — the manifest is the requirements file itself.
}

@Override
public Content provideComponent() throws IOException {
PythonControllerBase pythonController = getPythonController();
List<Map<String, Object>> dependencies =
pythonController.getDependencies(manifest.toString(), false);
printDependenciesTree(dependencies);
Sbom sbom = SbomFactory.newInstance();
sbom.addRoot(
toPurl(DEFAULT_PIP_ROOT_COMPONENT_NAME, DEFAULT_PIP_ROOT_COMPONENT_VERSION),
readLicenseFromManifest());
dependencies.forEach(
(component) ->
sbom.addDependency(
sbom.getRoot(),
toPurl((String) component.get("name"), (String) component.get("version")),
null));

var manifestContent = Files.readString(manifest);
handleIgnoredDependencies(manifestContent, sbom);
return new Content(
sbom.getAsJsonString().getBytes(StandardCharsets.UTF_8), Api.CYCLONEDX_MEDIA_TYPE);
protected Set<PackageURL> getIgnoredDependencies(String manifestContent) {
String[] lines = manifestContent.split(System.lineSeparator());
return Arrays.stream(lines)
.filter(this::containsIgnorePattern)
.map(PythonPipProvider::extractDepFull)
.map(this::splitToNameVersion)
.map(dep -> toPurl(dep[0], dep[1]))
.collect(Collectors.toSet());
}

private void printDependenciesTree(List<Map<String, Object>> dependencies)
throws JsonProcessingException {
if (debugLoggingIsNeeded()) {
String pythonControllerTree =
objectMapper.writerWithDefaultPrettyPrinter().writeValueAsString(dependencies);
log.info(
String.format(
"Python Generated Dependency Tree in Json Format: %s %s %s",
System.lineSeparator(), pythonControllerTree, System.lineSeparator()));
}
}

private void handleIgnoredDependencies(String manifestContent, Sbom sbom) {
Set<PackageURL> ignoredDeps = getIgnoredDependencies(manifestContent);
Set<String> ignoredDepsVersions =
ignoredDeps.stream()
.filter(dep -> !dep.getVersion().trim().equals("*"))
.map(PackageURL::getCoordinates)
.collect(Collectors.toSet());
Set<String> ignoredDepsNoVersions =
ignoredDeps.stream()
.filter(dep -> dep.getVersion().trim().equals("*"))
.map(PackageURL::getCoordinates)
.collect(Collectors.toSet());

// filter out by name only from sbom all exhortignore dependencies that their version will be
// resolved by pip.
sbom.setBelongingCriteriaBinaryAlgorithm(Sbom.BelongingCondition.NAME);
sbom.filterIgnoredDeps(ignoredDepsNoVersions);
boolean matchManifestVersions = Environment.getBoolean(PROP_MATCH_MANIFEST_VERSIONS, true);
// filter out by purl from sbom all exhortignore dependencies that their version hardcoded in
// requirements.txt -
// in case all versions in manifest matching installed versions of packages in environment.
if (matchManifestVersions) {
sbom.setBelongingCriteriaBinaryAlgorithm(Sbom.BelongingCondition.PURL);
sbom.filterIgnoredDeps(ignoredDepsVersions);
} else {
// in case version mismatch is possible (MATCH_MANIFEST_VERSIONS=false) , need to parse the
// name of package
// from the purl, and remove the package name from sbom according to name only
Set<String> deps =
ignoredDepsVersions.stream()
.map(
purlString -> {
try {
return new PackageURL(purlString).getName();
} catch (MalformedPackageURLException e) {
throw new RuntimeException(e);
}
})
.collect(Collectors.toSet());
sbom.setBelongingCriteriaBinaryAlgorithm(Sbom.BelongingCondition.NAME);
sbom.filterIgnoredDeps(deps);
}
}

/**
* Checks if a text line contains a Python pip ignore pattern. Handles both '#exhortignore' and
* '#trustify-da-ignore' with optional spacing.
*
* @param line the line to check
* @return true if the line contains a Python pip ignore pattern
*/
private boolean containsPythonIgnorePattern(String line) {
return line.contains("#" + IgnorePatternDetector.IGNORE_PATTERN)
|| line.contains("# " + IgnorePatternDetector.IGNORE_PATTERN)
|| line.contains("#" + IgnorePatternDetector.LEGACY_IGNORE_PATTERN)
|| line.contains("# " + IgnorePatternDetector.LEGACY_IGNORE_PATTERN);
}

private Set<PackageURL> getIgnoredDependencies(String requirementsDeps) {

String[] requirementsLines = requirementsDeps.split(System.lineSeparator());
Set<PackageURL> collected =
Arrays.stream(requirementsLines)
.filter(this::containsPythonIgnorePattern)
.map(PythonPipProvider::extractDepFull)
.map(this::splitToNameVersion)
.map(dep -> toPurl(dep[0], dep[1]))
// .map(packageURL -> packageURL.getCoordinates())
.collect(Collectors.toSet());

return collected;
private static String extractDepFull(String requirementLine) {
return requirementLine.substring(0, requirementLine.indexOf("#")).trim();
}

private String[] splitToNameVersion(String nameVersion) {
Expand All @@ -219,91 +65,4 @@ private String[] splitToNameVersion(String nameVersion) {
}
return result;
}

private static String extractDepFull(String requirementLine) {
return requirementLine.substring(0, requirementLine.indexOf("#")).trim();
}

private PackageURL toPurl(String name, String version) {

try {
return new PackageURL(Ecosystem.Type.PYTHON.getType(), null, name, version, null, null);
} catch (MalformedPackageURLException e) {
throw new RuntimeException(e);
}
}

private PythonControllerBase getPythonController() {
String pythonPipBinaries;
boolean useVirtualPythonEnv;
if (!Environment.get(PythonControllerBase.PROP_TRUSTIFY_DA_PIP_SHOW, "").trim().isEmpty()
&& !Environment.get(PythonControllerBase.PROP_TRUSTIFY_DA_PIP_FREEZE, "")
.trim()
.isEmpty()) {
pythonPipBinaries = "python;;pip";
useVirtualPythonEnv = false;
} else {
pythonPipBinaries = getExecutable("python", "--version");
useVirtualPythonEnv =
Environment.getBoolean(PythonControllerBase.PROP_TRUSTIFY_DA_PYTHON_VIRTUAL_ENV, false);
}

String[] parts = pythonPipBinaries.split(";;");
var python = parts[0];
var pip = parts[1];
PythonControllerBase pythonController;
if (this.pythonController == null) {
if (useVirtualPythonEnv) {
pythonController = new PythonControllerVirtualEnv(python);
} else {
pythonController = new PythonControllerRealEnv(python, pip);
}
} else {
pythonController = this.pythonController;
}
return pythonController;
}

private String getExecutable(String command, String args) {
String python = Operations.getCustomPathOrElse("python3");
String pip = Operations.getCustomPathOrElse("pip3");
try {
Operations.runProcess(python, args);
Operations.runProcess(pip, args);
} catch (Exception e) {
python = Operations.getCustomPathOrElse("python");
pip = Operations.getCustomPathOrElse("pip");
try {
Process process = new ProcessBuilder(command, args).redirectErrorStream(true).start();
int exitCode = process.waitFor();
if (exitCode != 0) {
throw new IOException(
"Python executable found, but it exited with error code " + exitCode);
}
} catch (IOException | InterruptedException ex) {
throw new RuntimeException(
String.format(
"Unable to find or run Python executable '%s'. Please ensure Python is installed"
+ " and available in your PATH.",
command),
ex);
}

try {
Process process = new ProcessBuilder("pip", args).redirectErrorStream(true).start();
int exitCode = process.waitFor();
if (exitCode != 0) {
throw new IOException("Pip executable found, but it exited with error code " + exitCode);
}
} catch (IOException | InterruptedException ex) {
throw new RuntimeException(
String.format(
"Unable to find or run Pip executable '%s'. Please ensure Pip is installed and"
+ " available in your PATH.",
command),
ex);
}
}
return String.format("%s;;%s", python, pip);
}
}
Loading
Loading