analyze: How to exclude dependencies when run analyze

[LizZhang315]

Describe the bug

1. Error1: run analyze in go project, go list output include my go project, and ort has a error:
   java.lang.IllegalArgumentException: A replace directive references a module in '/Users/mac/IdeaProjects/myproject' outside of analysis root which is not supported.

2. Error2: run ort analyze in my project root dir with "-i /myproject/root/dir", ort will do nothing and has a log "No definition files found." But there has a "go.mod" file of my project.

3. Error3: run ort analyze will download module of indirect module dependencies, is it right? The module is "git.sr.ht/~sbinet/gg", and ort will request "https://git.sr.ht/~sbinet/gg" to download the code, but the website is no longer accessible. How can I exclude this module when run analyze?

To Reproduce

Steps to reproduce the behavior:

use ort-71.5.0
run cmd:
cd /Users/mac/IdeaProjects/myproject
ort -P ort.analyzer.allowDynamicVersions=true --debug --stacktrace analyze -i /Users/mac/IdeaProjects/myproject/go.mod -o /Users/mac/IdeaProjects/myproject/ort/ort-analyzer

Expected behavior

run success

Console / log output

15:51:54.227 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'go list -deps -json=Module -buildvcs=false ./...' in '/Users/mac/IdeaProjects/myproject'...
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {}
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - [...skipping 15755 lines...]
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - }
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -      "Module": {
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -              "Path": "private-git-address/org/myproject/v2",
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -              "Main": true,
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -              "Dir": "/Users/mac/IdeaProjects/myproject",
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -              "GoMod": "/Users/mac/IdeaProjects/myproject/go.mod",
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -              "GoVersion": "1.24"
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -      }
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - }
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -      "Module": {
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -              "Path": "private-git-address/org/myproject/v2",
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -              "Main": true,
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -              "Dir": "/Users/mac/IdeaProjects/myproject",
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -              "GoMod": "/Users/mac/IdeaProjects/myproject/go.mod",
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -              "GoVersion": "1.24"
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -      }
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - }
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - 
15:51:54.535 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - (Above output is limited to each 20 heading and tailing lines.)
java.lang.IllegalArgumentException: A replace directive references a module in '/Users/mac/IdeaProjects/myproject' outside of analysis root which is not supported.
        at org.ossreviewtoolkit.plugins.packagemanagers.go.GoMod.toId(GoMod.kt:302)
        at org.ossreviewtoolkit.plugins.packagemanagers.go.GoMod.resolveDependencies(GoMod.kt:164)
        at org.ossreviewtoolkit.analyzer.PackageManager.resolveDependencies(PackageManager.kt:323)
        at org.ossreviewtoolkit.analyzer.PackageManagerRunner$run$3.invokeSuspend(Analyzer.kt:354)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
        at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)
        at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)
15:51:55.443 [DefaultDispatcher-worker-1] ERROR org.ossreviewtoolkit.analyzer.PackageManager - GoMod failed to resolve dependencies for path '.': IllegalArgumentException: A replace directive references a module in '/Users/mac/IdeaProjects/myproject' outside of analysis root which is not supported.
15:51:55.447 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.analyzer.PackageManager - GoMod resolved dependencies for path '.' in 25m 21.434117167s.
15:51:55.449 [DefaultDispatcher-worker-1] INFO  kotlinx.coroutines.CoroutineScope - Finished GoMod analysis.
15:51:55.454 [main] INFO  org.ossreviewtoolkit.analyzer.Analyzer - Calling after resolution hooks for 1 manager(s).
15:51:55.454 [main] INFO  org.ossreviewtoolkit.analyzer.Analyzer - Building analyzer result.
15:51:55.493 [main] INFO  org.ossreviewtoolkit.utils.config.ConfigurationResolver - Getting 0 package curation(s) from provider 'RepositoryConfiguration' took 295.542us.
15:51:55.493 [main] INFO  org.ossreviewtoolkit.utils.config.ConfigurationResolver - Getting 0 package curation(s) from provider 'DefaultDir' took 1.041us.
15:51:55.493 [main] INFO  org.ossreviewtoolkit.utils.config.ConfigurationResolver - Getting 0 package curation(s) from provider 'DefaultFile' took 791ns.
15:51:55.493 [main] INFO  org.ossreviewtoolkit.utils.config.ConfigurationResolver - Getting 0 package curation(s) from provider 'Spring' took 8.667us.
Wrote analyzer result to '/Users/mac/IdeaProjects/myproject/ort/ort-analyzer/analyzer-result.yml' (0.01 MiB) in 115.285208ms.
15:51:55.652 [main] DEBUG org.ossreviewtoolkit.plugins.commands.api.utils.ExtensionsKt - Output ORT result file has SHA-1 hash a34cd1448232e768d9a7c83273768a050d30ee82.
The analysis took 25m 21.497247s.
Found 0 project(s) and 0 package(s) in total (not counting excluded ones).
Applied 0 curation(s) from 0 of 4 provider(s).
Resolved issues: 0 errors, 0 warnings, 0 hints.
Unresolved issues: 1 error, 0 warnings, 0 hints.
There is 1 unresolved issue with a severity equal to or greater than the WARNING threshold.
15:51:55.675 [ForkJoinPool-1-worker-1] DEBUG org.eclipse.jgit.internal.util.ShutdownHook - Cleanup org.eclipse.jgit.util.FS$FileStoreAttributes$$Lambda/0x00000070014aa810@2479a3f9 during JVM shutdown

Environment

Output of the ort requirements command (ensure to remove any sensitive information manually):

<copy & paste console output to here; no screenshots please>

Or manually specify:

• ORT version:71.5.0
• Java version: jdk21
• OS:Mac M2 pro

And specify (relevant parts of) your ORT configuration (config.yml):

ort:
  forceOverwrite: true
  enableRepositoryPackageCurations: true
  enableRepositoryPackageConfigurations: true
  analyzer:
    skipExcluded: true
    allow_dynamic_versions: true
    enabled_package_managers: [ GoMod]
  downloader:
    skipExcluded: true
    allowMovingRevisions: true
  scanner:
    skipExcluded: true
    config:
      ScanCode:
        options:
          # commandLine: '--copyright,--license,--info,--strip-root,--timeout,300'
          commandLine: "--copyright --license --info --strip-root --timeout 300 --ignore *.ort.yml --ignore HERE_NOTICE --ignore META-INF/DEPENDENCIES --ignore tests/**"
  allowedProcessEnvironmentVariableNames:
    - GO111MODULE
    - GOPROXY
    - GOPRIVATE
    - GONOSUMDB

Additional context

.ort.yml

analyzer:
  skip_excluded: true
includes:
  paths:
    - pattern: "pkg/**"
      reason: "SOURCE_OF"
excludes:
  paths:
    - pattern: "mocks/**"
      reason: "TEST_TOOL_OF"
      comment: "This directory contains test data which are not distributed."
    - pattern: "*_test.go"
      reason: "TEST_OF"
    - pattern: "test_*.go"
      reason: "TEST_OF"
  scopes:
    - pattern: "vendor"
      reason: "DEV_DEPENDENCY_OF"
      comment: "Packages to build and test the main module."
curations:
  packages:
    - id: "Go::go.etcd.io/etcd@*"
      curations:
        vcs:
          type: "Git"
          url: "https://github.com/etcd-io/etcd"
          revision: "v3.3.17"
    - id: "Go::go.etcd.io/etcd/api/v3@*"
      curations:
        vcs:
          type: "Git"
          url: "https://github.com/etcd-io/etcd"
          path: "api"
          revision: "api/v3.5.9"
    - id: "Go::go.etcd.io/etcd/client/pkg/v3@*"
      curations:
        vcs:
          type: "Git"
          url: "https://github.com/etcd-io/etcd"
          path: "client/pkg"
          revision: "client/pkg/v3.5.9"
    - id: "Go::go.etcd.io/etcd/client/v2@*"
      curations:
        vcs:
          type: "Git"
          url: "https://github.com/etcd-io/etcd"
          path: "client/v2"
          revision: "client/v2.305.9"
    - id: "Go::go.etcd.io/etcd/client/v3@*"
      curations:
        vcs:
          type: "Git"
          url: "https://github.com/etcd-io/etcd"
          path: "client/v3"
          revision: "client/v3.5.9"
    - id: "Go::go.etcd.io/etcd/client/v3@*"
      curations:
        vcs:
          type: "Git"
          url: "https://github.com/etcd-io/etcd"
          path: "client/v3"
          revision: "client/v3.6.6"
    - id: "Go::go.etcd.io/etcd/pkg/v3@*"
      curations:
        vcs:
          type: "Git"
          url: "https://github.com/etcd-io/etcd"
          revision: "pkg/v3.5.9"
          path: "pkg"
    - id: "Go::go.etcd.io/etcd/raft/v3@*"
      curations:
        vcs:
          type: "Git"
          url: "https://github.com/etcd-io/etcd"
          revision: "raft/v3.5.9"
          path: "raft"
    - id: "Go::go.etcd.io/etcd/server/v3@*"
      curations:
        vcs:
          type: "Git"
          url: "https://github.com/etcd-io/etcd"
          revision: "server/v3.5.9"
          path: "server"
    - id: "Go::gopkg.in/yaml.v3:3.0.1"
      curations:
        homepage_url: "https://gopkg.in/yaml.v3"
        vcs:
          type: "Git"
          url: "https://github.com/go-yaml/yaml/tree/v3.0.1"
resolutions:
  issues:
    - message: "IOException: Could not resolve provenance for package "
      reason: SCANNER_ISSUE
      comment: "This is a false positive"
    - message: "A replace directive references a module in "
      reason: BUILD_TOOL_ISSUE
      comment: "This is a false positive"
package_configurations:
   - id: "GoMod::myproject:19c32c70d2ad95cb9c4eee37236e1d297ca0f938"
     vcs:
       type: "Git"
       url: "https://<private-git>/<org>/<myrpoject>"
       revision: "19c32c70d2ad95cb9c4eee37236e1d297ca0f938"
     license_finding_curations:
     - path: "pkg/**"
       concluded_license: "Apache-2.0"
       reason: "NOT_DETECTED"

[fviernau]

@LizZhang315 - is the issue also reproducible if you analyze the directory instead of the file, e.g. with [1]?

[1] ort -P ort.analyzer.allowDynamicVersions=true --debug --stacktrace analyze -i /Users/mac/IdeaProjects/myproject -o /Users/mac/IdeaProjects/myproject/ort/ort-analyzer

[fviernau]

@LizZhang315 can you confirm that the following reproduces the same issue?

./cli/build/install/ort/bin/ort analyze -i ./plugins/package-managers/go/src/funTest/assets/projects/synthetic/gomod/subpkg/go.mod -o .

I believe this to be a general issue with analyzing a single definition file instead of a directory (more common). In case the
definition file references other definition files, the analysis fails. To me this actually makes sense, because
the references have to be resolved as well to some data and for doing so, the target definition files have to be analyzed too.

I wonder if it would make sense to adjust the documentation, to say that when analyzing a single file, there must not be references to other definition files in order for it to work. What do you think @oss-review-toolkit/core-devs ?

[sschuberth]

I wonder if it would make sense to adjust the documentation, to say that when analyzing a single file, there must not be references to other definition files in order for it to work. What do you think @oss-review-toolkit/core-devs ?

I agree to that.

[sschuberth]

2. Error2: run ort analyze in my project root dir with "-i /myproject/root/dir", ort will do nothing and has a log "No definition files found." But there has a "go.mod" file of my project.

If you use -i /Users/mac/IdeaProjects/myproject instead of -i /Users/mac/IdeaProjects/myproject/go.mod that should not happen.

[sschuberth]

3. Error3: run ort analyze will download module of indirect module dependencies, is it right?

Yes.

The module is "git.sr.ht/~sbinet/gg", and ort will request "https://git.sr.ht/~sbinet/gg" to download the code, but the website is no longer accessible.

Actually, https://git.sr.ht/~sbinet/gg works just fine for me.

How can I exclude this module when run analyze?

There are means to deals with this, like package curations, but it seems this is not needed here.