Credentials for different URLs isn't possible if they are under the same domain

[edcope-ct]

Problem

We have a buildstream project that uses sources from private repositories.

These repositories require different credentials, but they are all under the same domain.
This is a problem, as netrc matches domain name but not url paths.

There needs to be a different way to authenticate based on URLs that isn't using netrc.

Potential solution

URLs could be matched based on globs. Services such as Gitlab lets you create tokens for a group, where any repositories under the group can use that token to pull the source or LFS tarballs.

These entries could be stored in user configuration, and I suggest prioritising authentication with it before using netrc as a fallback.
This wouldn't create compatibility issues, and allows for per-url authentication while still keeping a netrc file.

Example:

# URL will be matched against each entry. If URL matches an entry, those credentials are used.
authentication:
  # Match repository
  - url: "https://example.com/group1/repo1"
    username: "user1"
    password: "pass1"
  # Match repository group
  - url: "https://example.com/group1/*"
    username: "user2"
    password: "pass2"
  # Example of matching Gitlab LFS URL
  # Note that escape sequence is used to escape the ? character
  - url: 'https://example.com/api/v4/projects/group1%2Fnestedgroup%2Frepo/repository/files/lfs_object.tar.gz/raw\?ref=*&lfs=true'
    username: "user3"
    password: "user3"

Additional notes for plugins

This solution would allow for plugins to also authenticate using user configuration, without having to implement a custom solution for authenticating per-url.

The following merge request was created by me for buildstream-plugins-community, where I implement a solution similar to what I've described above, but credentials are obtained from ~/.gitconfig:

https://gitlab.com/BuildStream/buildstream-plugins-community/-/merge_requests/445.

[abderrahim]

Ultimately, this is not a buildstream problem. It's something for the source plugins to implement. That said, it would be indeed nice to have some guidance from buildstream core on how to implement that so that plugins follow a common configuration format.

For now, the buildstream kinda recommends netrc through its use by DownloadableFileSource.

For the proposed solution, I don't like that it means starting to have secrets be part of the user configuration. It would be better to keep the secrets separate, which is what netrc gives. It also means adding a new API to buildstream, which will probably take time to get a release and make use of it.

Another idea that could work is implementing bazel's credential helpers specification. Buildbox also implements it https://gitlab.com/BuildGrid/buildbox/buildbox/-/merge_requests/1115, and it's probably a reasonable specification for both sources and REAPI cache/remote execution servers to use.

It means that buildstream would offload the authentication mechanism to a different program, but it also means that buildstream doesn't need to care about supporting multiple urls under the same host that use different authentication mechanisms.

[edcope-ct]

I brought this up here as one of the sources used is tar.

We cannot use multiple tokens with .netrc as one token is required by a tar source, while another token is required to pull git repositories with git_repo source from buildstream-community-plugins.

With buildstream and plugins reading credentials from .netrc, this makes it impossible to use multiple tokens.

Yes I agree this will take some time to land and be part of a release. We are implementing a workaround on our project to use a single token for now, as we cannot compile the project on CI/CD otherwise.

I will take a look at the credential helpers specs, thanks for the suggestion.

[edcope-ct]

I had a look at the spec. Yes this seems reasonable to me as well.

[edcope-ct]

Here is an proposal for implementing credential-helper-spec.

This follows the spec defined in https://github.com/EngFlow/credential-helper-spec/blob/main/spec.md

This spec is used by existing tools such as bazel and buildbox: https://github.com/bazelbuild/proposals/blob/main/designs/2022-06-07-bazel-credential-helpers.md https://gitlab.com/BuildGrid/buildbox/buildbox/-/merge_requests/1115

To implement the spec, buildstream should provide:

• A bst flag --credential-helper <PATH> to specify the credential helper executable
• BST user config entry auth.credential-helper = "<PATH>"
• API for plugins to use for retrieving credentials

For the credential helper, buildstream should:

• Timeout after 5 seconds of execution
• Provide the helper executable the same environment variables as bst
• Set the helper executable working directory to be the repository root

The API would be a utility function get_credentials(uri: str) -> dict[str, Any]:

• It is a new, isolated function rather than changes to DownloadableFileSource.
  • There are plugin sources that pulls from remote (such as git_repo) that doesn't derive of off DownloadableFileSource
• The function will attempt authentication with a credential helper, and fallback to netrc.
• This function would have to sometimes cache the result as specified in the get-credentials response schema: https://github.com/EngFlow/credential-helper-spec/blob/main/schemas/get-credentials-response.schema.json
  • Cache would be stored in a global state
• Return value is a dictionary containing headers for making requests with.
• This change will not break existing projects using netrc for authentication.
• This function should be used within buildstream as well, for anywhere that previously only used netrc for credentials.