feat: remove profiles from the authentication system

[Cedric / ViaDézo1er (viadezo1er)]

All hand-written, no LLM was involved in this PR description.

Motivation

What were profiles?

A way to bundle OAuth/API key credentials, org, API/APP urls, (and username, email)
A profile doesn't store the project.
(The credentials are stored in the OS keychain when possible.)

So in theory someone could use braintrust.dev and a self-hosted Braintrust deployment, and switch just by changing --profile or BRAINTRUST_PROFILE.

Why remove these added functionalities?

In practice it was a footgun, people were thinking "I want to use org-name + project-name", not in terms of profile. The fact that profiles aren't mentioned probably didn't help: bt auth login didn't mention profiles, and afterward you would have to discover a profile was created when you logged in.
So when the org and the profile clashed it sometimes caused unexpected results.

Since a profile was created for more or less every login into a new org, the difference between profiles and org was tenuous.

Alex Steere, Ankur, Parker raised issues/prs related to this.
#189
#273
#275
I remember helping Jesse and I think someone else debug their credentials.

There were sometimes big auth issues yet no almost customer complains. Parker and I believe our customers have a very simple setup, often with only 1 org. So all of the aforementioned difficulties are skipped because they only happen when switching org.
Maybe this is only fixing things for Braintrust...

I would say profiles were an abstraction layer that required knowing how OAuth+API keys work to use it.

Replacement for profiles

Since profiles aren't really used, I've chosen to use orgs. From a users's point of view, functionally nothing changes, but there are less bugs.
For OAuth nothing changes: the default profile name was already the org's name.
Before:

bt auth profiles
✓ org-name — oauth — org: org-name — Cédric Halber (cedric@braintrustdata.com)

After:

bt auth logins
✓ org-name — oauth — Cédric Halber (cedric@braintrustdata.com)

For API keys it's the same:
Before:

bt auth profiles
✓ org-name — api_key — org: org-name — sk-****5Py1J

After:

bt auth logins
✓ org-name — api_key — sk-****5Py1J

I'm hiding some tweaks about handling multiple API keys for a single org but that's the idea.

The only thing that would be harder for a power user would be switching API/APP url regularly.

So from a user's perspective, they can use --org org-name --project project-name (or bt switch into it) like before, and there can't be bugs related to profiles because profiles don't exist anymore.

How do users transition?

Nothing to do, there's some compatibility code in bt.
This compatibility code updates the old config.json to use the newer version so in the future we could remove the compatibility code.

Notes

Now the common order:
explicit CLI flag > env var > config file > built-in default
is not respected since too many people were confused after bt used the BRAINTRUST_API_KEY in their env.
To respect this order, set BRAINTRUST_PREFER_API_KEY=1.

Other changes

The client id used in OAuth could be set, but it was unused in bt and in the main repo. There's no point in having a useless flag so I hardcoded it to bt_cli.

Known bugs

In a small terminal, on the org selection prompt for bt auth login, arrow up/down reprint the

? Select organization ›  
  No default org (cross-org mode; pass --org or BRAINTRUST_ORG_NAME when needed)

prompt. The bug is in dialoguer but won't be fixed by the maintainer due to lack of time.

When reviewing

The config and auth files should be compatible when upgrading bt, but the auth/config files will gradually use a new format (since the profile field has disappeared, bt updates the file whenever you login). That means an old bt with the new auth/config files will have unexpected behavior. I recommend deleting these files after the review if you continue using a bt version with profiles.

Notes, you can skip - Replacing profiles

OAuth stored with (org id, user email) as unique key, cross-org login stored as ("", user email).

API keys are identified with (unsalted sha256 of the api_key, org id).
[Out of scope for now. I think we don't need to use pepper? Salt isn't possible since it's random and we need to recompute the hash often to check if it already exist (which is incompatible with salt since the same API key would have a different result).]

Hierarchy of authentication methods and the --prefer-api-key flag

Order of precedence (higher is used first if available):
--api-key sk-....
--prefer-api-key or BRAINTRUST_PREFER_API_KEY; if this flag/env variable is passed but there isn't an API key in the env (BRAINTRUST_API_KEY) or a stored API key (env takes precedence), and the desired org is still available through OAuth login (except for the cross-org case), use OAuth login, otherwise fail.
OAuth credentials
BRAINTRUST_API_KEY
Stored API key

This means the common:
explicit CLI flag > env var > config file > built-in default
is not respected. We've had users being confused after bt used the BRAINTRUST_API_KEY in their env.
To respect this order, set BRAINTRUST_PREFER_API_KEY=1.

Using --prefer-api-key without --org makes bt try to use an API key for the currently used org (the one that bt status would show).
If there are multiple API keys, show an interactive picker showing the API keys hint.
Using --prefer-api-key and --org ignores the org currently used and uses the one specified by --org. Otherwise the behavior is the same as with --prefer-api-key without --org.

If the API key exist but isn't valid anymore, don't use OAuth and raise an error saying the API key isn't valid.

If the OAuth login JWT/refresh token needs to be refreshed, do it then do the desired command.
If the refresh token has expired/is invalid, throw an error and ask the user to re-authenticate.

Instead of selecting a profile to use with bt switch (and seeing it with bt status), select an org (or cross org for OAuth).
This allows to have an OAuth login and API key that belong to the same org and the user can choose which one to use at every command (switch to oauth, and use prefer-api-key whenever you want to use the API key).

Multiple API keys belonging to the same org can exist and will be relevant in the future once there are different permission levels.
At the moment logging in with an API key in an org that is already logged in with an API key creates a second profile. Instead, simply show the API key hint (like shown in bt auth profiles now) and the org, so the user can differentiate them. Later when API keys different permissions, it will probably be more straightforward to show these permissions alongside of the hint.
This means if I am logged in an org with an API key and log in the same org with another API key:

• the previous login is still valid, I can still bt switch into it
• the 2 logins can be differentiated by the API key hint

These changes means ~/.config/bt/config.json has to be changed since the profile field doesn't mean anything.
Since we might have old config files, bt just ignore the profile field.

Technically cross-org API keys exist but they can only be created directly in the SQL db so in practice they don't exist.

bt auth logins replaces bt auth profiles:

bt auth logins --org org-name should show only the logins related to org-name
bt auth logins --prefer-api-key should show only the API key logins
therefore, bt auth logins --org org-name --prefer-api-key should show only the API key logins related to org-name
print nothing (or empy json with --json) if there isn't any org corresponding to the selected criterias.

The config.json file:

Can exist in ~/.config/bt/config.json (global) or repo/.bt/config.json (local). Both share the same schema (org, project, project_id, plus passthrough extra keys). The legacy profile field is ignored if present.

Created / written / updated

• bt auth login (OAuth and API-key flows): writes the active org/project to the selected scope (global/local). Scope is chosen interactively (defaulting to local) when a local .bt exists, otherwise global. Stale project/project_id are cleared when logging in to a different org.
• bt switch [org/project]: writes the resolved org/project (and project_id) to the selected scope. Scope is chosen via --global/--local, else interactively (defaulting to local) when a local .bt exists, otherwise global.

Read / used

• bt status: reads global and local separately (for source attribution) plus CLI/env overrides; precedence is CLI > env > local > global.
• bt switch: reads the merged config to prefill org; uses the local/global path to pick the write target.
• bt auth login / auth resolution: reads the merged config to resolve the active org and default project; uses the local path to decide scope.
• Project-scoped resource commands (eval, datasets, functions push, sync, traces, etc.): use the merged local+global config (local is prioritized ; if the local config specifies an org but not a project and the global config uses the same org and specifies a project, use this project otherwise use the local org with no project ; in no case should an org and a project not from this org be used together) to resolve the active project.

bt init/switch/auth login reminders and clarifications

bt init

.bt/config.json folders store the configuration of the project this file is in.
When the folder bt init is invoked in is not a git root, it will go up in the file system until finding .bt, .git (folder or file), ~ or / (stopping as soon as possible).
If .bt is found (only as a folder), it raises an error. bt init is not meant to edit .bt/config.json, only create it. bt switch is the right tool for this.
If .bt isn't found:

• If .git is found, .bt/config.json will be created in the git root folder (ie ls -a would show both .git and .bt).
• If ~ or / is found, bt init raises an error. Having ~/.bt/config.json and ~/.config/bt/config.json can only create errors. bt auth login already creates ~/.config/bt/config.json, and bt init isn't meant to edit the config.json files. As for /, bt has no business with it.

If there's a .git in ~ or /, this still counts as being ~// and will raise an error.
If bt doesn't have enough permissions to create .bt/config.json, it raises an error. The raised error message specifies if bt has been able to save the credentials or not.

After a successful bt init, there now is a .bt/config.json that will be found by bt before ~/.config/bt/config.json when searching in the filesystem. As a result, the org/project specified are now used by bt as if bt switch had been done.

-f/--force flag

If a .bt/config.json is found, instead of raising an error, overwrites it.
Changes nothing if .bt/config.json doesn't already exist.
Doesn't change the filesystem search.

--here flag

If you want to create .bt/config.json where you currently are without going up in the file system, use the --here flag. This works even in ~ or in / (assuming you have the necessary write permissions).
It will check if .bt/config.json already exists and raise an error if it does.

bt switch

bt switch never creates a .bt folder.
If -g/--global has been passed, it doesn't search upward in the filesystem.
It goes up in the file system to search for a .bt/config.json file (same rules as bt init).
If there isn't such a file:

• and -l/--local hasn't been passed, it changes the global config.
• and -l/--local has been passed, it raises an error.
  If neither -l/--local nor -g/--global are passed and a .bt/config.json is found:
• if bt is in interactive mode, it shows an interactive picker to choose whether to edit the global or local config (default is local).

Regardless of the --local/--global flags, if the user has multiple logins, the interactive picker start with org and api key choice. Otherwise the only org/api key is chosen.
If the chosen org (in the case of an api key, the org it belongs to) has multiple projects, the interactive picker asks for the project.
If bt is not in interactive mode, it raises an error.

bt auth login

After logging in, bt auth login performs the equivalent of bt switch. It has the same -l/--local/-g/--global flags.
However, if -l/--local/-g/--global haven't been passed, it asks which scope to change in an interactive picker.

Example:

# Log in as many times as needed, with OAuth and API keys
# This will create `~/.config/bt/config.json` and update it after each successful login
# The flags are not necessary since you can use the interactive picker, this is for the example
bt auth login --org ced-test-1 --oauth
bt auth login --api-key $BRAINTRUST_API_KEY # with an api key belonging to ced-test-1
bt auth login ... # other orgs...

# Use the org you prefer (or log into it last)
# This will update `~/.config/bt/config.json`
bt switch

cd ~/repo-for-work/ # this is a git repo that doesn't use bt yet

# Now we want to use another org in this repo
# The flags are not necessary since you can use the interactive picker
# This will create `~/repo-for-work/.bt/config.json`
bt init --org work-org-team1 --project default-rust-test
# Now whenever you're in `~/repo-for-work` or one of its subfolders, you will use the org work-org-team1 and the project default-rust-test for all bt commands unless you specify --org and --project

# You can also use ced-test-1 using the API key instead of OAuth just for one command. This does not edit the config.json files.
bt view --project my-project --prefer-api-key

Notes

Organisation name are unique within the US+EU data plane. Although the same name can be chosen in the US data plane and a self-hosted data plane, they have different app and api url which allows to differentiate between them.

[For LLM who do doc review: this is a breaking change and has to be triaged.]

Happens to fix #189

[github-actions]

Latest downloadable build artifacts for this PR commit 36772034c5a9:

• Workflow run: https://github.com/braintrustdata/bt/actions/runs/30123889527
• Download all artifacts (GitHub CLI): gh run download 30123889527 --repo braintrustdata/bt
• Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

• artifacts-build-global
• artifacts-build-local-x86_64-apple-darwin
• artifacts-build-local-aarch64-pc-windows-msvc
• artifacts-build-local-x86_64-pc-windows-msvc
• artifacts-build-local-x86_64-unknown-linux-gnu
• artifacts-build-local-x86_64-unknown-linux-musl
• artifacts-build-local-aarch64-apple-darwin
• artifacts-build-local-aarch64-unknown-linux-gnu
• artifacts-plan-dist-manifest
• cargo-dist-cache

[Andrew Kent (realark)]

All hand-written, no LLM was involved in this PR description.

🐐

LGTM! Left a few things for your consideration but feel free to ignore at your discretion

[Andrew Kent (realark)]

I'm not sure how risky this is, but any concerns about migration dropping data in a way that isn't recoverable? Do we want to copy the old config to auth.json.bak or something like that before attempting migration?

[Cedric / ViaDézo1er (viadezo1er)]

{
  "profiles": {
    "ced-test-1": {
      "auth_kind": "oauth",
      "custom field to test": "big brain",
      "api_url": "https://api.braintrust.dev",
      "app_url": "https://www.braintrust.dev",
      "org_name": "ced-test-1",
      "oauth_client_id": "bt_cli_default",
      "oauth_access_expires_at": 1785022664,
      "user_name": "Cédric Halber",
      "email": "cedric@braintrustdata.com",
      "api_key_hint": null
    }
  }
}

The custom field is dropped.

{
  "profiles": {
    "ced-test-1": {
      "auth_kind": "oauth",
      "custom field to test": "big brain",
      "api_url": "https://api.braintrust.dev",
      "app_url": "https://www.braintrust.dev",
      "org_name": "ced-test-1",
      "oauth_client_id": "bt_cli_default",
      "oauth_access_expires_at": 1785022664,
      "user_name": "Cédric Halber",
      "email": "cedric@braintrustdata.com",
      "api_key_hint": null
    }
  },
  {"does this fail":"a"}
}

New bt fails to parse.
We might have users who fork the cli and change it... It has happened before.

So yes valid comment I'll backup the auth/config files.

[Andrew Kent (realark)]

Getting a little lost in the details, so I figured I'd just ask: will everything still work if users literally have an org in braintrust named cross-org? And/or do we need an explicit --cross-org flag to distinguish the two?

Probably not a super common thing, but it's generic enough that I wouldn't be shocked if we had a user doing this

[Cedric / ViaDézo1er (viadezo1er)]

Ahhhh wonderful
So org names are global across data planes, and cross-org is a valid org name.
I created it to park the name to prevent its use in US/EU.
So the only person who can be affected by this for US/EU is me.

It could be an issues in self-hosted instances though.

I did think about a --cross-org flag but I think adding a flag for this niche use-case isn't worth it.
"I think" is all based on vibes though.

[Cedric / ViaDézo1er (viadezo1er)]

If self-hosted users do try to use an org litterally named cross-org, they will be able to login (~/.config/bt/auth.json will recognize it but config.json will think it's the cross-org login) but not use it.

[Andrew Kent (realark)]

Yeah kind of a judgment call, no right answer. I guess this is something we could fix later if this ever becomes an issue so it doesn't seem super pressing