diff --git a/skills/sdk-install/braintrust-url-formats.md b/skills/sdk-install/braintrust-url-formats.md new file mode 100644 index 0000000..55813bb --- /dev/null +++ b/skills/sdk-install/braintrust-url-formats.md @@ -0,0 +1,32 @@ +# Braintrust URL Formats + +## App Links (Current Format) + +### Experiments + +`https://www.braintrust.dev/app/{org}/p/{project}/experiments/{experiment_name}?r={root_span_id}&s={span_id}` + +### Datasets + +`https://www.braintrust.dev/app/{org}/p/{project}/datasets/{dataset_name}?r={root_span_id}` + +### Project Logs + +`https://www.braintrust.dev/app/{org}/p/{project}/logs?r={root_span_id}&s={span_id}` + +## Legacy Object URLs + +`https://www.braintrust.dev/app/object?object_type=...&object_id=...&id=...` + +## URL Parameters + +| Parameter | Description | +| --------- | --------------------------------------------------------- | +| r | The root_span_id - identifies a trace | +| s | The span_id - identifies a specific span within the trace | +| id | Legacy parameter for root_span_id in object URLs | + +## Notes + +- The `r=` parameter is always the root_span_id +- For logs and experiments, use `s=` to reference a specific span within a trace diff --git a/skills/sdk-install/csharp.md b/skills/sdk-install/csharp.md new file mode 100644 index 0000000..e301469 --- /dev/null +++ b/skills/sdk-install/csharp.md @@ -0,0 +1,155 @@ +# C# SDK Install + +Reference guide for installing the Braintrust C# SDK. + +- SDK repo: https://github.com/braintrustdata/braintrust-sdk-dotnet +- NuGet: https://www.nuget.org/packages/Braintrust.Sdk +- Requires .NET 8.0+ + +## Find the latest version of the SDK + +Look up the latest version from NuGet **without installing anything**. Do not guess -- use a read-only query so the environment stays unchanged until you pin the exact version. + +```bash +dotnet package search Braintrust.Sdk --exact-match +``` + +Then install that exact version: + +### .NET CLI + +```bash +dotnet add package Braintrust.Sdk --version +``` + +### Or add to .csproj + +```xml + + + +``` + +## Initialize the SDK + +```csharp +using Braintrust.Sdk; +using Braintrust.Sdk.Config; + +var apiKey = Environment.GetEnvironmentVariable("BRAINTRUST_API_KEY"); +Braintrust? braintrust = null; +System.Diagnostics.ActivitySource? activitySource = null; + +if (!string.IsNullOrEmpty(apiKey)) +{ + // Set the project name in code (do NOT require an env var for project name). + var config = BraintrustConfig.Of( + ("BRAINTRUST_API_KEY", apiKey), + ("BRAINTRUST_DEFAULT_PROJECT_NAME", "my-project") + ); + + braintrust = Braintrust.Get(config); + activitySource = braintrust.GetActivitySource(); +} +``` + +`Braintrust.Get(config)` is the main entry point. The SDK requires an API key to be present, so initialize Braintrust conditionally and run the application normally when `BRAINTRUST_API_KEY` is missing. `GetActivitySource()` returns the `System.Diagnostics.ActivitySource` used to create spans. + +## Install instrumentation + +The C# SDK instruments LLM clients by wrapping them. Only instrument clients that are actually present in the project. + +### OpenAI (`OpenAI` NuGet package) + +```bash +dotnet add package OpenAI +``` + +Create an instrumented OpenAI client: + +```csharp +using Braintrust.Sdk; +using Braintrust.Sdk.Config; +using Braintrust.Sdk.Instrumentation.OpenAI; + +var btApiKey = Environment.GetEnvironmentVariable("BRAINTRUST_API_KEY"); +var openAIApiKey = Environment.GetEnvironmentVariable("OPENAI_API_KEY"); + +if (!string.IsNullOrEmpty(btApiKey) && !string.IsNullOrEmpty(openAIApiKey)) +{ + var config = BraintrustConfig.Of( + ("BRAINTRUST_API_KEY", btApiKey), + ("BRAINTRUST_DEFAULT_PROJECT_NAME", "my-project") + ); + var braintrust = Braintrust.Get(config); + var activitySource = braintrust.GetActivitySource(); + var client = BraintrustOpenAI.WrapOpenAI(activitySource, openAIApiKey); + + // Optional: create a root activity so you can generate a permalink. + using var activity = activitySource.StartActivity("braintrust-openai-example"); + + var chatClient = client.GetChatClient("gpt-5-mini"); + var response = await chatClient.CompleteChatAsync( + new ChatMessage[] + { + new SystemChatMessage("You are a helpful assistant."), + new UserChatMessage("What is the capital of France?") + } + ); + + if (activity != null) + { + var projectUri = await braintrust.GetProjectUriAsync(); + var url = $"{projectUri}/logs?r={activity.TraceId}&s={activity.SpanId}"; + Console.WriteLine($"View your data in Braintrust: {url}"); + } +} +``` + +### Custom spans + +For business logic that isn't an LLM call, create spans manually with the `ActivitySource`: + +```csharp +using (var activity = activitySource.StartActivity("my-operation")) +{ + activity?.SetTag("some.attribute", "value"); + // LLM calls inside here are automatically nested under this span +} +``` + +## Run the application + +Try to figure out how to run the application from the project structure: + +- **dotnet run**: `dotnet run` or `dotnet run --project path/to/Project.csproj` +- **ASP.NET**: `dotnet run` (typically starts Kestrel) +- **Published app**: `dotnet path/to/app.dll` +- **Visual Studio / Rider**: run from IDE + +If you can't determine how to run the app, ask the user. + +## Generate a permalink (required) + +The installer must produce a permalink to the emitted trace/logs in its final output. + +In .NET, the most reliable permalink can be generated from the root Activity's TraceId/SpanId: + +```csharp +if (braintrust != null && activitySource != null) +{ + using var activity = activitySource.StartActivity("braintrust-install-verify"); + if (activity != null) + { + // Perform a real operation that triggers LLM spans / instrumentation here. + + var projectUri = await braintrust.GetProjectUriAsync(); + var url = $"{projectUri}/logs?r={activity.TraceId}&s={activity.SpanId}"; + Console.WriteLine($"View your data in Braintrust: {url}"); + } +} +``` + +The final assistant response must include the printed URL. + +If the SDK-generated URL is not available, construct the permalink manually using the URL format documented in `braintrust-url-formats.md` as described in the agent task (Step 5). diff --git a/skills/sdk-install/go.md b/skills/sdk-install/go.md new file mode 100644 index 0000000..d5b5fb7 --- /dev/null +++ b/skills/sdk-install/go.md @@ -0,0 +1,130 @@ +# Go SDK Install + +Reference guide for installing the Braintrust Go SDK. + +- SDK repo: https://github.com/braintrustdata/braintrust-sdk-go +- pkg.go.dev: https://pkg.go.dev/github.com/braintrustdata/braintrust-sdk-go +- Requires Go 1.22+ + +## Install the SDK + +```bash +go get github.com/braintrustdata/braintrust-sdk-go +``` + +## Initialize the SDK + +Every Go project needs OpenTelemetry setup and a Braintrust client. + +```go +package main + +import ( + "context" + "log" + + "github.com/braintrustdata/braintrust-sdk-go" + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/sdk/trace" +) + +func main() { + ctx := context.Background() + + tp := trace.NewTracerProvider() + defer tp.Shutdown(ctx) + otel.SetTracerProvider(tp) + + _, err := braintrust.New(tp, braintrust.WithProject("my-project")) + if err != nil { + log.Fatal(err) + } +} +``` + +`braintrust.New` reads `BRAINTRUST_API_KEY` from the environment automatically. + +## Install instrumentation + +The Go SDK uses [Orchestrion](https://github.com/DataDog/orchestrion) to automatically inject tracing at compile time -- no wrapper code needed in the application. + +**1. Install orchestrion:** + +```bash +go install github.com/DataDog/orchestrion@v1.6.1 +``` + +**2. Create `orchestrion.tool.go` in the project root:** + +To instrument all supported providers: + +```go +//go:build tools + +package main + +import ( + _ "github.com/DataDog/orchestrion" + _ "github.com/braintrustdata/braintrust-sdk-go/trace/contrib/all" +) +``` + +Or import only the integrations the project actually uses: + +```go +//go:build tools + +package main + +import ( + _ "github.com/DataDog/orchestrion" + _ "github.com/braintrustdata/braintrust-sdk-go/trace/contrib/anthropic" // anthropic-sdk-go + _ "github.com/braintrustdata/braintrust-sdk-go/trace/contrib/genai" // Google GenAI + _ "github.com/braintrustdata/braintrust-sdk-go/trace/contrib/github.com/sashabaranov/go-openai" // sashabaranov/go-openai + _ "github.com/braintrustdata/braintrust-sdk-go/trace/contrib/langchaingo" // LangChainGo + _ "github.com/braintrustdata/braintrust-sdk-go/trace/contrib/openai" // openai-go +) +``` + +**3. Build with orchestrion:** + +```bash +orchestrion go build ./... +``` + +Or set GOFLAGS to use orchestrion automatically: + +```bash +export GOFLAGS="-toolexec='orchestrion toolexec'" +go build ./... +``` + +After this, LLM client calls are automatically traced with no code changes. + +### Supported providers + +Orchestrion supports these providers (import the corresponding `trace/contrib/` package in `orchestrion.tool.go`): + +| Provider | Import path | +| ---------------------- | --------------------------------------------------------------------------------------------- | +| OpenAI (`openai-go`) | `github.com/braintrustdata/braintrust-sdk-go/trace/contrib/openai` | +| Anthropic | `github.com/braintrustdata/braintrust-sdk-go/trace/contrib/anthropic` | +| Google GenAI / Gemini | `github.com/braintrustdata/braintrust-sdk-go/trace/contrib/genai` | +| LangChainGo | `github.com/braintrustdata/braintrust-sdk-go/trace/contrib/langchaingo` | +| sashabaranov/go-openai | `github.com/braintrustdata/braintrust-sdk-go/trace/contrib/github.com/sashabaranov/go-openai` | +| All of the above | `github.com/braintrustdata/braintrust-sdk-go/trace/contrib/all` | + +## Run the application + +Try to figure out how to run the application from the project structure: + +- **go run**: `go run .` or `go run ./cmd/myapp` +- **Orchestrion**: `orchestrion go run .` +- **Makefile**: check for `run`, `serve`, or similar targets +- **Docker**: check for a `Dockerfile` + +If you can't determine how to run the app, ask the user. + +## Generate a permalink (required) + +Follow the permalink generation steps in the agent task (Step 5). Use the value passed to `braintrust.WithProject(...)` as the project name. diff --git a/skills/sdk-install/instrument-task.md b/skills/sdk-install/instrument-task.md new file mode 100644 index 0000000..df53a57 --- /dev/null +++ b/skills/sdk-install/instrument-task.md @@ -0,0 +1,130 @@ +# Braintrust SDK Installation (Agent Instructions) + +## Hard Rules + +{RUN_MODE_CONTEXT} + +- **Only add Braintrust code.** Do not refactor or modify unrelated code. +- **Pin exact versions.** Never use `latest`. +- **Set the project name in code.** Do NOT configure project name via env vars. +- **App must run without Braintrust.** If `BRAINTRUST_API_KEY` is missing at runtime, do not crash. +- **Abort install if API key is not set.** (Do not modify runtime behavior.) +- **Do not guess APIs.** Use official documentation/examples only. +- **Do not add eval code** unless explicitly requested. +- **Do not add manual flush/shutdown logic.** +- **If SDK is already installed/configured, do not duplicate work.** + +--- + +## Execution Requirements + +Before writing any code: + +1. Create a **checklist** from the steps below. +2. Execute each step in order. +3. Do not skip steps. + +--- + +## Steps + +{LANGUAGE_CONTEXT} + +### 1. Verify API Key (Install Precondition) + +Check if `BRAINTRUST_API_KEY` is exported: + +```bash +if env | grep 'BRAINTRUST_API_KEY=' >/dev/null 2>&1 ; then echo "api key set" ; else echo "api key NOT set"; fi +``` + +If not set, **abort installation immediately**. + +--- + +### 2. Detect Language + +Determine the project language using concrete signals: + +- `package.json` → TypeScript +- `requirements.txt` or `pyproject.toml` → Python +- `pom.xml` or `build.gradle` → Java +- `go.mod` → Go +- `Gemfile` → Ruby +- `.csproj` → C# + +If the language is not obvious from standard build/dependency files: + +- infer it from concrete repo evidence (e.g., entrypoint file extensions, build scripts, framework config) +- State the single strongest piece of evidence you used +- If still ambiguous (polyglot/monorepo), ask the user which service/app to instrument and wait for the response before proceeding +- If the inferred language is not in the supported list, **abort the install**. + +If none match, **abort installation**. + +--- + +### 3. Install SDK (Language-Specific) + +Read the install guide for the detected language from the local docs: + +| Language | Local doc | +| ---------- | --------------------------------- | +| Java | `{SDK_INSTALL_DIR}/java.md` | +| TypeScript | `{SDK_INSTALL_DIR}/typescript.md` | +| Python | `{SDK_INSTALL_DIR}/python.md` | +| Go | `{SDK_INSTALL_DIR}/go.md` | +| Ruby | `{SDK_INSTALL_DIR}/ruby.md` | +| C# | `{SDK_INSTALL_DIR}/csharp.md` | + +Requirements: + +- Pin an exact SDK version (resolve via package manager). +- Modify only dependency files and a minimal application entry point (e.g., main/bootstrap). +- Do not change unrelated code. + +--- + +### 4. Verify Installation (MANDATORY) + +- Run the application. +- Confirm at least one log/trace is emitted to Braintrust. +- Confirm no runtime errors. +- Confirm the app still runs if `BRAINTRUST_API_KEY` is unset. + +If you do not know how to run the app, ask the user and wait for the response before proceeding. + +--- + +### 5. Verify in Braintrust (CRITICAL) + +The permalink must be included in the final output. This confirms the full installation succeeded. + +The project must be set in code during installation — do not guess the project name from context. + +**How to obtain the permalink:** + +Most language SDKs print a direct URL to the emitted trace after the app runs. Capture that URL and print it. + +If the SDK does not print a URL, construct one manually using the URL format documented in `{SDK_INSTALL_DIR}/braintrust-url-formats.md`: + +``` +https://www.braintrust.dev/app/{org}/p/{project_name}/logs?r={root_span_id} +``` + +- `org`: your Braintrust organization slug +- `project_name`: the project name set in code +- `root_span_id`: the trace/span ID returned or logged by the SDK + +--- + +### 6. Final Summary + +Summarize: + +- What SDK version was installed +- Where code was modified +- What logs/traces were emitted +- The Braintrust permalink (required) + +{WORKFLOW_CONTEXT} diff --git a/skills/sdk-install/java.md b/skills/sdk-install/java.md new file mode 100644 index 0000000..5b2c91b --- /dev/null +++ b/skills/sdk-install/java.md @@ -0,0 +1,161 @@ +# Java SDK Install + +Reference guide for installing the Braintrust Java SDK. + +- SDK repo: https://github.com/braintrustdata/braintrust-sdk-java +- Maven Central: https://central.sonatype.com/artifact/dev.braintrust/braintrust-sdk-java/versions +- Requires Java 17+ + +## Find the latest version of the SDK + +Look up the latest version from Maven Central **without modifying the project**. Do not guess -- use a read-only query so dependencies stay unchanged until you pin the exact version. + +```bash +curl -s 'https://search.maven.org/solrsearch/select?q=g:dev.braintrust+AND+a:braintrust-sdk-java&rows=1&wt=json' | python3 -c "import sys,json; print(json.load(sys.stdin)['response']['docs'][0]['latestVersion'])" +``` + +Then add the dependency with that exact version: + +### Gradle + +```groovy +dependencies { + implementation 'dev.braintrust:braintrust-sdk-java:' +} +``` + +### Maven + +```xml + + dev.braintrust + braintrust-sdk-java + + +``` + +### SBT + +```scala +libraryDependencies += "dev.braintrust" % "braintrust-sdk-java" % "" +``` + +### Generic fallback + +If the project uses a different build tool, the Maven coordinates are: + +- Group: `dev.braintrust` +- Artifact: `braintrust-sdk-java` + +## Initialize the SDK + +```java +import dev.braintrust.Braintrust; +import dev.braintrust.config.BraintrustConfig; + +var config = BraintrustConfig.builder() + .defaultProjectName("my-project") + .build(); +var braintrust = Braintrust.get(config); +var openTelemetry = braintrust.openTelemetryCreate(); +``` + +`Braintrust.get()` is the main entry point. It reads `BRAINTRUST_API_KEY` from the environment automatically. + +## Install instrumentation + +The Java SDK instruments existing LLM clients by wrapping them. Find which clients the project already uses and wrap them as shown below. Only instrument frameworks that are actually present in the project. + +### OpenAI (`com.openai:openai-java`) + +Wrap the existing `OpenAIClient`: + +```java +import dev.braintrust.instrumentation.openai.BraintrustOpenAI; + +OpenAIClient openAIClient = BraintrustOpenAI.wrapOpenAI(openTelemetry, existingOpenAIClient); +``` + +### Anthropic (`com.anthropic:anthropic-java`) + +Wrap the existing `AnthropicClient`: + +```java +import dev.braintrust.instrumentation.anthropic.BraintrustAnthropic; + +AnthropicClient anthropicClient = BraintrustAnthropic.wrap(openTelemetry, existingAnthropicClient); +``` + +### Google GenAI / Gemini (`com.google.genai:google-genai`) + +Wrap the existing `Client.Builder`: + +```java +import dev.braintrust.instrumentation.genai.BraintrustGenAI; + +Client geminiClient = BraintrustGenAI.wrap(openTelemetry, existingClientBuilder); +``` + +### LangChain4j (`dev.langchain4j:langchain4j`) + +Wrap an existing `OpenAiChatModel.Builder`: + +```java +import dev.braintrust.instrumentation.langchain.BraintrustLangchain; + +ChatModel model = BraintrustLangchain.wrap(openTelemetry, existingOpenAiChatModelBuilder); +``` + +For LangChain4j AI Services, wrap the `AiServices` builder directly. This instruments LLM calls, tool calls, and concurrent tool execution: + +```java +import dev.braintrust.instrumentation.langchain.BraintrustLangchain; + +Assistant assistant = BraintrustLangchain.wrap( + openTelemetry, + AiServices.builder(Assistant.class) + .chatModel(existingChatModel) + .tools(new MyTools())); +``` + +### Spring AI + +For Spring Boot apps using Spring AI, register Braintrust beans and wrap the underlying LLM client. Example with Google GenAI: + +```java +@Bean +public Braintrust braintrust() { + return Braintrust.get(BraintrustConfig.fromEnvironment()); +} + +@Bean +public OpenTelemetry openTelemetry(Braintrust braintrust) { + return braintrust.openTelemetryCreate(); +} + +@Bean +public ChatModel chatModel(OpenTelemetry openTelemetry) { + Client genAIClient = BraintrustGenAI.wrap(openTelemetry, new Client.Builder()); + return GoogleGenAiChatModel.builder() + .genAiClient(genAIClient) + .defaultOptions(GoogleGenAiChatOptions.builder() + .model("gemini-2.0-flash-lite") + .build()) + .build(); +} +``` + +## Run the application + +Try to figure out how to run the application from the project structure: + +- **Gradle**: `./gradlew run`, `./gradlew bootRun` (Spring Boot), or a custom run task +- **Maven**: `mvn exec:java`, `mvn spring-boot:run` (Spring Boot) +- **SBT**: `sbt run` +- **Plain jar**: `java -jar ` + +If you can't determine how to run the app, ask the user. + +## Generate a permalink (required) + +Follow the permalink generation steps in the agent task (Step 5). Use the value passed to `defaultProjectName(...)` as the project name. diff --git a/skills/sdk-install/python.md b/skills/sdk-install/python.md new file mode 100644 index 0000000..dd2e00b --- /dev/null +++ b/skills/sdk-install/python.md @@ -0,0 +1,193 @@ +# Python SDK Install + +Reference guide for installing the Braintrust Python SDK. + +- SDK repo: https://github.com/braintrustdata/braintrust-sdk-python +- PyPI: https://pypi.org/project/braintrust/ +- Requires Python 3.9+ + +## Find the latest version of the SDK + +Look up the latest version from PyPI **without installing anything**. Do not guess -- use a read-only query so the environment stays unchanged until you pin the exact version. + +```bash +pip index versions braintrust +``` + +Then install that exact version with the project's package manager: + +### pip + +```bash +pip install braintrust== +``` + +### poetry + +```bash +poetry add braintrust== +``` + +### uv + +```bash +uv add braintrust== +``` + +## Initialize the SDK + +```python +import braintrust + +braintrust.init_logger(project="my-project") +``` + +`init_logger` is the main entry point for tracing. It reads `BRAINTRUST_API_KEY` from the environment automatically. + +## Install instrumentation + +The Python SDK supports two approaches: **auto-instrumentation** (recommended) and **manual wrapping**. + +### Auto-instrumentation (recommended) + +`auto_instrument()` automatically patches all supported libraries that are installed. Call it once at startup, before creating any clients. This is the simplest approach. + +```python +import braintrust + +braintrust.init_logger(project="my-project") +braintrust.auto_instrument() +``` + +After calling `auto_instrument()`, any supported client created afterwards is automatically traced -- no wrapping needed: + +```python +import openai + +client = openai.OpenAI() + +import anthropic + +client = anthropic.Anthropic() +``` + +Supported libraries: OpenAI, Anthropic, LiteLLM, Pydantic AI, Google GenAI, Agno, Claude Agent SDK (Anthropic), DSPy. + +You can selectively disable specific integrations: + +```python +braintrust.auto_instrument(litellm=False, dspy=False) +``` + +### Manual wrapping + +If you prefer explicit control, wrap individual clients instead. + +#### OpenAI (`openai`) + +```python +from braintrust import wrap_openai +from openai import OpenAI + +client = wrap_openai(OpenAI()) +``` + +#### Anthropic (`anthropic`) + +```python +import anthropic +from braintrust import wrap_anthropic + +client = wrap_anthropic(anthropic.Anthropic()) +``` + +#### LiteLLM (`litellm`) + +```python +import litellm +from braintrust import wrap_litellm + +litellm = wrap_litellm(litellm) +``` + +### OpenAI Agents SDK (`openai-agents`) + +Install with the extra and register the trace processor: + +```bash +pip install "braintrust[openai-agents]" +``` + +```python +from agents import Agent, Runner, set_trace_processors +from braintrust import init_logger +from braintrust.wrappers.openai import BraintrustTracingProcessor + +set_trace_processors([BraintrustTracingProcessor(init_logger("my-project"))]) + +agent = Agent(name="Assistant", instructions="You are a helpful assistant.") +result = await Runner.run(agent, "Hello!") +``` + +### LangChain (`langchain`) + +Install the callback handler package: + +```bash +pip install braintrust-langchain +``` + +```python +from braintrust import init_logger +from braintrust_langchain import BraintrustCallbackHandler, set_global_handler +from langchain_openai import ChatOpenAI + +init_logger(project="my-project") + +handler = BraintrustCallbackHandler() +set_global_handler(handler) + +model = ChatOpenAI() +response = await model.ainvoke("What is the capital of France?") +``` + +### LlamaIndex (`llama-index`) + +LlamaIndex traces via OpenTelemetry. Install with the otel extra: + +```bash +pip install "braintrust[otel]" llama-index +``` + +Set environment variables: + +```bash +export BRAINTRUST_API_KEY=your-api-key +export BRAINTRUST_PARENT=project_name:my-project +``` + +```python +import os + +import llama_index.core + +braintrust_api_url = os.environ.get("BRAINTRUST_API_URL", "https://api.braintrust.dev") +llama_index.core.set_global_handler("arize_phoenix", endpoint=f"{braintrust_api_url}/otel/v1/traces") +``` + +## Run the application + +Try to figure out how to run the application from the project structure: + +- **Script**: `python main.py`, `python -m mypackage` +- **Poetry**: `poetry run python main.py` +- **uv**: `uv run python main.py` +- **Django**: `python manage.py runserver` +- **FastAPI**: `uvicorn app:app --reload` +- **Flask**: `flask run` + +If you can't determine how to run the app, ask the user. + +## Generate a permalink (required) + +Follow the permalink generation steps in the agent task (Step 5). Use the `project=` argument passed to `init_logger` as the project name. diff --git a/skills/sdk-install/ruby.md b/skills/sdk-install/ruby.md new file mode 100644 index 0000000..39dde42 --- /dev/null +++ b/skills/sdk-install/ruby.md @@ -0,0 +1,128 @@ +# Ruby SDK Install + +Reference guide for installing the Braintrust Ruby SDK. + +- SDK repo: https://github.com/braintrustdata/braintrust-sdk-ruby +- RubyGems: https://rubygems.org/gems/braintrust +- Requires Ruby 3.1+ + +## Find the latest version of the SDK + +Look up the latest version from RubyGems **without installing anything**. Do not guess -- use a read-only query so the environment stays unchanged. + +```bash +gem search braintrust --remote --versions +``` + +## Install the SDK + +The SDK has three setup approaches. Choose the one that fits the project best. + +### Option A: Setup script (recommended for most apps) + +Add to the Gemfile with the `require` option. This auto-instruments all supported libraries at load time -- no additional code needed. + +```ruby +gem "braintrust", require: "braintrust/setup" +``` + +Then run: + +```bash +bundle install +``` + +Configure the project name in code (preferred over env vars): + +```ruby +require "braintrust" + +Braintrust.init(default_project: "my-project") +``` + +**Important**: The application must call `Bundler.require` for this to work (Rails does this by default). If not, add `require "braintrust/setup"` to an initializer file. + +### Option B: CLI command (no source code changes) + +Install the gem system-wide: + +```bash +gem install braintrust +``` + +Then wrap the application's start command: + +```bash +braintrust exec -- ruby app.rb +braintrust exec -- bundle exec rails server +``` + +To limit which providers are instrumented: + +```bash +braintrust exec --only openai -- ruby app.rb +``` + +### Option C: Braintrust.init (explicit control) + +Add to the Gemfile: + +```ruby +gem "braintrust" +``` + +Then call `Braintrust.init` in your code: + +```ruby +require "braintrust" + +Braintrust.init(default_project: "my-project") +``` + +Options for `Braintrust.init`: + +| Option | Default | Description | +| ----------------- | ----------------------------------- | --------------------------------------------------------------------------- | +| `default_project` | `ENV['BRAINTRUST_DEFAULT_PROJECT']` | Default project for spans | +| `auto_instrument` | `true` | `true`, `false`, or Hash with `:only`/`:except` keys to filter integrations | +| `api_key` | `ENV['BRAINTRUST_API_KEY']` | API key | + +## Install instrumentation + +By default, all three setup approaches auto-instrument every supported library that is installed. No wrapping code is needed. + +### Supported providers (auto-instrumented) + +| Provider | Gem | Integration name | +| --------- | ------------- | ---------------- | +| OpenAI | `openai` | `:openai` | +| | `ruby-openai` | `:ruby_openai` | +| Anthropic | `anthropic` | `:anthropic` | +| Multiple | `ruby_llm` | `:ruby_llm` | + +### Selectively enabling integrations + +```ruby +Braintrust.init(auto_instrument: { only: [:openai] }) +``` + +Or via environment variables: + +```bash +export BRAINTRUST_INSTRUMENT_ONLY=openai,anthropic +``` + +## Run the application + +Try to figure out how to run the application from the project structure: + +- **Rails**: `bundle exec rails server` or `bin/rails server` +- **Rack/Sinatra**: `bundle exec rackup` or `ruby app.rb` +- **Script**: `bundle exec ruby main.rb` or `ruby main.rb` +- **CLI wrap**: `braintrust exec -- ` + +If you can't determine how to run the app, ask the user. + +## Generate a permalink (required) + +Follow the permalink generation steps in the agent task (Step 5). Use the `default_project` argument passed to `Braintrust.init` as the project name. diff --git a/skills/sdk-install/typescript.md b/skills/sdk-install/typescript.md new file mode 100644 index 0000000..26e9c1b --- /dev/null +++ b/skills/sdk-install/typescript.md @@ -0,0 +1,187 @@ +# TypeScript SDK Install + +Reference guide for installing the Braintrust TypeScript SDK. + +- SDK repo: https://github.com/braintrustdata/braintrust-sdk-javascript +- npm: https://www.npmjs.com/package/braintrust +- Requires Node.js 18+ + +## Find the latest version of the SDK + +Look up the latest version from npm. Do not guess -- use the package manager to find the actual latest version. + +### npm + +```bash +npm view braintrust version +``` + +### yarn + +```bash +yarn info braintrust version +``` + +### pnpm + +```bash +pnpm view braintrust version +``` + +## Install the SDK + +Install with exact versions. + +If the repo uses `pnpm` (e.g. `pnpm-lock.yaml` exists), use `pnpm` rather than `npm install`. + +### npm + +```bash +npm install --save-exact braintrust@ --no-audit --no-fund +``` + +### yarn + +```bash +yarn add --exact braintrust@ +``` + +### pnpm + +```bash +pnpm add --save-exact braintrust@ +``` + +## Initialize the SDK + +```typescript +import { initLogger } from "braintrust"; + +const logger = initLogger({ + projectName: "my-project", + apiKey: process.env.BRAINTRUST_API_KEY, +}); +``` + +`initLogger` is the main entry point for tracing. It reads `BRAINTRUST_API_KEY` from the environment automatically if `apiKey` is not provided. If `initLogger` is not called, all wrapping functions are no-ops. + +## Install instrumentation + +The TypeScript SDK instruments existing LLM clients by wrapping them. Find which clients/frameworks the project already uses and wrap them as shown below. Only instrument frameworks that are actually present in the project. + +### OpenAI (`openai`) + +Wrap the existing `OpenAI` client: + +```typescript +import OpenAI from "openai"; +import { wrapOpenAI } from "braintrust"; + +const client = wrapOpenAI(new OpenAI({ apiKey: process.env.OPENAI_API_KEY })); +``` + +### Anthropic (`@anthropic-ai/sdk`) + +Wrap the existing `Anthropic` client: + +```typescript +import Anthropic from "@anthropic-ai/sdk"; +import { wrapAnthropic } from "braintrust"; + +const client = wrapAnthropic( + new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY }), +); +``` + +### Vercel AI SDK (`ai`) -- module-level wrapper + +Wrap the `ai` module to automatically trace `generateText`, `streamText`, `generateObject`, and `streamObject`: + +```typescript +import { wrapAISDK } from "braintrust"; +import * as ai from "ai"; +import { openai } from "@ai-sdk/openai"; + +const { generateText, streamText } = wrapAISDK(ai); + +const { text } = await generateText({ + model: openai("gpt-5-mini"), + prompt: "What is the capital of France?", +}); +``` + +### Vercel AI SDK (`ai`) -- model-level wrapper + +Alternatively, wrap individual model instances: + +```typescript +import { wrapAISDKModel } from "braintrust"; +import { openai } from "@ai-sdk/openai"; + +const model = wrapAISDKModel(openai("gpt-5-mini")); +``` + +### OpenAI Agents SDK (`@openai/agents`) + +Install the trace processor package and register it: + +```bash +npm install @braintrust/openai-agents @openai/agents +``` + +```typescript +import { initLogger } from "braintrust"; +import { OpenAIAgentsTraceProcessor } from "@braintrust/openai-agents"; +import { Agent, run, addTraceProcessor } from "@openai/agents"; + +const logger = initLogger({ projectName: "my-project" }); +const processor = new OpenAIAgentsTraceProcessor({ logger }); +addTraceProcessor(processor); + +const agent = new Agent({ + name: "Assistant", + model: "gpt-5-mini", + instructions: "You are a helpful assistant.", +}); + +const result = await run(agent, "Hello!"); +``` + +### LangChain.js (`@langchain/core`) + +Install the callback handler package and pass it to LangChain calls: + +```bash +npm install @braintrust/langchain-js +``` + +```typescript +import { initLogger } from "braintrust"; +import { BraintrustCallbackHandler } from "@braintrust/langchain-js"; +import { ChatOpenAI } from "@langchain/openai"; + +initLogger({ projectName: "my-project" }); + +const handler = new BraintrustCallbackHandler(); +const model = new ChatOpenAI(); + +await model.invoke("What is the capital of France?", { + callbacks: [handler], +}); +``` + +## Run the application + +Try to figure out how to run the application from the project structure: + +- **npm scripts**: check `package.json` for `start`, `dev`, or similar scripts +- **Next.js**: `npm run dev` or `npx next dev` +- **ts-node**: `npx ts-node src/index.ts` +- **tsx**: `npx tsx src/index.ts` +- **Node with TypeScript**: `npx tsc && node dist/index.js` + +If you can't determine how to run the app, ask the user. + +## Generate a permalink (required) + +Follow the permalink generation steps in the agent task (Step 5). Use the `projectName` argument passed to `initLogger` as the project name. diff --git a/src/setup/mod.rs b/src/setup/mod.rs index c76c717..b42a543 100644 --- a/src/setup/mod.rs +++ b/src/setup/mod.rs @@ -20,9 +20,11 @@ use crate::ui::{self, with_spinner}; mod agent_stream; mod docs; +mod sdk_install_docs; pub use docs::DocsArgs; +const INSTRUMENT_TASK_TEMPLATE: &str = include_str!("../../skills/sdk-install/instrument-task.md"); const SHARED_SKILL_BODY: &str = include_str!("../../skills/shared/braintrust-cli-body.md"); const SHARED_WORKFLOW_GUIDE: &str = include_str!("../../skills/shared/workflows.md"); const SHARED_SKILL_TEMPLATE: &str = include_str!("../../skills/shared/skill_template.md"); @@ -100,6 +102,11 @@ struct AgentsSetupArgs { /// Number of concurrent workers for docs prefetch/download. #[arg(long, default_value_t = crate::sync::default_workers())] workers: usize, + + /// Grant the agent full permissions and run it in the background without prompting. + /// Equivalent to choosing "Background" with all tool restrictions lifted. + #[arg(long)] + yolo: bool, } #[derive(Debug, Clone, Args)] @@ -150,6 +157,23 @@ struct InstrumentSetupArgs { /// Suppress streaming agent output; show a spinner and print results at the end #[arg(long, short = 'q')] quiet: bool, + + /// Language(s) to instrument (repeatable; case-insensitive). + /// When provided, the agent skips language auto-detection and instruments + /// the specified language(s) directly. + /// Accepted values: python, typescript, javascript, go, csharp, c#, java, ruby + #[arg(long = "language", value_enum, ignore_case = true)] + languages: Vec, + + /// Run the agent in interactive mode: inherits the terminal so the user can + /// approve/deny tool uses directly. Conflicts with --quiet and --yolo. + #[arg(long, short = 'i', conflicts_with_all = ["quiet", "yolo"])] + interactive: bool, + + /// Grant the agent full permissions and run it in the background without prompting. + /// Skips the run-mode selection question. Conflicts with --interactive. + #[arg(long, conflicts_with = "interactive")] + yolo: bool, } #[derive(Debug, Clone, Args)] @@ -344,7 +368,7 @@ pub async fn run_setup_top(base: BaseArgs, args: SetupArgs) -> Result<()> { Some(SetupSubcommand::Doctor(doctor)) => run_doctor(base, doctor), None => { if should_prompt_setup_action(&base, &args.agents) { - run_setup_wizard(base).await + run_setup_wizard(base, args.agents.yolo).await } else { run_setup(base, args.agents).await } @@ -354,7 +378,7 @@ pub async fn run_setup_top(base: BaseArgs, args: SetupArgs) -> Result<()> { pub use docs::run_docs_top; -async fn run_setup_wizard(mut base: BaseArgs) -> Result<()> { +async fn run_setup_wizard(mut base: BaseArgs, yolo: bool) -> Result<()> { let mut had_failures = false; // ── Step 1: Auth ── @@ -399,12 +423,7 @@ async fn run_setup_wizard(mut base: BaseArgs) -> Result<()> { let home = home_dir().ok_or_else(|| anyhow!("failed to resolve HOME/USERPROFILE"))?; let local_root = resolve_local_root_for_scope(scope)?; let detected = detect_agents(local_root.as_deref(), &home); - let agent_defaults = resolve_selected_agents(&[], &detected); - let agents = - prompt_agents_selection(&agent_defaults)?.ok_or_else(|| anyhow!("setup cancelled"))?; - if agents.is_empty() { - bail!("no agents selected"); - } + let agents = resolve_selected_agents(&[], &detected); Some((scope, agents, home)) } else { None @@ -413,7 +432,8 @@ async fn run_setup_wizard(mut base: BaseArgs) -> Result<()> { if wants_skills { eprintln!(" {}", style("Skills:").bold()); if let Some((scope, ref agents, _)) = setup_context { - let agent_args: Vec = agents.iter().map(|a| agent_to_agent_arg(*a)).collect(); + let agent_args: Vec = + agents.iter().map(|a| map_agent_to_agent_arg(*a)).collect(); let args = AgentsSetupArgs { agents: agent_args, local: matches!(scope, InstallScope::Local), @@ -423,6 +443,7 @@ async fn run_setup_wizard(mut base: BaseArgs) -> Result<()> { no_fetch_docs: true, refresh_docs: false, workers: crate::sync::default_workers(), + yolo: false, }; let outcome = execute_skills_setup(&base, &args, true).await?; for r in &outcome.results { @@ -445,7 +466,7 @@ async fn run_setup_wizard(mut base: BaseArgs) -> Result<()> { had_failures = true; } } - if outcome.installed_count == 0 { + if outcome.installed_count == 0 && !agents.is_empty() { had_failures = true; } } @@ -473,6 +494,9 @@ async fn run_setup_wizard(mut base: BaseArgs) -> Result<()> { refresh_docs: false, workers: crate::sync::default_workers(), quiet: false, + languages: Vec::new(), + interactive: false, + yolo, }, ) .await?; @@ -580,15 +604,6 @@ fn maybe_init(org: &str, project: &crate::projects::api::Project) -> Result AgentArg { - match agent { - Agent::Claude => AgentArg::Claude, - Agent::Codex => AgentArg::Codex, - Agent::Cursor => AgentArg::Cursor, - Agent::Opencode => AgentArg::Opencode, - } -} - async fn run_setup(base: BaseArgs, args: AgentsSetupArgs) -> Result<()> { let outcome = execute_skills_setup(&base, &args, false).await?; if base.json { @@ -718,6 +733,35 @@ enum InstrumentAgentArg { Opencode, } +/// Languages supported by `--language`. Variants map to canonical display +/// names used in the agent task prompt. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, ValueEnum)] +enum LanguageArg { + Python, + /// TypeScript / JavaScript + #[value(name = "typescript", alias = "javascript")] + TypeScript, + Go, + /// C# / csharp + #[value(name = "csharp", alias = "c#")] + CSharp, + Java, + Ruby, +} + +impl LanguageArg { + fn display_name(self) -> &'static str { + match self { + LanguageArg::Python => "Python", + LanguageArg::TypeScript => "TypeScript", + LanguageArg::Go => "Go", + LanguageArg::CSharp => "C#", + LanguageArg::Java => "Java", + LanguageArg::Ruby => "Ruby", + } + } +} + fn should_prompt_setup_action(base: &BaseArgs, args: &AgentsSetupArgs) -> bool { if base.json || !ui::is_interactive() { return false; @@ -753,6 +797,18 @@ async fn run_instrument_setup(base: BaseArgs, args: InstrumentSetupArgs) -> Resu } let selected_workflows = resolve_instrument_workflow_selection(&args)?; + + let selected_languages: Vec = if !args.languages.is_empty() { + args.languages.clone() + } else if ui::is_interactive() && !args.yes { + let Some(langs) = prompt_instrument_language_selection()? else { + bail!("instrument setup cancelled by user"); + }; + langs + } else { + Vec::new() + }; + let show_progress = !base.json; let mut warnings = Vec::new(); let mut notes = Vec::new(); @@ -789,6 +845,7 @@ async fn run_instrument_setup(base: BaseArgs, args: InstrumentSetupArgs) -> Resu no_fetch_docs: false, refresh_docs: args.refresh_docs, workers: args.workers, + yolo: false, }; let outcome = execute_skills_setup(&base, &setup_args, false).await?; detected = outcome.detected_agents; @@ -800,22 +857,79 @@ async fn run_instrument_setup(base: BaseArgs, args: InstrumentSetupArgs) -> Resu } } + // Determine run mode: interactive TUI vs background (autonomous). + // --yolo: background, full bypassPermissions (no restrictions) + // --interactive: interactive TUI + // --yes or non-interactive terminal: background, restricted to language package managers + // Otherwise: ask the user. + let (run_interactive, bypass_permissions) = if args.interactive { + (true, false) + } else if args.yolo { + (false, true) + } else if args.yes || !ui::is_interactive() { + (false, false) + } else { + let pkg_mgrs = package_manager_cmds_for_languages(&selected_languages).join(", "); + let background_label = format!( + "Background (automatic) — runs autonomously; \ + allowed package managers: {pkg_mgrs}" + ); + let choices = [ + background_label.as_str(), + "Interactive TUI — agent opens in its terminal UI; \ + you review and approve each tool use", + ]; + let selection = Select::with_theme(&ColorfulTheme::default()) + .with_prompt("How do you want to run the agent?") + .items(&choices) + .default(0) + .interact_opt()?; + let Some(index) = selection else { + bail!("instrument setup cancelled by user"); + }; + let interactive = index == 1; + (interactive, false) + }; + + let docs_output_dir = root.join(".bt").join("skills").join("docs"); + sdk_install_docs::write_sdk_install_docs(&docs_output_dir)?; + let task_path = root .join(".bt") .join("skills") .join("AGENT_TASK.instrument.md"); write_text_file( &task_path, - &render_instrument_task(&root, &selected_workflows), + &render_instrument_task( + &docs_output_dir, + &selected_workflows, + &selected_languages, + run_interactive, + ), )?; - let invocation = - resolve_instrument_invocation(selected, args.agent_cmd.as_deref(), &task_path)?; notes.push(format!( "Instrumentation task prompt written to {}.", task_path.display() )); + let invocation = resolve_instrument_invocation( + selected, + args.agent_cmd.as_deref(), + &task_path, + run_interactive, + bypass_permissions, + &selected_languages, + )?; + + if run_interactive { + eprintln!(); + eprintln!("Claude Code is opening in interactive mode."); + eprintln!("The instrumentation task is pre-loaded. Press Enter to begin."); + eprintln!("Task file: {}", task_path.display()); + eprintln!(); + } + let show_output = !base.json && !args.quiet; let status = if args.quiet && !base.json { with_spinner( @@ -915,6 +1029,42 @@ fn prompt_instrument_workflow_selection() -> Result>> { })) } +fn prompt_instrument_language_selection() -> Result>> { + // Index 0 = "All / auto-detect". Indices 1-6 map to specific languages. + let choices = [ + "All languages (auto-detect)", + "Python", + "TypeScript / JavaScript", + "Go", + "Java", + "Ruby", + "C#", + ]; + let defaults = [true, false, false, false, false, false, false]; + let selected = MultiSelect::with_theme(&ColorfulTheme::default()) + .with_prompt("Which language(s) to instrument?") + .items(&choices) + .defaults(&defaults) + .interact_opt()?; + Ok(selected.map(|indices| { + if indices.is_empty() || indices.contains(&0) { + return Vec::new(); // auto-detect + } + indices + .iter() + .filter_map(|&i| match i { + 1 => Some(LanguageArg::Python), + 2 => Some(LanguageArg::TypeScript), + 3 => Some(LanguageArg::Go), + 4 => Some(LanguageArg::Java), + 5 => Some(LanguageArg::Ruby), + 6 => Some(LanguageArg::CSharp), + _ => None, + }) + .collect() + })) +} + fn map_agent_to_agent_arg(agent: Agent) -> AgentArg { match agent { Agent::Claude => AgentArg::Claude, @@ -1019,13 +1169,55 @@ fn prompt_instrument_agent(default_agent: Agent) -> Result { Ok(ALL_AGENTS[index]) } +/// Returns the package-manager command names allowed in background (non-yolo) mode. +/// +/// When `languages` is empty (auto-detect), every supported package manager is included. +/// Otherwise only the managers for the specified languages are returned. +fn package_manager_cmds_for_languages(languages: &[LanguageArg]) -> Vec<&'static str> { + use std::collections::BTreeSet; + let unique: BTreeSet = languages.iter().copied().collect(); + + if unique.is_empty() { + return vec![ + "uv", "pip", "pip3", "poetry", "pipenv", "npm", "npx", "yarn", "pnpm", "bun", "deno", + "go", "gradle", "gradlew", "mvn", "mvnw", "gem", "bundle", "dotnet", + ]; + } + + let mut v: Vec<&'static str> = Vec::new(); + for lang in &unique { + match lang { + LanguageArg::Python => v.extend(["uv", "pip", "pip3", "poetry", "pipenv"]), + LanguageArg::TypeScript => v.extend(["npm", "npx", "yarn", "pnpm", "bun", "deno"]), + LanguageArg::Go => v.extend(["go"]), + LanguageArg::Java => v.extend(["gradle", "gradlew", "mvn", "mvnw"]), + LanguageArg::Ruby => v.extend(["gem", "bundle"]), + LanguageArg::CSharp => v.extend(["dotnet"]), + } + } + v +} + +/// Returns the `--allowedTools` string for Claude from a language list. +fn allowed_bash_tools_for_languages(languages: &[LanguageArg]) -> String { + package_manager_cmds_for_languages(languages) + .iter() + .map(|c| format!("Bash({c}:*)")) + .collect::>() + .join(" ") +} + enum InstrumentInvocation { Program { program: String, args: Vec, stdin_file: Option, + /// Path to a file whose content is passed as the initial user prompt (positional arg). prompt_file_arg: Option, + /// Hardcoded initial user prompt string (alternative to prompt_file_arg). + initial_prompt: Option, stream_json: bool, + interactive: bool, }, Shell(String), } @@ -1034,6 +1226,9 @@ fn resolve_instrument_invocation( agent: Agent, agent_cmd: Option<&str>, task_path: &Path, + interactive: bool, + bypass_permissions: bool, + languages: &[LanguageArg], ) -> Result { if let Some(command) = agent_cmd { let trimmed = command.trim(); @@ -1049,31 +1244,77 @@ fn resolve_instrument_invocation( args: vec!["exec".to_string(), "-".to_string()], stdin_file: Some(task_path.to_path_buf()), prompt_file_arg: None, + initial_prompt: None, stream_json: false, + interactive, }, - Agent::Claude => InstrumentInvocation::Program { - program: "claude".to_string(), - args: vec![ - "-p".to_string(), - "--permission-mode".to_string(), - "acceptEdits".to_string(), - "--verbose".to_string(), - "--output-format".to_string(), - "stream-json".to_string(), - "--include-partial-messages".to_string(), - "--disallowedTools".to_string(), - "EnterPlanMode".to_string(), - ], - stdin_file: Some(task_path.to_path_buf()), - prompt_file_arg: None, - stream_json: true, - }, + Agent::Claude => { + if interactive { + // In interactive mode the full task goes into --append-system-prompt so + // Claude already knows what to do. A short initial user message is passed + // as the positional arg so Claude immediately starts working — the user only + // needs to press Enter once on a short, clear prompt rather than a wall of + // raw task markdown. + let task_content = std::fs::read_to_string(task_path) + .with_context(|| format!("failed to read task file {}", task_path.display()))?; + InstrumentInvocation::Program { + program: "claude".to_string(), + args: vec![ + "--append-system-prompt".to_string(), + task_content, + "--disallowedTools".to_string(), + "EnterPlanMode".to_string(), + "--name".to_string(), + "Braintrust: Instrument".to_string(), + ], + stdin_file: None, + prompt_file_arg: None, + initial_prompt: Some( + "Please begin the Braintrust instrumentation task.".to_string(), + ), + stream_json: false, + interactive: true, + } + } else { + let mut claude_args = vec![ + "-p".to_string(), + "--permission-mode".to_string(), + if bypass_permissions { + "bypassPermissions".to_string() + } else { + "acceptEdits".to_string() + }, + "--verbose".to_string(), + "--output-format".to_string(), + "stream-json".to_string(), + "--include-partial-messages".to_string(), + ]; + if !bypass_permissions { + let allowed = allowed_bash_tools_for_languages(languages); + claude_args.push("--allowedTools".to_string()); + claude_args.push(allowed); + } + claude_args.push("--disallowedTools".to_string()); + claude_args.push("EnterPlanMode".to_string()); + InstrumentInvocation::Program { + program: "claude".to_string(), + args: claude_args, + stdin_file: Some(task_path.to_path_buf()), + prompt_file_arg: None, + initial_prompt: None, + stream_json: true, + interactive: false, + } + } + } Agent::Opencode => InstrumentInvocation::Program { program: "opencode".to_string(), args: vec!["run".to_string()], stdin_file: None, prompt_file_arg: Some(task_path.to_path_buf()), + initial_prompt: None, stream_json: false, + interactive, }, Agent::Cursor => InstrumentInvocation::Program { program: "cursor-agent".to_string(), @@ -1086,7 +1327,9 @@ fn resolve_instrument_invocation( ], stdin_file: None, prompt_file_arg: Some(task_path.to_path_buf()), + initial_prompt: None, stream_json: true, + interactive, }, }; Ok(invocation) @@ -1115,7 +1358,9 @@ async fn run_agent_invocation( args, stdin_file, prompt_file_arg, + initial_prompt, stream_json, + interactive, } => { let mut command = Command::new(program); command.args(args).current_dir(root); @@ -1129,6 +1374,9 @@ async fn run_agent_invocation( } command.arg(prompt); } + if let Some(prompt) = initial_prompt { + command.arg(prompt); + } if let Some(path) = stdin_file { let file = fs::File::open(path).with_context(|| { format!("failed to open task prompt file {}", path.display()) @@ -1136,6 +1384,14 @@ async fn run_agent_invocation( command.stdin(Stdio::from(file)); } + if *interactive { + // Inherit all streams so the user can interact with the agent directly. + return command + .status() + .await + .with_context(|| format!("failed to run agent command in {}", root.display())); + } + if !show_output { command.stdout(Stdio::null()).stderr(Stdio::null()); return command @@ -1160,31 +1416,67 @@ async fn run_agent_invocation( } } -fn render_instrument_task(repo_root: &Path, workflows: &[WorkflowArg]) -> String { - let docs_output_dir = repo_root.join(".bt").join("skills").join("docs"); - let workflow_list = workflows +fn render_instrument_task( + docs_output_dir: &Path, + workflows: &[WorkflowArg], + languages: &[LanguageArg], + interactive: bool, +) -> String { + use std::collections::BTreeSet; + let sdk_install_dir = docs_output_dir.join("sdk-install"); + + // Deduplicate languages (TypeScript and JavaScript both map to the same variant). + let unique_langs: BTreeSet = languages.iter().copied().collect(); + let language_context = if unique_langs.is_empty() { + String::new() + } else { + let names: Vec = unique_langs + .iter() + .map(|l| format!("**{}**", l.display_name())) + .collect(); + let list = if names.len() == 1 { + names[0].clone() + } else { + let (last, rest) = names.split_last().unwrap(); + format!("{} and {}", rest.join(", "), last) + }; + format!( + "### Language Override\n\n\ + Instrument {}. \ + Skip Step 2 (language auto-detection) and proceed directly to Step 3 \ + for the specified language(s).\n", + list + ) + }; + + // When non-instrument workflows are selected the agent should use local + // bt CLI skills rather than the MCP server. + let workflow_context = if workflows .iter() - .map(|workflow| workflow.as_str()) - .collect::>() - .join(", "); - format!( - r#"Instrument this repository with Braintrust tracing. - -Requirements: -1. Review the Braintrust instrumentation docs in `{}`. -2. Focus on these workflow docs: {}. -3. Use the installed Braintrust agent skills in this repo and prefer local `bt` CLI commands to verify setup. -4. Do not rely on the Braintrust MCP server for this setup flow. -5. Add tracing/instrumentation to the application code. -6. Keep behavior intact; avoid unrelated refactors. -7. If tests exist, run the smallest relevant tests after instrumentation. - -Output: -- Updated source files with Braintrust instrumentation. -- A short summary of what was instrumented and why."#, - docs_output_dir.display(), - workflow_list - ) + .any(|w| !matches!(w, WorkflowArg::Instrument)) + { + "## Agent Skills\n\n\ + Use the installed Braintrust agent skills from `.agents/skills/braintrust/`. \ + When verifying data in Braintrust, prefer local `bt` CLI commands over direct \ + API calls. Do not rely on the Braintrust MCP server for data queries.\n" + .to_string() + } else { + String::new() + }; + + let run_mode_context = if interactive { + "- **Interactive mode:** You can ask the user questions through the chat interface.\n" + } else { + "- **Non-interactive mode:** You cannot ask the user questions. \ + If a step requires user input (e.g., ambiguous language in a polyglot repo, \ + unknown run command), abort with a clear explanation of what is needed.\n" + }; + + INSTRUMENT_TASK_TEMPLATE + .replace("{SDK_INSTALL_DIR}", &sdk_install_dir.display().to_string()) + .replace("{LANGUAGE_CONTEXT}", &language_context) + .replace("{WORKFLOW_CONTEXT}", &workflow_context) + .replace("{RUN_MODE_CONTEXT}", run_mode_context) } struct McpSetupOutcome { @@ -2420,7 +2712,7 @@ fn print_wizard_step(number: u8, label: &str) { fn print_wizard_agent_result(result: &AgentInstallResult) { let (indicator, status_text) = match result.status { InstallStatus::Installed => (style("✓").green(), "installed"), - InstallStatus::Skipped => (style("—").dim(), "skipped"), + InstallStatus::Skipped => (style("—").dim(), "already configured"), InstallStatus::Failed => (style("✗").red(), "failed"), }; eprintln!( @@ -2673,6 +2965,7 @@ mod tests { no_fetch_docs: true, refresh_docs: false, workers: crate::sync::default_workers(), + yolo: false, }; let home = std::env::temp_dir(); let selection = resolve_setup_selection(&args, &home).expect("resolve setup selection"); @@ -2699,6 +2992,9 @@ mod tests { refresh_docs: false, workers: crate::sync::default_workers(), quiet: false, + languages: Vec::new(), + interactive: false, + yolo: false, }; let selected = @@ -2719,6 +3015,9 @@ mod tests { refresh_docs: false, workers: crate::sync::default_workers(), quiet: false, + languages: Vec::new(), + interactive: false, + yolo: false, }; let selected = @@ -2729,7 +3028,12 @@ mod tests { #[test] fn render_instrument_task_includes_local_cli_and_no_mcp_guidance() { let root = PathBuf::from("/tmp/repo"); - let task = render_instrument_task(&root, &[WorkflowArg::Instrument, WorkflowArg::Observe]); + let task = render_instrument_task( + &root, + &[WorkflowArg::Instrument, WorkflowArg::Observe], + &[], + false, + ); assert!(task.contains("Use the installed Braintrust agent skills")); assert!(task.contains("prefer local `bt` CLI commands")); assert!(task.contains("Do not rely on the Braintrust MCP server")); @@ -2744,8 +3048,9 @@ mod tests { #[test] fn codex_instrument_invocation_uses_exec_with_stdin_prompt() { let task_path = PathBuf::from("/tmp/AGENT_TASK.instrument.md"); - let invocation = resolve_instrument_invocation(Agent::Codex, None, &task_path) - .expect("resolve instrument invocation"); + let invocation = + resolve_instrument_invocation(Agent::Codex, None, &task_path, false, false, &[]) + .expect("resolve instrument invocation"); match invocation { InstrumentInvocation::Program { @@ -2754,6 +3059,7 @@ mod tests { stdin_file, prompt_file_arg, stream_json, + .. } => { assert_eq!(program, "codex"); assert_eq!(args, vec!["exec".to_string(), "-".to_string()]); @@ -2766,10 +3072,11 @@ mod tests { } #[test] - fn claude_instrument_invocation_uses_print_with_stdin_prompt() { + fn claude_instrument_invocation_uses_print_with_bypass_permissions() { let task_path = PathBuf::from("/tmp/AGENT_TASK.instrument.md"); - let invocation = resolve_instrument_invocation(Agent::Claude, None, &task_path) - .expect("resolve instrument invocation"); + let invocation = + resolve_instrument_invocation(Agent::Claude, None, &task_path, false, true, &[]) + .expect("resolve instrument invocation"); match invocation { InstrumentInvocation::Program { @@ -2778,6 +3085,7 @@ mod tests { stdin_file, prompt_file_arg, stream_json, + .. } => { assert_eq!(program, "claude"); assert_eq!( @@ -2785,7 +3093,7 @@ mod tests { vec![ "-p".to_string(), "--permission-mode".to_string(), - "acceptEdits".to_string(), + "bypassPermissions".to_string(), "--verbose".to_string(), "--output-format".to_string(), "stream-json".to_string(), @@ -2802,11 +3110,120 @@ mod tests { } } + #[test] + fn claude_background_invocation_uses_accept_edits_with_language_scoped_tools() { + let task_path = PathBuf::from("/tmp/AGENT_TASK.instrument.md"); + // Python only → only Python package managers should be allowed. + let invocation = resolve_instrument_invocation( + Agent::Claude, + None, + &task_path, + false, + false, + &[LanguageArg::Python], + ) + .expect("resolve instrument invocation"); + + match invocation { + InstrumentInvocation::Program { program, args, .. } => { + assert_eq!(program, "claude"); + let pm_idx = args.iter().position(|a| a == "--permission-mode").unwrap(); + assert_eq!(args[pm_idx + 1], "acceptEdits"); + + let at_idx = args.iter().position(|a| a == "--allowedTools").unwrap(); + let allowed = &args[at_idx + 1]; + // Python managers present + assert!(allowed.contains("Bash(uv:*)"), "uv should be allowed"); + assert!(allowed.contains("Bash(pip:*)"), "pip should be allowed"); + // Non-Python managers absent + assert!( + !allowed.contains("Bash(npm:*)"), + "npm must not appear for Python-only" + ); + assert!( + !allowed.contains("Bash(go:*)"), + "go must not appear for Python-only" + ); + } + InstrumentInvocation::Shell(_) => panic!("expected program invocation"), + } + } + + #[test] + fn claude_background_invocation_with_no_language_allows_all_package_managers() { + let task_path = PathBuf::from("/tmp/AGENT_TASK.instrument.md"); + let invocation = + resolve_instrument_invocation(Agent::Claude, None, &task_path, false, false, &[]) + .expect("resolve instrument invocation"); + + match invocation { + InstrumentInvocation::Program { args, .. } => { + let at_idx = args.iter().position(|a| a == "--allowedTools").unwrap(); + let allowed = &args[at_idx + 1]; + assert!(allowed.contains("Bash(uv:*)")); + assert!(allowed.contains("Bash(npm:*)")); + assert!(allowed.contains("Bash(go:*)")); + assert!(allowed.contains("Bash(gradle:*)")); + assert!(allowed.contains("Bash(gem:*)")); + assert!(allowed.contains("Bash(dotnet:*)")); + } + InstrumentInvocation::Shell(_) => panic!("expected program invocation"), + } + } + + #[test] + fn claude_interactive_instrument_invocation_uses_system_prompt_no_print_flag() { + let dir = tempfile::tempdir().expect("tempdir"); + let task_path = dir.path().join("AGENT_TASK.instrument.md"); + std::fs::write(&task_path, "## Task\nInstrument this repo.").expect("write task"); + + let invocation = + resolve_instrument_invocation(Agent::Claude, None, &task_path, true, false, &[]) + .expect("resolve instrument invocation"); + + match invocation { + InstrumentInvocation::Program { + program, + args, + stdin_file, + prompt_file_arg, + initial_prompt, + stream_json, + interactive, + } => { + assert_eq!(program, "claude"); + assert!( + !args.contains(&"-p".to_string()), + "interactive mode must not pass -p" + ); + assert!( + args.contains(&"--append-system-prompt".to_string()), + "task should be in system prompt" + ); + assert!(args.contains(&"--disallowedTools".to_string())); + assert!(args.contains(&"--name".to_string())); + assert_eq!(stdin_file, None); + assert_eq!( + prompt_file_arg, None, + "task is in system prompt, not prompt_file_arg" + ); + assert!( + initial_prompt.is_some(), + "short initial message must be set to trigger Claude" + ); + assert!(!stream_json); + assert!(interactive); + } + InstrumentInvocation::Shell(_) => panic!("expected program invocation"), + } + } + #[test] fn opencode_instrument_invocation_uses_run_with_prompt_arg() { let task_path = PathBuf::from("/tmp/AGENT_TASK.instrument.md"); - let invocation = resolve_instrument_invocation(Agent::Opencode, None, &task_path) - .expect("resolve instrument invocation"); + let invocation = + resolve_instrument_invocation(Agent::Opencode, None, &task_path, false, false, &[]) + .expect("resolve instrument invocation"); match invocation { InstrumentInvocation::Program { @@ -2815,6 +3232,7 @@ mod tests { stdin_file, prompt_file_arg, stream_json, + .. } => { assert_eq!(program, "opencode"); assert_eq!(args, vec!["run".to_string()]); @@ -2829,8 +3247,9 @@ mod tests { #[test] fn cursor_instrument_invocation_uses_print_with_prompt_arg() { let task_path = PathBuf::from("/tmp/AGENT_TASK.instrument.md"); - let invocation = resolve_instrument_invocation(Agent::Cursor, None, &task_path) - .expect("resolve instrument invocation"); + let invocation = + resolve_instrument_invocation(Agent::Cursor, None, &task_path, false, false, &[]) + .expect("resolve instrument invocation"); match invocation { InstrumentInvocation::Program { @@ -2839,6 +3258,7 @@ mod tests { stdin_file, prompt_file_arg, stream_json, + .. } => { assert_eq!(program, "cursor-agent"); assert_eq!( diff --git a/src/setup/sdk_install_docs.rs b/src/setup/sdk_install_docs.rs new file mode 100644 index 0000000..a6db342 --- /dev/null +++ b/src/setup/sdk_install_docs.rs @@ -0,0 +1,40 @@ +use std::path::Path; + +use anyhow::Result; + +use super::write_text_file; + +const PYTHON_DOCS: &str = include_str!("../../skills/sdk-install/python.md"); +const TYPESCRIPT_DOCS: &str = include_str!("../../skills/sdk-install/typescript.md"); +const GO_DOCS: &str = include_str!("../../skills/sdk-install/go.md"); +const RUBY_DOCS: &str = include_str!("../../skills/sdk-install/ruby.md"); +const JAVA_DOCS: &str = include_str!("../../skills/sdk-install/java.md"); +const CSHARP_DOCS: &str = include_str!("../../skills/sdk-install/csharp.md"); +const URL_FORMATS_DOCS: &str = include_str!("../../skills/sdk-install/braintrust-url-formats.md"); + +const INDEX: &str = "# SDK Install Docs + +Per-language SDK installation guides. Read the file for the detected language. + +- [Python](python.md) +- [TypeScript](typescript.md) +- [Go](go.md) +- [Ruby](ruby.md) +- [Java](java.md) +- [C#](csharp.md) +- [Braintrust URL Formats](braintrust-url-formats.md) +"; + +/// Write all SDK install docs to `/sdk-install/`. +pub fn write_sdk_install_docs(output_dir: &Path) -> Result<()> { + let dir = output_dir.join("sdk-install"); + write_text_file(&dir.join("python.md"), PYTHON_DOCS)?; + write_text_file(&dir.join("typescript.md"), TYPESCRIPT_DOCS)?; + write_text_file(&dir.join("go.md"), GO_DOCS)?; + write_text_file(&dir.join("ruby.md"), RUBY_DOCS)?; + write_text_file(&dir.join("java.md"), JAVA_DOCS)?; + write_text_file(&dir.join("csharp.md"), CSHARP_DOCS)?; + write_text_file(&dir.join("braintrust-url-formats.md"), URL_FORMATS_DOCS)?; + write_text_file(&dir.join("_index.md"), INDEX)?; + Ok(()) +}