Skip to content

test #109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Andrew Kent (realark) wants to merge 1 commit into from
Closed

test #109

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
130 changes: 1 addition & 129 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,129 +1 @@
# Braintrust Java Tracing & Eval SDK

[![javadoc](https://javadoc.io/badge2/dev.braintrust/braintrust-sdk-java/javadoc.svg)](https://javadoc.io/doc/dev.braintrust/braintrust-sdk-java)
[![CI](https://github.com/braintrustdata/braintrust-sdk-java/actions/workflows/ci.yml/badge.svg)](https://github.com/braintrustdata/braintrust-sdk-java/actions/workflows/ci.yml)

## Overview

This library provides tools for **evaluating** and **tracing** AI applications in [Braintrust](https://www.braintrust.dev). Use it to:

- **Evaluate** your AI models with custom test cases and scoring functions
- **Trace** LLM calls and monitor AI application performance with OpenTelemetry
- **Integrate** seamlessly with OpenAI, Anthropic, and other LLM providers

This SDK is currently in BETA status and APIs may change.

## Quickstart

The fastest way to report data to Braintrust is to add the [braintrust java agent](https://central.sonatype.com/artifact/dev.braintrust/braintrust-java-agent) to your jvm startup args.

springboot+gradle example:

```build.gradle
configurations {
btAgent
}
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-web'
btAgent "dev.braintrust:braintrust-java-agent:<version-goes-here>"
}
bootRun {
jvmArgs = [
// NOTE: if you're running with other java agents, add the braintrust agent last
"-javaagent:${configurations.btAgent.singleFile.absolutePath}",
]
}
```

This will automatically instrument major AI clients and frameworks. No code changes required. A list of supported instrumentation can be found [here](./braintrust-sdk/instrumentation)

NOTE: Additional steps may be required for users running with other `-javaagent` flags. Consult [braintrust docs](https://www.braintrust.dev/docs/instrument/trace-llm-calls#java) for details.

## Eval Quickstart

Add the [Braintrust SDK](https://central.sonatype.com/artifact/dev.braintrust/braintrust-sdk-java) to your package manager.

gradle example:
```gradle
dependencies {
implementation 'dev.braintrust:braintrust-sdk-java:<version-goes-here>'
}
```

Use the SDK to create and send your eval:
```java
var braintrust = Braintrust.get();
braintrust.openTelemetryCreate();
var openAIClient = OpenAIOkHttpClient.fromEnv();

Function<String, String> getFoodType =
(String food) -> {
var request =
ChatCompletionCreateParams.builder()
.model(ChatModel.GPT_4O_MINI)
.addSystemMessage("Return a one word answer")
.addUserMessage("What kind of food is " + food + "?")
.build();
var response = openAIClient.chat().completions().create(request);
return response.choices().get(0).message().content().orElse("").toLowerCase();
};

var eval = braintrust.<String, String>evalBuilder()
.name("java-eval-x-" + System.currentTimeMillis())
.cases(
DatasetCase.of("asparagus", "vegetable"),
DatasetCase.of("banana", "fruit"))
.taskFunction(getFoodType)
.scorers(
Scorer.of(
"exact_match",
(expected, result) -> expected.equals(result) ? 1.0 : 0.0))
.build();
var result = eval.run();
System.out.println("\n\n" + result.createReportString());
```

### Manual Instrumentation

Alternatively, sdk users can manually apply instrumentation instead of using the java agent.

```java
var braintrust = Braintrust.get();
var openTelemetry = braintrust.openTelemetryCreate();
OpenAIClient openAIClient = BraintrustOpenAI.wrapOpenAI(openTelemetry, OpenAIOkHttpClient.fromEnv());

var request =
ChatCompletionCreateParams.builder()
.model(ChatModel.GPT_4O_MINI)
.addUserMessage("What is the capital of France?")
.build();
// openai calls will be traced and reported to braintrust
var response = openAIClient.chat().completions().create(request);
```

A list of supported instrumentation can be found [here](./braintrust-sdk/instrumentation)

## Running Examples

Example source code can be found [here](./examples/src/main/java/dev/braintrust/examples)

```bash
export BRAINTRUST_API_KEY="your-braintrust-api-key"
export OPENAI_API_KEY="your-oai-api-key" # to run oai examples
export ANTHROPIC_API_KEY="your-anthropic-api-key" # to run anthropic examples
# install java 17 or later
brew install openjdk@17 # macOS
sudo apt install openjdk-17-jdk # ubuntu
# to run a specific example
./gradlew :examples:runSimpleOpenTelemetry
# to see all examples
./gradlew :examples:tasks --group="Braintrust SDK Examples"
```

## Logging

The SDK uses a standard slf4j logger and will use the default log level (or not log at all if slf4j is not installed).

All Braintrust loggers will log into the `dev.braintrust` namespace. To adjust the log level, consult your logger documentation.

For example, to enable debug logging for slf4j-simple you would set the system property `org.slf4j.simpleLogger.log.dev.braintrust=DEBUG`
I'm just testing CI. Don't actually merge this
Loading