---
name: braintrust
description: Use when building, evaluating, and improving AI applications. Reach
  for this skill when you need to instrument applications with tracing, run
  systematic evaluations, analyze production logs, build datasets, or deploy AI
  systems with observability and version control.
metadata:
  mintlify-proj: braintrust
  version: "1.0"
---

# Braintrust Skill

## Product summary

Braintrust is an active observability platform for AI applications that captures traces from LLM calls and application logic, analyzes production behavior, runs systematic evaluations, and deploys changes with version control. Agents use Braintrust to instrument applications (via SDKs in TypeScript, Python, Go, Java, Ruby, C#), run experiments to compare prompts and models, build evaluation datasets from production logs, and monitor deployed systems.

**Key files and commands:**
- `.bt/config.json` — Project configuration (created by `bt init`)
- `BRAINTRUST_API_KEY` — Environment variable for authentication
- `bt` CLI — Command-line interface for tracing, evals, querying, and syncing
- SDKs: `braintrust` (TypeScript/Python), `github.com/braintrust-dev/braintrust-go` (Go), etc.
- Primary docs: https://docs.braintrust.dev

## When to use

Reach for this skill when:
- **Instrumenting**: Adding tracing to capture LLM calls, application logic, and user feedback from AI applications
- **Observing**: Analyzing production logs, filtering traces, discovering patterns, or investigating issues
- **Evaluating**: Running experiments to compare prompts, models, or agent configurations; building datasets; writing scorers
- **Deploying**: Versioning and deploying prompts, functions, or using the Braintrust Gateway for multi-provider routing
- **Annotating**: Adding human feedback, labels, or corrections to traces; building evaluation datasets from production
- **Automating**: Setting up online scoring rules, automations, or CI/CD integration for regression detection

Do not use for: Account creation, billing, or SSO setup (admin-only tasks).

## Quick reference

### Authentication and setup

| Task | Command/Code |
|------|--------------|
| Create API key | Go to Settings > API keys in Braintrust UI |
| Set API key | `export BRAINTRUST_API_KEY="your-key"` |
| Initialize project | `bt init` (creates `.bt/config.json`) |
| Switch project/org | `bt switch` |
| Check current context | `bt status` |

### Core SDK patterns

| Pattern | TypeScript | Python |
|---------|-----------|--------|
| Initialize logger | `initLogger({ projectName: "..." })` | `braintrust.init_logger(project="...")` |
| Log a trace | `logger.log({ input, output })` | `logger.log(input=..., output=...)` |
| Create experiment | `init({ project: "...", experiment: "..." })` | `braintrust.init(project="...", experiment="...")` |
| Create dataset | `initDataset("project", { dataset: "name" })` | `braintrust.init_dataset(project="...", name="...")` |
| Run evaluation | `Eval({ data, task, scores })` | `braintrust.Eval(data=..., task=..., scores=...)` |
| Flush pending spans | `await logger.flush()` | `logger.flush()` |

### CLI commands

| Command | Purpose |
|---------|---------|
| `bt trace enable <agent>` | Enable tracing for a coding agent (Claude, Cursor, etc.) |
| `bt eval <file>` | Run evaluation from JS/TS/Python/Go file |
| `bt sql "SELECT ..."` | Query logs with SQL |
| `bt view logs` | Browse logs in terminal UI |
| `bt sync pull` | Download logs/experiments as NDJSON |
| `bt sync push` | Upload local NDJSON to Braintrust |
| `bt prompts list` | List prompts in current project |
| `bt datasets list` | List datasets |
| `bt experiments list` | List experiments |

### Environment variables

| Variable | Purpose |
|----------|---------|
| `BRAINTRUST_API_KEY` | API key for authentication (required) |
| `BRAINTRUST_DEFAULT_PROJECT` | Default project name for spans |
| `BRAINTRUST_ORG_NAME` | Organization name (when accessing multiple orgs) |
| `BRAINTRUST_DEBUG` | Set to `true` for debug logging |
| `BRAINTRUST_ENABLE_TRACE_CONSOLE_LOG` | Set to `true` to print spans to console |

## Decision guidance

### When to use logger vs. experiment

| Use case | Logger | Experiment |
|----------|--------|-----------|
| Capture production traces | ✓ | — |
| Run one-off evaluations | — | ✓ |
| Build datasets from logs | ✓ | — |
| Compare multiple runs | — | ✓ |
| Mutable/iterative results | ✓ | — |
| Immutable snapshots | — | ✓ |

### When to use playground vs. experiment vs. online scoring

| Approach | Use for | Mutability |
|----------|---------|-----------|
| Playground | Rapid iteration, prototyping | Mutable (overwrites) |
| Experiment | Immutable snapshots, CI/CD, tracking progress | Immutable |
| Online scoring | Production monitoring, continuous evaluation | Async, real-time |

### When to use code scorer vs. LLM-as-a-judge vs. autoeval

| Scorer type | Best for | Pros | Cons |
|-------------|----------|------|------|
| Code scorer | Deterministic checks (exact match, format) | Fast, cheap, precise | Requires logic |
| LLM-as-a-judge | Subjective quality (helpfulness, tone) | Flexible, handles nuance | Slower, more expensive |
| Autoeval | Common patterns (similarity, relevance) | Pre-built, no code | Limited to built-in metrics |

## Workflow

### Typical task: Instrument, evaluate, and improve an AI application

1. **Set up authentication**
   - Create API key in Braintrust UI (Settings > API keys)
   - Set `BRAINTRUST_API_KEY` environment variable
   - Run `bt init` to link your project directory

2. **Instrument your application**
   - Install SDK: `npm install braintrust` (TypeScript) or `pip install braintrust` (Python)
   - Call `initLogger()` at application startup
   - Wrap LLM calls with auto-instrumentation or manual `logger.log()`
   - Deploy to production; traces flow to Braintrust automatically

3. **Observe production behavior**
   - Go to Logs page in Braintrust UI
   - Filter and search traces using SQL or UI filters
   - Use Loop to ask natural language questions about your data
   - Enable Patterns to surface recurring issues automatically

4. **Build evaluation datasets**
   - Identify interesting traces in production logs
   - Add human feedback (scores, corrections, labels)
   - Promote annotated traces to a dataset
   - Alternatively, create dataset from scratch with `initDataset()`

5. **Write scorers**
   - For deterministic checks: write code-based scorer
   - For subjective quality: use LLM-as-a-judge scorer
   - Test scorer on obvious cases (correct output should score ~1, wrong ~0)

6. **Run experiments**
   - Create experiment with `init({ experiment: "name" })`
   - Define task function (what to evaluate)
   - Pass dataset and scorers to `Eval()`
   - Compare results against baseline experiment

7. **Deploy improvements**
   - Promote winning prompt/model to production
   - Use `logger.invoke()` to call deployed prompts
   - Set up online scoring rules to monitor production quality
   - Create dashboards to track metrics over time

### Typical task: Run an evaluation from code

```typescript
import { Eval, initDataset } from "braintrust";

const dataset = initDataset("My Project", { dataset: "test-cases" });

await Eval("My Project", {
  data: dataset,
  task: async (input) => {
    // Your task: call model, agent, or function
    return await myModel.call(input);
  },
  scores: [
    {
      name: "correctness",
      score: async (input, output, expected) => {
        // Return 0-1 score
        return output === expected ? 1 : 0;
      },
    },
  ],
  maxConcurrency: 10, // Limit concurrency to avoid rate limits
});
```

## Common gotchas

- **Missing API key**: Always set `BRAINTRUST_API_KEY` before running code. Check with `echo $BRAINTRUST_API_KEY`.
- **Forgetting to flush**: Call `logger.flush()` before process exit to ensure pending spans are sent. Use `BRAINTRUST_FLUSH_ON_EXIT=true` (default) to auto-flush.
- **Mixing logger and experiment spans**: Don't use `initLogger()` and `init()` in the same trace hierarchy. Use separate instances.
- **Unlimited concurrency in evals**: Set `maxConcurrency` to avoid "too many open files" errors. Start with 10, tune up/down.
- **Scorer not testing obvious cases**: Always test scorers on inputs where the correct score is clear (1 for correct, 0 for wrong). If it fails, fix the scorer before trusting results.
- **Experiment name collisions**: Experiment names must be unique within a project. Use timestamps or version numbers to avoid conflicts.
- **Dataset search limited to 1000 records**: Use "Add to dataset" search in UI for larger datasets, or query via API.
- **Traces stuck in progress**: Ensure spans are flushed and process exits cleanly. Check logs for errors.
- **Custom code scorers can't import packages**: Inline all dependencies or use LLM-as-a-judge instead.
- **OTEL payload limits**: Large spans may be rejected. Keep individual span payloads under size limits; split if needed.

## Verification checklist

Before submitting work with Braintrust:

- [ ] API key is set and valid (`bt status` shows correct org/project)
- [ ] Traces are being logged (check Logs page in UI or `bt view logs`)
- [ ] Spans are flushed before process exit (`logger.flush()` called or `BRAINTRUST_FLUSH_ON_EXIT=true`)
- [ ] Experiment names are unique within the project
- [ ] Scorers tested on obvious cases (correct output scores ~1, wrong ~0)
- [ ] Dataset has sufficient examples (10-200 for most use cases)
- [ ] Concurrency is limited in evals (`maxConcurrency` set to avoid resource exhaustion)
- [ ] Comparison baseline is set when comparing experiments
- [ ] Online scoring rules have appropriate filters and sample rates
- [ ] Dashboards and alerts are configured for production monitoring

## Resources

- **Comprehensive navigation**: https://docs.braintrust.dev/llms.txt
- **Getting started**: https://docs.braintrust.dev (main docs site)
- **Workflow overview**: https://docs.braintrust.dev/workflow
- **Instrument your app**: https://docs.braintrust.dev/instrument
- **Run evaluations**: https://docs.braintrust.dev/evaluate/run-evaluations
- **CLI reference**: https://docs.braintrust.dev/reference/cli/overview
- **SDK reference**: https://docs.braintrust.dev/sdks (TypeScript, Python, Go, Java, Ruby, C#)
- **API reference**: https://docs.braintrust.dev/api-reference

---

> For additional documentation and navigation, see: https://docs.braintrust.dev/llms.txt