> ## Documentation Index
> Fetch the complete documentation index at: https://docs.braintrust.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Harbor

> Sync Harbor agent evaluation results to Braintrust with the native Harbor job plugin, capturing datasets, experiments, scores, and agent traces

[Harbor](https://harborframework.com/) is a Python framework for running AI agent evaluations in isolated Docker containers. Braintrust's native Harbor job plugin syncs evaluation results (datasets, experiments, scores, and agent traces) to Braintrust after each Harbor run.

**Python**

## Setup

Install Braintrust alongside Harbor. Requires `harbor>=0.16.0` and Python 3.12 or later. Docker must be running for Harbor to execute tasks.

**1. Install packages**

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
pip install braintrust harbor
```

**2. Set environment variables**

```bash title=".env" theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
BRAINTRUST_API_KEY=<your-braintrust-api-key>
OPENAI_API_KEY=<your-model-provider-api-key>
```

## Run evaluations

Pass `--plugin braintrust` to `harbor run` to enable the Braintrust plugin. The plugin is discovered automatically through Harbor's `braintrust` entry point after installation.

> [!NOTE]
>
> The Harbor integration is activated through the `--plugin braintrust` flag in the `harbor run` CLI, not through `auto_instrument()`. No import or patching is required in your task code.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
harbor run \
  --path task/ \
  --agent terminus-2 \
  --model openai/gpt-5-mini \
  --job-name my-eval \
  --plugin braintrust \
  --yes
```

By default, results are synced to a Braintrust project named `Harbor`. After the run completes, the plugin creates a sync manifest at `jobs/my-eval/braintrust-sync.json`.

### Configure with environment variables

Set `HARBOR_BRAINTRUST_PROJECT` to override the default project name on every run:

```bash title=".env" theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
HARBOR_BRAINTRUST_PROJECT=my-project
```

Or pass `--plugin-kwarg project_name=my-project` directly:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
harbor run \
  --path task/ \
  --agent terminus-2 \
  --model openai/gpt-5-mini \
  --job-name my-eval \
  --plugin braintrust \
  --plugin-kwarg project_name=my-project \
  --yes
```

### Verifier output

In the default `attachments=all` mode, the score and verification spans include Harbor's captured `test-stdout.txt`, optional `test-stderr.txt`, and `ctrf.json` output when present. To capture only structured CTRF output without raw logs, pass `attachments=structured`. To disable verifier and reward attachments entirely, pass `attachments=none`.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
harbor run \
  --path task/ \
  --agent terminus-2 \
  --model openai/gpt-5-mini \
  --job-name my-eval \
  --plugin braintrust \
  --plugin-kwarg attachments=structured \
  --yes
```

Or set the environment variable:

```bash title=".env" theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
HARBOR_BRAINTRUST_ATTACHMENTS=structured
```

> [!WARNING]
>
> Raw verifier logs are redacted only by `redact_patterns`, so set it when the evaluation environment holds secrets.

Pass `--plugin-kwarg 'redact_patterns=["regex"]'` to `harbor run`, or set `HARBOR_BRAINTRUST_REDACT_PATTERNS` to a JSON array of regex patterns, to redact matching strings from raw log output.

## Backfill a past job

To re-sync a completed Harbor job without re-running the evaluation, use `backfill_job`:

```python title="backfill.py" theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import asyncio
from pathlib import Path
from braintrust.integrations.harbor import backfill_job

asyncio.run(backfill_job(Path("jobs/my-eval")))  # Uses the same project as the initial sync
```

Pass `project_name` to override the project:

```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
asyncio.run(backfill_job(Path("jobs/my-eval"), project_name="my-project"))
```

Backfill uses deterministic dataset and experiment identities, so it reconciles with existing Braintrust data instead of creating duplicates.

## What Braintrust receives

After each Harbor evaluation run, Braintrust receives:

* **A managed dataset** with the task cases used in the evaluation.
* **An experiment row** for each trial, with the agent's final output as the experiment output and the Harbor task's reference answer as the expected output.
* **Scores** derived from Harbor verifier rewards, mapped to Braintrust score fields, with verifier output files (`test-stdout.txt`, `test-stderr.txt`, and `ctrf.json` when present) attached to each score span in `attachments=all` mode (the default). With Harbor v0.25 or later, the trial's `metadata.harbor` records Harbor's `reward_valid` and `failure_cause`, and rewards that Harbor marks invalid (`reward_valid` is `false`) produce no scores and add a warning to `metadata.harbor.warnings`.
* **Agent traces** (when trajectory tracing is enabled), showing the agent's tool calls and model turns using Harbor's ATIF format.

## Resources

* [Harbor documentation](https://harborframework.com/docs)
* [Harbor plugin example](https://github.com/braintrustdata/braintrust-sdk-python/tree/main/examples/harbor)
* [Braintrust Python SDK reference](https://docs.braintrust.dev/sdks/python/versions/latest.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.