> ## Documentation Index > Fetch the complete documentation index at: https://docs.braintrust.dev/llms.txt > Use this file to discover all available pages before exploring further. # bt eval > Run JavaScript, Python, and Go evaluation files against Braintrust Run evaluation files against Braintrust. Supports JavaScript, Python, and Go. > [!NOTE] > > `bt eval` is currently macOS and Linux only. ## File selection * `bt eval` — discover and run all eval files in the current directory (recursive) * `bt eval tests/` — discover eval files under a specific directory * `bt eval "tests/**/*.eval.ts"` — glob pattern * `bt eval a.eval.ts b.eval.ts` — one or more explicit files When you run `bt eval` with no arguments, files inside `node_modules`, `.venv`, `venv`, `site-packages`, `dist-packages`, `__pycache__`, and `vendor` are excluded from discovery. Explicit files, directories, and globs bypass these exclusions. ## Runtime configuration > [tab] TypeScript > > Requires Node.js 18.19.0+ or 20.6.0+. Bun 1.0+ and Deno with Node compatibility mode are also supported. > > By default, `bt eval` auto-detects an available runner (`tsx`, `vite-node`, `ts-node`, `ts-node-esm`, then `deno`), checking project-local binaries before `PATH`. Set one explicitly with `--runner` / `BRAINTRUST_EVAL_RUNNER`: > > ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} > bt eval --runner vite-node tutorial.eval.ts > bt eval --runner tsx tutorial.eval.ts > ``` > > `bt eval` automatically resolves locally installed binaries from `node_modules/.bin`, so you can write `--runner tsx` instead of `--runner ./node_modules/.bin/tsx` (for example). If you see ESM or top-level await errors, try `--runner vite-node`. > [tab] Python > > Use `--language python` to force language detection. By default, if `VIRTUAL_ENV` is set, `bt` uses that virtualenv's Python. Otherwise, it searches `PATH` for `python3` or `python`. To use a specific interpreter, set `BRAINTRUST_EVAL_PYTHON_RUNNER` to its name or path (e.g. `python3.11`). The `--num-workers` flag controls concurrency for Python execution. > > ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} > bt eval my_eval.py > bt eval --language python --num-workers 4 my_eval.py > ``` > [tab] Go > > > [!WARNING] > > > > Go evals are in [public preview](https://docs.braintrust.dev/feature-lifecycle.md) and can change before reaching general availability. > > A Go eval is a `package main` that uses the [Go SDK](https://docs.braintrust.dev/sdks/go/api-reference.md#remote-evals)'s `evalrunner` package. Go has no runtime loading, so `bt` doesn't supply a runner. Instead, `bt` compiles your package and runs the resulting program. > > **Run a package** > > Go compiles a directory rather than a single file, so point `bt` at the package directory: > > ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} > bt eval ./cmd/evals > ``` > > **Automatic discovery** > > When a directory contains no JavaScript or Python eval files, `bt` runs `go list` to find the `package main` packages that use `evalrunner`. > > * Eval files need no special name. > * Discovery requires a `go.mod` file and a Go toolchain. With `bt` v0.27.0 or later, discovery finds the `go.mod` file in a parent directory, so it works from any folder in the module. > * Discovery must find exactly one Go eval package. If it finds more than one, `bt` exits with an error, so pass the package directory you want. > * With `bt` v0.27.1 or later, when discovery fails inside a Go module, the error names the cause, such as a missing Go toolchain, a failing `go list`, or no package that imports `evalrunner`. > * Package layout comes from `go list -json`, so the Go toolchain handles build constraints, `go.work`, and vendoring. > > **Force Go** > > Use `--language go` when a directory also contains JavaScript or Python eval files. Pass the package directory itself, because `bt` doesn't search its subdirectories: > > ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} > bt eval --language go ./cmd/evals > ``` > > With `bt` v0.27.0 or later, `bt` exits with an error if the package doesn't import `evalrunner`. > > **Run a prebuilt binary** > > To run without a Go toolchain (`bt` v0.27.1 or later), build the binary ahead of time and pass it to `--runner`. `bt` runs the binary with no arguments and passes everything it needs through environment variables: > > ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} > go build -o bin/evals ./cmd/evals > bt eval --runner ./bin/evals > ``` > > `bt` detects a Go eval binary on its own when `--runner` is a path, such as `./bin/evals`, and any other inputs are directories. Add `--language go` when you pass a bare name that `bt` finds on `PATH`, or when the binary is packed by a tool such as UPX. With `--language go`, `bt` ignores file and directory arguments. > > With `--watch`, `bt` re-runs when the binary changes, so rebuild it to trigger a run. > > **Choose the Go toolchain** > > `bt` uses the first of these that is set or found: > > 1. `BRAINTRUST_EVAL_GO_BIN` > 2. `BRAINTRUST_EVAL_GO` > 3. `$GOROOT/bin/go` > 4. `go` on `PATH` > > Set one of the variables when a version manager such as mise or asdf keeps the toolchain off your default `PATH`. `bt` uses a variable's value as-is, so an incorrect path causes an error instead of falling back to the next option. In `bt` versions before v0.26.0, the variables are named `BT_EVAL_GO_BIN` and `BT_EVAL_GO`. See the [v0.26.0 migration instructions](https://docs.braintrust.dev/reference/cli/migrate.md#v0-26-0). > > > [!NOTE] > > > > The first run of a package with dependencies includes compile time before any results appear. When `bt` builds the package, `--watch` re-runs only when the package's own `.go` files change. It doesn't detect changes in imported packages. ## Evaluator concurrency Use `--max-concurrency` to limit how many top-level evaluators run at the same time. This limit does not change the task or scorer concurrency configured inside an individual evaluator. ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} bt eval --max-concurrency 2 tests/ ``` The value must be greater than zero. `--max-concurrency` cannot be used with `--dev`. Go evals require Go SDK v0.18.0 or later to run evaluators concurrently. Earlier versions run them one at a time. ## JavaScript auto-instrumentation For JavaScript and TypeScript evals, `bt eval` applies Braintrust auto-instrumentation before it loads your eval files. This lets supported AI clients and instrumented libraries create trace spans automatically, including clients constructed inside dependencies. If the installed Braintrust SDK cannot load the auto-instrumentation hook, `bt` prints a warning and continues without it. To disable auto-instrumentation for a run, pass `--no-auto-instrumentation` or set `BRAINTRUST_EVAL_NO_AUTO_INSTRUMENTATION=1`. ## Sampling modes Run a subset of your evaluation data as a non-final smoke run to catch obvious regressions before committing to the full dataset. ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} bt eval --first 20 qa.eval.ts # First 20 examples, non-final bt eval --sample 20 qa.eval.ts # Random 20 examples, non-final bt eval --sample 20 --sample-seed 7 qa.eval.ts # Reproducible random sample bt eval qa.eval.ts # Full dataset, final ``` > [!NOTE] > > When `--first` or `--sample` is used, the experiment summary is labeled as non-final in Braintrust. Omitting both flags runs the full dataset and marks the summary as final. ## Flags | Flag | Env var | Description | | - | - | - | | `-r` / `--runner ` | `BRAINTRUST_EVAL_RUNNER` | Runner binary (`tsx`, `bun`, `ts-node`, `python`, etc.), or a prebuilt Go eval binary | | `-l` / `--language ` | `BRAINTRUST_EVAL_LANGUAGE` | Force language: `java-script`, `python`, or `go` | | `--filter ` | `BRAINTRUST_EVAL_FILTER` | Run only evaluators matching the pattern | | `--first ` | `BRAINTRUST_EVAL_FIRST` | Run only the first N examples (non-final smoke run) | | `--sample ` | `BRAINTRUST_EVAL_SAMPLE` | Run a deterministic random sample of N examples (non-final smoke run) | | `--sample-seed ` | `BRAINTRUST_EVAL_SAMPLE_SEED` | Integer seed for `--sample` | | `--param ` | | Pass a named parameter into evaluators that declare a parameters schema (repeatable; also accepts a JSON object string) | | `--matrix-param ` | | Run one experiment per Cartesian-product combination of parameter values (repeatable). Requires exactly one evaluator (use `--filter` to select it). Incompatible with `--watch`, `--dev`, and `--list` | | `--watch` / `-w` | `BRAINTRUST_EVAL_WATCH` | Re-run when input files change | | `--no-send-logs` | `BRAINTRUST_EVAL_LOCAL` | Run without sending results to Braintrust | | `--max-concurrency ` | `BRAINTRUST_EVAL_MAX_CONCURRENCY` | Maximum number of top-level evaluators to run concurrently. Cannot be used with `--dev` | | `--num-workers ` | `BRAINTRUST_EVAL_NUM_WORKERS` | Worker threads for Python execution | | `--verbose` / `-v` | `BRAINTRUST_VERBOSE` | Show full errors and stderr from eval files | | `--list` | `BRAINTRUST_EVAL_LIST` | List evaluators without running them | | `--jsonl` | `BRAINTRUST_EVAL_JSONL` | Output one JSON summary per evaluator (for scripts). See also the global `--json` flag ([overview](https://docs.braintrust.dev/reference/cli/overview.md#global-flags)), which formats all CLI output as JSON rather than per-evaluator summaries. | | `--terminate-on-failure` | `BRAINTRUST_EVAL_TERMINATE_ON_FAILURE` | Stop after the first failing evaluator | | `--no-auto-instrumentation` | `BRAINTRUST_EVAL_NO_AUTO_INSTRUMENTATION` | Disable automatic instrumentation for JavaScript and TypeScript evals | | `--dev` | `BRAINTRUST_EVAL_DEV` | Start a local web server for browser-based eval development | | `--dev-host ` | `BRAINTRUST_EVAL_DEV_HOST` | Host interface for the eval dev server (default: `localhost`) | | `--dev-port ` | `BRAINTRUST_EVAL_DEV_PORT` | Port for the eval dev server (default: `8300`) | | `--dev-org-name ` | `BRAINTRUST_EVAL_DEV_ORG_NAME` | Restrict eval dev server access to a specific org | | `--dev-allowed-origin ` | `BRAINTRUST_EVAL_DEV_ALLOWED_ORIGIN` | Additional allowed CORS origin(s) for the eval dev server (repeatable; env var accepts comma-separated values) | ## Summary output When using `--jsonl` or reading SSE output, each evaluator summary object includes these fields: | Field | Type | Description | | - | - | - | | `runMode` | `"full"` \| `"first"` \| `"sample"` | How the eval was run | | `isFinal` | `boolean` | Whether this is a final (full-dataset) run | | `runLabel` | `string` | Human-readable description of the run mode | | `sampleCount` | `number` | Number of examples sampled (only present when `--first` or `--sample` is used) | | `sampleSeed` | `number` | Seed used for random sampling (only present when `--sample` is used) | ## Parameters `--param` overrides values for evaluators that declare a `parameters` schema via `loadParameters()` (TypeScript) or `load_parameters()` (Python). This is the same parameters system used by [remote evals](https://docs.braintrust.dev/evaluate/remote-evals.md), where parameters are version-tracked in Braintrust and appear as UI controls in the playground. See [Create evaluation parameters](https://docs.braintrust.dev/evaluate/write-parameters.md) for how to define and load parameters. Each evaluator only receives the keys it declares. Extra keys are silently filtered, so a single command can target multiple evaluators with different schemas without errors. ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} bt eval --param model=gpt-4o --param count=5 my.eval.ts bt eval --param '{"model":"gpt-4o","count":5}' my.eval.ts ``` Parameters are validated against the evaluator's declared schema before execution. Evaluators without a `parameters` schema are unaffected. ## Parameter matrix `--matrix-param` works with the same parameters system as `--param`. Specify multiple values for one or more parameters and `bt eval` runs one experiment per combination, naming each ` [key=value, ...]`. ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} # Sweep a single parameter across three values bt eval --matrix-param model=gpt-4o,gpt-4o-mini,o1-mini my.eval.ts # Sweep two parameters — runs 2 × 3 = 6 experiments bt eval --matrix-param model=gpt-4o,gpt-4o-mini --matrix-param temperature=0.0,0.5,1.0 my.eval.ts ``` For values that contain commas, use a JSON array: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} bt eval --matrix-param model='["gpt-4o","claude-3-5-haiku-20241022"]' my.eval.ts ``` `--matrix-param` requires exactly one evaluator to be selected. If your file exports multiple evaluators, use `--filter` to narrow down to one. It is not supported for eval files that export `btEvalMain`. ## Passing arguments to the eval file Use `--` to forward extra arguments to the eval file via `process.argv`: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} bt eval foo.eval.ts -- --description "Prod" --shard 1/4 ``` ## Running in CI Set `BRAINTRUST_API_KEY` instead of using OAuth login: ```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} # GitHub Actions example - name: Run evals env: BRAINTRUST_API_KEY: ${{ secrets.BRAINTRUST_API_KEY }} run: bt eval tests/ ``` Use `--no-input` and `--json` for non-interactive output: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} BRAINTRUST_API_KEY=... bt eval tests/ --no-input --json ``` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.