bt scorers

bt scorers manages Braintrust scorers from the CLI. Use bt scorers create to create prompt-based LLM scorers and classifiers. The other subcommands are aliases for bt functions scoped to scorers.

Subcommands

Subcommand	Description
`bt scorers list`	List scorers in the current project
`bt scorers create [name]`	Create a prompt-based LLM scorer or classifier
`bt scorers view [slug]`	View a scorer’s details
`bt scorers delete [slug]`	Delete a scorer
`bt scorers invoke [slug]`	Invoke a scorer

Examples

bt scorers list
bt scorers view my-scorer
bt scorers create "Helpfulness" --model gpt-5.4-nano --messages @messages.json \
  --choice-scores '{"A":1,"B":0}'
bt scorers delete my-scorer

--messages, --choice-scores, --classifications, --metadata, and --response-format accept inline values, @PATH to read from a file, or - when piping content from stdin. Braintrust validates a scorer definition before saving it. Validation errors identify invalid or unsupported model parameters and may suggest a corrected value or recommend removing the parameter. With --json, validation errors appear in error.message.

Command-specific flags

Command	Flag	Description
`bt scorers create`	`--name <NAME>`	Scorer name (alternative to the positional name)
`bt scorers create`	`--slug`, `-s <SLUG>`	Unique scorer slug. Defaults to a slug generated from the name
`bt scorers create`	`--description`, `-d <TEXT>`	Scorer description
`bt scorers create`	`--messages <SOURCE>`	Chat messages as a JSON array. Required
`bt scorers create`	`--model`, `-m <MODEL>`	Model used by the LLM judge. Required
`bt scorers create`	`--choice-scores <SOURCE>`	Choice-to-score mapping for score output. Scores must be between 0 and 1. Required unless `--classifications` is provided. Cannot be used with `--classifications`
`bt scorers create`	`--classifications <SOURCE>`	Classification labels as a JSON array. Required unless `--choice-scores` is provided. Cannot be used with `--choice-scores`
`bt scorers create`	`--allow-no-match`	Allow a classifier to return no matching classification. Requires `--classifications`
`bt scorers create`	`--use-cot[=true\|false]`	Whether the scorer should use chain-of-thought reasoning. Defaults to `true`
`bt scorers create`	`--pass-threshold <NUMBER>`	Score threshold for passing, between 0 and 1. Cannot be used with `--classifications`
`bt scorers create`	`--metadata <SOURCE>`	Metadata as YAML
`bt scorers create`	`--preprocessor <SOURCE>`	Preprocessor that builds trace template variables. See Preprocessors
`bt scorers create`	`--preprocessor-export <NAME>`	Named or default export to inline from a `--preprocessor @PATH` file with multiple definitions
`bt scorers create`	`--if-exists <MODE>`	Conflict behavior when a scorer with the same slug already exists: `error`, `ignore`, or `replace`. Defaults to `error`
`bt scorers view`	`--slug`, `-s <SLUG>`	Scorer slug (alternative to the positional argument)
`bt scorers view`	`--id <ID>`	Function ID selector
`bt scorers view`	`--version <VERSION>`	Version selector
`bt scorers view`	`--web`	Open the scorer in your browser
`bt scorers delete`	`--slug`, `-s <SLUG>`	Scorer slug (alternative to the positional argument)
`bt scorers delete`	`--force`, `-f`	Skip the confirmation prompt
`bt scorers invoke`	`--slug`, `-s <SLUG>`	Scorer slug (alternative to the positional argument)
`bt scorers invoke`	`--input`, `-i <INPUT>`	JSON input to the scorer
`bt scorers invoke`	`--message`, `-m <MESSAGE>`	User message. Pass multiple times to provide multiple messages
`bt scorers invoke`	`--mode <MODE>`	Response format: `auto`, `json`, `text`, or `parallel`
`bt scorers invoke`	`--version <VERSION>`	Pin to a specific function version

Preprocessors

A preprocessor turns a trace into the messages a scorer receives for trace template variables such as {{preprocessed}}. In bt v0.26.0 or later, --preprocessor on bt scorers create accepts:

<SLUG>: A preprocessor pushed to the project, for example with bt preprocessors push.
global:<NAME>: A built-in preprocessor, for example global:thread.
@PATH: A local preprocessor file, inlined into the scorer. If the file exports more than one definition, select one with --preprocessor-export <NAME>, or --preprocessor-export default for the default export.
none: Disable preprocessing.

Without --preprocessor, the project’s default preprocessor applies.

bt scorers create "Helpfulness" --model gpt-5.4-nano --messages @messages.json \
  --choice-scores '{"A":1,"B":0}' --preprocessor @./braintrust-preprocessors/conversation.preprocessor.ts

Prompt options

bt scorers create also accepts prompt configuration flags:

Flag	Description
`--temperature <NUMBER>`	Sampling temperature, between 0 and 2. Some models support a smaller range or do not support custom temperatures
`--max-tokens <N>`	Maximum number of generated tokens. Must be greater than 0
`--top-p <NUMBER>`	Nucleus sampling probability, between 0 and 1
`--top-k <N>`	Top-k sampling value, between 1 and 100. Availability depends on the model provider
`--frequency-penalty <NUMBER>`	Frequency penalty. Availability and range depend on the model
`--presence-penalty <NUMBER>`	Presence penalty. Availability and range depend on the model
`--stop-sequence <TEXT>`	Stop sequence. Repeat to specify multiple sequences
`--tool-choice <CHOICE>`	Tool choice: `auto`, `none`, `required`, or a specific function name
`--reasoning-effort <EFFORT>`	Reasoning effort for supported models: `none`, `minimal`, `low`, `medium`, or `high`
`--reasoning-enabled[=true\|false]`	Enable reasoning for models that configure reasoning with a token budget
`--reasoning-budget <N>`	Reasoning token budget, between 0 and 32768, for supported models
`--verbosity <VERBOSITY>`	Response verbosity for supported models: `low`, `medium`, or `high`
`--use-cache[=true\|false]`	Whether to use Braintrust’s completion cache
`--response-format <FORMAT\|SOURCE>`	Response format: `text`, `json-object`, or a response format JSON object
`--template-format <FORMAT>`	Prompt template syntax: `mustache`, `jinja`, or `none`. Jinja is stored as Braintrust’s `nunjucks` format; `nunjucks` and `jinja2` are accepted aliases

For shared function behavior and flags, see bt functions.

API

CLI

Terraform

SQL

Subcommands

Examples

Command-specific flags

Preprocessors

Prompt options

​Subcommands

​Examples

​Command-specific flags

​Preprocessors

​Prompt options

Subcommands

Examples

Command-specific flags

Preprocessors

Prompt options