API reference

​

Tracing

​

initLogger()

import { initLogger } from "braintrust";

const logger = initLogger({
  projectName: "Support bot",
});

logger.log({
  input: { question: "How do I reset my password?" },
  output: { answer: "Use the account recovery flow." },
  metadata: { route: "/support" },
});

await logger.flush();

• projectName (string): project name for logs. If omitted, logs go to the global project.
• projectId (string): project ID. Takes precedence over projectName.
• apiKey (string): API key. Defaults to BRAINTRUST_API_KEY.
• appUrl (string): Braintrust app URL. Defaults to https://www.braintrust.dev.
• orgName (string): organization name, useful when credentials can access multiple orgs.
• asyncFlush (boolean): defaults to true. Set false to make logger calls like logger.log() awaitable.
• setCurrent (boolean): defaults to true. Controls whether currentLogger() returns this logger and what logger auto-instrumentation logs through.
• forceLogin (boolean): log in again even if the SDK already has credentials.
• fetch (typeof fetch): custom fetch implementation for SDK requests. Takes a function that follows the same API as native fetch.
• debugLogLevel ("error" | "warn" | "info" | "debug" | false): enables SDK troubleshooting output. Set false to also silence the SDK’s default warnings.

• log(event) → string: logs a span and returns its row ID. Returns Promise<string> when the logger uses asyncFlush: false.
• traced(callback, args?) → the callback’s result: creates a root span under the logger and ends it when the callback finishes. Spans started within the callback are nested under the traced span.
• startSpan(args?) → Span: starts a root span that you end manually.
• logFeedback(event) → void: adds feedback scores, expected values, tags, or comments to an existing row.
• updateSpan(event) → void: updates an already-written span by ID. Flush the original span before updating it.
• export() → Promise<string>: a serialized parent string for distributed tracing.
• flush() → Promise<void>: sends pending rows to Braintrust, resolving when they’re all flushed.
• trace(body, options?), debug(body, options?), info(body, options?), warn(body, options?), error(body, options?), fatal(body, options?) → string: emits an application log at that severity level and returns its row ID. options.metadata adds metadata. Returns Promise<string> when the logger uses asyncFlush: false. trace() doesn’t start a trace.
• fmt`...` (template literal tag) → a template object: preserves a template literal and its values for a severity method, such as logger.info(logger.fmt`User ${userId} signed in`). See Preserve message templates.

​

wrapConsole()

import { initLogger, wrapConsole } from "braintrust";

const logger = initLogger({ projectName: "Support bot" });
const unwrapConsole = wrapConsole();

console.info("Payment %s started", "pay_123");

unwrapConsole();
await logger.flush();

• target (Pick<Console, "log" | "trace" | "debug" | "info" | "warn" | "error">): the console to wrap. Defaults to the global console.
• options.logger (Logger): the logger that receives console messages. Defaults to the current logger at the time of each call.

​

currentLogger()

import { currentLogger } from "braintrust";

currentLogger()?.log({ input: "ping", output: "pong" });

​

traced()

import { traced } from "braintrust";

const answer = await traced(
  async (span) => {
    span.log({ input: { question: "What changed?" } });
    const output = await answerQuestion("What changed?");
    span.log({ output });
    return output;
  },
  {
    name: "answerQuestion",
    type: "function",
  },
);

• callback ((span) => R, required): the function to trace. Braintrust passes the span as its argument.
• args (optional): span options.
  • name (string): span name shown in Braintrust.
  • type (SpanType): span type, such as function, task, tool, llm, score, or eval.
  • spanAttributes (Record<PropertyKey, unknown>): extra span attributes.
  • startTime (number): start time as a Unix timestamp in seconds.
  • parent (string | PropagationContext): exported parent span string or W3C trace context.
  • event (ExperimentLogPartialArgs): initial event data to log on the span.
  • propagatedEvent (ExperimentLogPartialArgs): event data propagated from another span.
  • spanId (string): explicit span ID.
  • parentSpanIds (ParentSpanIds | MultiParentSpanIds): explicit parent span IDs.
  • setCurrent (boolean): controls whether currentSpan() returns this span while the callback runs. Defaults to true.
  • asyncFlush (boolean): controls whether the span flushes asynchronously.

​

wrapTraced()

import { wrapTraced } from "braintrust";

const classifyText = wrapTraced(
  async function classifyText(text: string) {
    return "positive";
  },
  {
    name: "classifyText",
    type: "function",
  },
);

await classifyText("Great result");

• fn ((...args) => R, required): the function to wrap. Its return value is returned by the wrapped function.
• args (optional): span options for each call to the wrapped function.
  • name (string): span name shown in Braintrust. If omitted, Braintrust uses the wrapped function’s name.
  • type (SpanType): span type, such as function, task, tool, llm, score, or eval. If omitted, defaults to function.
  • spanAttributes (Record<PropertyKey, unknown>): extra span attributes.
  • startTime (number): start time as a Unix timestamp in seconds.
  • parent (string | PropagationContext): exported parent span string or W3C trace context.
  • event (ExperimentLogPartialArgs): initial event data to log on the span.
  • propagatedEvent (ExperimentLogPartialArgs): event data propagated from another span.
  • spanId (string): explicit span ID.
  • parentSpanIds (ParentSpanIds | MultiParentSpanIds): explicit parent span IDs.
  • setCurrent (boolean): controls whether currentSpan() returns this span while the wrapped function runs. Defaults to true.
  • asyncFlush (boolean): controls whether the span flushes asynchronously.
  • noTraceIO (boolean): for generator functions only, prevents wrapTraced() from automatically logging arguments as input and collected yielded values as output. For other functions, use traced() and call span.log() yourself.

​

startSpan()

import { startSpan } from "braintrust";

const span = startSpan({ name: "retrieve", type: "function" });

try {
  span.log({ input: { query: "password reset" } });
  const documents = await retrieveDocuments("password reset");
  span.log({ output: documents });
} finally {
  span.end();
}

• args (optional): span options.
  • name (string): span name shown in Braintrust.
  • type (SpanType): span type, such as function, task, tool, llm, score, or eval.
  • spanAttributes (Record<PropertyKey, unknown>): extra span attributes.
  • startTime (number): start time as a Unix timestamp in seconds.
  • parent (string | PropagationContext): exported parent span string or W3C trace context.
  • event (ExperimentLogPartialArgs): initial event data to log on the span.
  • propagatedEvent (ExperimentLogPartialArgs): event data propagated from another span.
  • spanId (string): explicit span ID.
  • parentSpanIds (ParentSpanIds | MultiParentSpanIds): explicit parent span IDs.
  • asyncFlush (boolean): controls whether the span flushes asynchronously.

​

currentSpan()

import { currentSpan } from "braintrust";

currentSpan().log({
  metadata: { cacheHit: true },
});

​

withCurrent()

import { startSpan, withCurrent, currentSpan } from "braintrust";

const span = startSpan({ name: "manual parent" });

await withCurrent(span, async () => {
  currentSpan().log({ metadata: { inside: true } });
});

span.end();

• span (Span, required): the span to make current inside the callback.
• callback ((span) => R, required): the work to run with span current. Its return value is returned.

​

withParent()

import { initLogger, startSpan, traced, withParent } from "braintrust";

initLogger({ projectName: "Support bot" });

// parent is a string, meaning it is serializable and can be transferred between processes or services.
const parent = await startSpan({ name: "foo" }).export();

await withParent(parent, async () => {
  await traced(async (span) => {
    span.log({ input: "child work" });
  });
});

• parent (string, required): an exported parent string, from a span’s or logger’s export().
• callback (() => R, required): the work to run under that parent. Its return value is returned.

​

injectTraceContext()

import { injectTraceContext, wrapTraced } from "braintrust";

const processRequest = wrapTraced(async function processRequest(request) {
  return await fetch("/api/process", {
    method: "POST",
    body: request,
    // Pass your existing headers so trace context is merged into them rather
    // than replacing them.
    headers: injectTraceContext({ "Content-Type": "application/json" }),
  });
});

• carrier (optional): an existing header carrier to add trace context to and return, such as a Headers instance, a plain object, or an ordinary Node.js http request object. Pass one to combine trace-context headers with the headers you are already sending. Omit it to get a fresh headers object.
• options.span (Span, optional): the span to inject. Defaults to the current span.

​

extractTraceContextFromHeaders()

import { extractTraceContextFromHeaders, traced } from "braintrust";

async function handleRequest(req: Request) {
  const parent = extractTraceContextFromHeaders(req.headers);

  return await traced(
    async (span) => {
      const result = await processData(req.body);
      span.log({ input: req.body, output: result });
      return result;
    },
    { name: "handler", parent },
  );
}

• headers (required): the inbound request headers, such as a Web Request’s Headers, a plain object, or an ordinary Node.js http request object. Header lookups are case-insensitive.

​

logError()

import { currentSpan, logError } from "braintrust";

try {
  await runStep();
} catch (error) {
  logError(currentSpan(), error);
  throw error;
}

• span (Span, required): the span to attach the error to.
• error (unknown, required): the caught error, whose message and stack are logged.

​

permalink()

import { traced, permalink } from "braintrust";

const slug = await traced((span) => span.export(), { name: "request" });
const url = await permalink(slug);

​

flush()

import { flush } from "braintrust";

await flush();

​

setMaskingFunction()

import { setMaskingFunction } from "braintrust";

setMaskingFunction((value) => {
  // `value` is a logged field's full value, which may be a string, array, or
  // object. Walk it and return a masked copy with any sensitive data redacted.
  return value;
});

​

configureInstrumentation()

import { configureInstrumentation } from "braintrust/instrumentation";

configureInstrumentation({
  spanCustomizers: [
    {
      onSpanExport(data) {
        if ("output" in data) data.output = "[redacted]";
        return data;
      },
    },
  ],
});

const { initLogger } = await import("braintrust");
const { default: OpenAI } = await import("openai");

• integrations (InstrumentationIntegrationsConfig): an object of boolean flags keyed by integration name, such as openai, anthropic, aisdk, googleGenAI, groq, and langchain. All integrations are enabled by default. Setting a flag to false doesn’t disable the integration. To disable integrations, use the BRAINTRUST_DISABLE_INSTRUMENTATION environment variable. The livekit and mcp keys also accept options objects. See the LiveKit Agents and MCP SDK integration guides for configuration details.
• spanCustomizers (readonly SpanCustomizer[]): span customizers that modify each outgoing record from an SDK span before upload. They run in array order. A later call that passes spanCustomizers replaces the list.
• streamingAttachmentBufferBytes (number): target size in bytes of each attachment that instrumentation uploads from streaming media, such as LiveKit Agents call audio. Defaults to 4 MiB (4194304). Must be a positive safe integer. Other values throw an error. Instrumentation splits media only at valid boundaries, and whole-file responses can exceed this size. Audio recordings count raw PCM before compression, so compressed uploads are smaller. It doesn’t limit total memory and doesn’t turn on attachment capture.

​

SpanCustomizer

• onSpanExport(data) → SpanExportData: called synchronously on each record the SDK logs. The SDK logs a span as several records as its data becomes available, such as one with the input and a later one with the output, so the method runs several times per span. See How customizers run. SpanExportData is Record<string, unknown>. Return the modified record or a replacement plain object.

• Customizers run on all native SDK spans, including spans you create with traced(), startSpan(), and logger.log(). They don’t run on dataset rows or feedback. In v3.35.0, they run only on spans created by instrumentation and wrappers.
• With v3.38.0 or later and @braintrust/otel v1.1.0 or later, customizers also run on OpenTelemetry spans that BraintrustSpanProcessor or BraintrustExporter sends to Braintrust, and in OpenTelemetry compatibility mode.
• Each record holds only part of the span, so a field can be absent from any single call. Check for a field before you change it.
• The SDK restores protected fields, such as id, span_id, root_span_id, and project_id, after every call.

​

Evaluations

​

Eval()

import { Eval } from "braintrust";

await Eval("My Project", {
  data: [
    {
      input: "How do I reset my password?",
      expected: "Use the password reset flow.",
    },
  ],
  task: async (input) => {
    return await answerQuestion(input);
  },
  scores: [
    ({ output, expected }) => ({
      name: "exact_match",
      score: output === expected ? 1 : 0,
    }),
  ],
});

await Eval("Support bot", {
  data: [{ input: "ping", expected: "pong" }],
  timeout: 30_000,
  task: async (input, { signal }) => {
    const response = await fetch("https://example.com/answer", {
      method: "POST",
      body: input,
      signal,
    });
    return await response.text();
  },
  scores: [({ output, expected }) => (output === expected ? 1 : 0)],
});

​

defineWorkflowEval()

import {
  WorkflowBatchTask,
  WorkflowEvalMemoryStore,
  defineWorkflowEval,
} from "braintrust";

const evaluation = defineWorkflowEval<string, string, string>(
  "capital-cities-eval",
  {
    store: new WorkflowEvalMemoryStore(),
    data: [
      { id: "france", input: "Capital of France?", expected: "Paris" },
      { id: "japan", input: "Capital of Japan?", expected: "Tokyo" },
    ],
    task: new WorkflowBatchTask({
      batching: { maxSize: 100 },
      submit: (items) =>
        submitBatch(
          items.map(({ customId, input }) => ({
            customId,
            prompt: `${input} Reply with only the city name.`,
          })),
        ),
      completion: { mode: "poll", poll: pollBatch },
      collect: async function* (submission) {
        for await (const entry of collectText(submission)) {
          yield "error" in entry
            ? entry
            : { customId: entry.customId, result: { output: entry.text } };
        }
      },
    }),
    scores: [exactMatch],
  },
);

​

Workflow evaluation methods

import { setTimeout as sleep } from "node:timers/promises";

let result = await evaluation.start();
while (result.status === "waiting") {
  await sleep(60_000);
  result = await evaluation.poll({ runId: result.runId });
}
console.log(result.summary);

• start(options?): creates a new run, saves the cases, and submits the first requests.
  • options.parameters: runtime values for the evaluation’s parameters.
  • options.noSendLogs: true: builds the summary locally without sending logs to Braintrust.
  • Every call starts a new run and submits every request to the provider again. To continue a run, pass its runId to poll() or processSubmissionResult() instead.
  • Without a workflow task or scorer, start() runs the whole evaluation at once, like Eval().
• status({ runId }): reads saved progress without contacting the provider or advancing work.
• poll({ runId }): checks existing polling submissions once, collects completed results, and advances the cases that are ready. Scorer requests submitted during this call are polled on a later call. It also resubmits any submission whose submit callback threw, for both polling and webhook completion.
• processSubmissionResult({ runId?, submissionId?, externalId? }): tells the SDK that one request finished, so it collects the result and advances the cases in it. Call it from your webhook handler.
  • Identify the request by externalId, the provider ID your getExternalId function returned, or by submissionId, the SDK’s ID for the request. Pass at least one. If you pass both, they must identify the same submission.
  • runId is required with submissionId. With externalId alone, you can omit runId when each provider ID belongs to only one run of the evaluation. If several runs share the externalId, it throws an error and you must pass runId.
  • Throws Error("No submission matches this result") if the ID doesn’t match a request in this run, and an error if the run isn’t in the store. Throws an error if the submission’s submit callback threw. The next poll() resubmits it.
  • Doesn’t check the webhook event. Before calling it, check that the event came from the provider and that the request succeeded.

• status ("waiting" | "completed"): the run’s status.
• runId (string): identifier to save and pass to status(), poll(), and processSubmissionResult().
• pending ({ poll: number; webhook: number }): number of submitted requests not yet collected, split into poll and webhook counts. A batch counts as one request, and so does a submission waiting to be resubmitted.
• summary (ExperimentSummary): the experiment’s scores and a link to it.

​

WorkflowTask and WorkflowScorer

import { WorkflowTask } from "braintrust";

// In createEvaluation(), this is the task passed to
// defineWorkflowEval<string, string, string>().
task: new WorkflowTask({
  // Called once for each case in data, or once per trial if you set
  // trialCount. Sends the case as a single-request batch.
  submit: ({ input }, context) =>
    submitBatch(
      [
        {
          customId: context.submissionId,
          prompt: `${input} Reply with only the city name.`,
        },
      ],
    ),
  completion: { mode: "poll", poll: pollBatch },
  // Returns the batch's only result. Throwing makes the SDK call collect
  // again on the next poll().
  collect: async (submission) => {
    for await (const entry of collectText(submission)) {
      if ("error" in entry) {
        throw new Error(`Request failed: ${JSON.stringify(entry.error)}`);
      }
      return { output: entry.text };
    }
    throw new Error(`Batch ${submission.batchId} has no result`);
  },
});

• name (string, required for WorkflowScorer): scorer name. Must be unique and non-empty, and must stay the same when you resume a run. WorkflowTask has no name.
• submit(item, context) (required): called to submit the request. WorkflowScorer calls it after the task output is ready. Returns JSON-serializable submission data, which the SDK saves and passes to the completion and collection callbacks.
• completion (required): how the SDK learns that the request finished. See completion configuration.
• collect(submission, context) (required): called after the request completes. WorkflowTask returns one task result, and WorkflowScorer returns one scorer result. Must tolerate replay, including concurrent webhook delivery.

• WorkflowTask: id, input, expected, metadata, tags, parameters, and trialIndex. The id identifies one case and trial, such as "france:trial:0", and trialIndex counts from 0.
• WorkflowScorer: the same fields except parameters, plus the task’s output.

• WorkflowTask: { output, metadata?, tags? }, the task output. Returned metadata merges into case metadata. Returned tags replace case tags.
• WorkflowScorer: { score }, a score in the same formats accepted by plain scorer functions, including a number, a score object, or an array of scores. An unnamed numeric score uses the scorer’s name.

​

WorkflowBatchTask and WorkflowBatchScorer

import { WorkflowBatchScorer, type EvalScorerArgs } from "braintrust";

const llmJudge = new WorkflowBatchScorer({
  // Names the score in Braintrust. Keep it the same across resumes.
  name: "llmJudge",
  // Waits for up to 100 task outputs, or until no more can arrive, then
  // submits them in one grading batch.
  batching: { maxSize: 100 },
  // A scorer defined outside defineWorkflowEval() has no type arguments to
  // infer from, so annotate the items with EvalScorerArgs.
  submit: (
    items: (EvalScorerArgs<string, string, string> & { customId: string })[],
  ) =>
    submitBatch(
      items.map(({ customId, input, output, expected }) => ({
        customId,
        prompt: [
          "Does the answer name the expected city? Reply with only 1 for yes or 0 for no.",
          `Question: ${input}`,
          `Expected: ${expected}`,
          `Answer: ${output}`,
        ].join("\n"),
      })),
    ),
  // Reuses the task's polling callback, since both are OpenAI batches.
  completion: { mode: "poll", poll: pollBatch },
  // Called after pollBatch() reports the grading batch complete.
  collect: async function* (submission) {
    for await (const entry of collectText(submission)) {
      if ("error" in entry) {
        yield entry;
        continue;
      }
      const verdict = entry.text.trim();
      // Records an unexpected reply as a failed score for that case.
      yield verdict === "0" || verdict === "1"
        ? { customId: entry.customId, result: { score: Number(verdict) } }
        : {
            customId: entry.customId,
            error: `Unexpected judge response: ${verdict}`,
          };
    }
  },
});

• name (string, required for WorkflowBatchScorer): scorer name. Must be unique and non-empty, and must stay the same when you resume a run. WorkflowBatchTask has no name.
• batching (required): how the SDK groups items into batches.
  • maxSize (number, required): the maximum number of items in one batch. Must be a positive integer.
  • maxWaitMs (number, WorkflowBatchScorer only): how long a partial batch waits for more items before the SDK submits it. Must be 0 or greater.
• submit(items, context) (required): called to submit one batch. Receives up to maxSize items and returns JSON-serializable submission data, which the SDK saves and passes to the completion and collection callbacks.
• completion (required): how the SDK learns that the batch finished. See completion configuration.
• collect(submission, context) (required): called after the batch completes. Returns one entry per item, as an array, an iterable, or an async iterable. Must tolerate replay, including concurrent webhook delivery.

• { customId, result }: the item’s result. For WorkflowBatchTask, result is { output, metadata?, tags? }. For WorkflowBatchScorer, it’s { score }. Both take the same formats as the results a WorkflowTask or WorkflowScorer returns.
• { customId, error }: the item failed.

• WorkflowBatchTask: start() submits every case and trial, in batches of up to maxSize items.
• WorkflowBatchScorer: submits a batch when maxSize items have task outputs, or when no more items can get one. If you set maxWaitMs, it also submits a partial batch that has waited that long. The SDK checks the wait during each start(), poll(), and processSubmissionResult() call and doesn’t start timers, so a partial batch is submitted on the first call after maxWaitMs passes.

​

Completion configuration

async function pollBatch({ batchId }: Submission) {
  const batch = await client.batches.retrieve(batchId);
  switch (batch.status) {
    case "completed":
    case "failed":
    case "expired":
    case "cancelled":
      return { status: "complete" } as const;
    default:
      return { status: "pending" } as const;
  }
}

// In createEvaluation(), this is the task passed to
// defineWorkflowEval<string, string, string>(), and completionMode is a
// "poll" | "webhook" parameter.
new WorkflowBatchTask({
  batching: { maxSize: 100 },
  submit: (items) =>
    submitBatch(
      items.map(({ customId, input }) => ({
        customId,
        prompt: `${input} Reply with only the city name.`,
      })),
    ),
  completion:
    completionMode === "poll"
      ? { mode: "poll", poll: pollBatch }
      : { mode: "webhook", getExternalId: ({ batchId }) => batchId },
  collect: async function* (submission) {
    for await (const entry of collectText(submission)) {
      yield "error" in entry
        ? entry
        : { customId: entry.customId, result: { output: entry.text } };
    }
  },
});

• { mode: "poll", poll }: poll(submission, context) is called for each pending submission when your application calls poll({ runId }). It returns { status: "pending" }, { status: "complete" }, or { status: "failed", error }. Completion triggers collection. If a callback fails, poll() still checks the other submissions and advances the run before it throws. With one failure, it throws that error. With several, it throws an AggregateError. The submission stays pending, so the next poll() call checks it again.
• { mode: "webhook", getExternalId }: getExternalId(submission, context) is called right after submit returns. It synchronously returns a non-empty provider request ID to associate with the saved submission. Pass that ID as externalId to processSubmissionResult() after verifying a completion event. For a batch task or scorer, return the provider’s batch ID.

​

WorkflowEvalStore

import { createClient } from "redis";
import { WorkflowEvalRedisStore } from "braintrust";

const redis = createClient({ url: process.env.REDIS_URL });
const store = new WorkflowEvalRedisStore({ client: redis });

• new WorkflowEvalMemoryStore(): keeps state in memory for single-process runs. State is lost when the process exits.
• new WorkflowEvalRedisStore({ client, keyPrefix?, ttlMs? }): accepts a connected client from redis (node-redis), ioredis, or @upstash/redis. keyPrefix defaults to "braintrust-eval:". Records and progress sets expire after seven days by default. Set ttlMs to a positive value longer than the full evaluation and expected resumption delays. Close the client when your application shuts down.

• read(key: string): Promise<Uint8Array | undefined>: read the saved value, or return undefined if absent.
• write(key: string, value: Uint8Array): Promise<void>: save a value.
• getOrSet(key: string, value: Uint8Array): Promise<{ value: Uint8Array; created: boolean }>: atomically insert a value only when the key is absent. Return the stored bytes as value and whether this call inserted them as created.
• addToSet(key: string, member: string): Promise<void>: atomically add a unique member to a set.
• getSetSize(key: string): Promise<number>: return the number of unique members.

​

agentAssertionScorer()

import { Eval, agentAssertionScorer } from "braintrust";

await Eval("agent-eval", {
  data: () => [{ input: "What is the capital of Estonia?" }],
  task: async (input) => {
    // Your agent logic here
    return { answer: "Tallinn is the capital of Estonia." };
  },
  scores: [
    agentAssertionScorer(({ output, assert }) => [
      assert.contains(output.answer, /Tallinn/i, "mentions Tallinn"),
      assert.calledTool("web_search", { times: 1 }, "searched once"),
      assert.maxToolCalls(3, "bounded tool use"),
    ]),
  ],
});

​

buildLocalSummary()

import { buildLocalSummary } from "braintrust";

const summary = buildLocalSummary(evaluatorDef, results);

• evaluatorDef (EvaluatorDef, required): your Eval() evaluator object plus the SDK-populated projectName and evalName fields.
  type EvaluatorDef = Evaluator & {
    projectName: string;
    evalName: string;
  };
  
• results (EvalResult[], required): the row-level eval results to summarize.
• precomputedScores (ScoreAccumulator, optional): score totals and counts to use instead of deriving them from results. Pass it when you ran with returnResults: false (so per-row results aren’t available) or you’ve already accumulated scores yourself. It maps each score’s name to a running total and count:
  type ScoreAccumulator = Record<
    string,
    {
      total: number;
      count: number;
    }
  >;
  

​

reportFailures()

import { reportFailures } from "braintrust";

reportFailures(evaluatorDef, failingResults, {
  verbose: true,
  jsonl: false,
});

• evaluatorDef (EvaluatorDef, required): evaluator metadata used in error output.
• failingResults (EvalResult[], required): results whose error field is set.
• options ({ verbose: boolean; jsonl: boolean }, required): verbose prints full errors. jsonl emits a JSON line containing evaluator name and error strings.

​

BaseExperiment()

import { BaseExperiment, Eval } from "braintrust";

await Eval("My Project", {
  data: BaseExperiment({ name: "retrieval-baseline" }),
  task: async (input) => await answerQuestion(input),
  scores: [qualityScorer],
});

1. During experiment registration, the SDK sends ancestor_commits when no explicit baseExperimentId or baseExperimentName is set.
2. ancestor_commits contains up to 1,000 commit hashes from the current branch, starting at the merge base with the default remote base branch and ending at HEAD. For a clean working tree, the merge base is computed from HEAD^; for a dirty working tree, it is computed from HEAD.
3. Braintrust chooses the newest experiment in the same project on the nearest commit in that ancestor list that you can read.
4. The chosen experiment ID is stored as the new experiment’s base experiment and then opened as dataset data for BaseExperiment().

​

Experiments

​

initExperiment()

import { initExperiment } from "braintrust";

const experiment = initExperiment({
  project: "Support bot",
  experiment: "retrieval-baseline",
  description: "Manual retrieval evaluation",
});

experiment.log({
  input: "How do I reset my password?",
  output: "Use the account recovery flow.",
  expected: "Follow the password reset instructions.",
  scores: { correctness: 0.8 },
});

const summary = await experiment.summarize();
await experiment.flush();

• project (string): project name. Required unless projectId is set.
• projectId (string): project ID. Takes precedence over project.
• experiment (string): experiment name. Generated automatically if omitted.
• description (string): experiment description.
• dataset (string | Dataset): dataset to associate with the experiment.
• metadata (Record<string, unknown>): experiment metadata.
• tags (string[]): experiment tags.
• baseExperiment (string): base experiment name for comparison.
• baseExperimentId (string): base experiment ID. Takes precedence over baseExperiment.
• update (boolean): continue logging to an existing experiment with the same name.
• open (boolean): open an existing experiment in read-only mode.
• isPublic (boolean): whether links are public. Defaults to private.
• setCurrent (boolean): defaults to true. Controls whether global log() uses this experiment.

​

Datasets

​

initDataset()

import { initDataset } from "braintrust";

const dataset = initDataset({
  project: "Support bot",
  dataset: "Password reset questions",
});

dataset.insert({
  input: "How do I reset my password?",
  expected: "Use the password reset flow.",
  metadata: { topic: "account" },
});

await dataset.flush();

• project (string): project name. Required unless projectId is set.
• projectId (string): project ID. Takes precedence over project.
• dataset (string): dataset name. Generated automatically if omitted.
• description (string): dataset description.
• metadata (Record<string, unknown>): dataset metadata.
• version (string): when fetching or iterating rows, read the dataset at this transaction ID. Takes precedence over snapshotName and environment.
• snapshotName (string): when fetching or iterating rows, read the dataset version captured by this named snapshot. Takes precedence over environment.
• environment (string): when fetching or iterating rows, read the dataset version assigned to this environment slug.
• useOutput (boolean): deprecated legacy mode that maps expected to output when fetching rows.

​

Prompts and functions

​

loadPrompt()

import { loadPrompt } from "braintrust";

const prompt = await loadPrompt({
  projectName: "Support bot",
  slug: "answer-question",
  environment: "production",
  defaults: {
    model: "gpt-5-mini",
  },
});

const built = prompt.build({
  question: "How do I reset my password?",
  tone: "concise",
}, {
  strict: true,
});

• projectName (string): project containing the prompt. Required unless projectId or id is set.
• projectId (string): project ID. Takes precedence over projectName.
• slug (string): prompt slug. Required unless id is set.
• id (string): prompt ID. Takes precedence over project and slug.
• version (string): prompt version. Takes precedence over environment.
• environment (string): environment assignment such as production or staging.
• defaults (Record<string, unknown>): default model, parameter, message, or completion fields used by build(). Saved prompt fields override these defaults.
• noTrace (boolean): if true, built prompt metadata is not attached to traces.

• args (object, required): the template variables. Object keys become variables, and the whole value is available as input. Rendered with Mustache by default, with non-string values JSON-stringified.
• options (optional):
  • flavor ("chat" | "completion"): defaults to "chat". Use "completion" for completion prompts.
  • messages (Message[]): additional chat messages to append. If the prompt already has a system message, extra system messages are omitted.
  • strict (boolean): validates that referenced template variables are present.
  • templateFormat ("mustache" | "nunjucks" | "none"): overrides the prompt’s template format. Nunjucks requires @braintrust/template-nunjucks. none leaves template strings unrendered.

​

getPromptVersions()

import { getPromptVersions, loadPrompt } from "braintrust";

const versions = await getPromptVersions(projectId, promptId);
const prompt = await loadPrompt({
  id: promptId,
  version: versions[0],
});

• projectId (string, required): the project’s ID.
• promptId (string, required): the prompt’s ID.

​

loadParameters()

import { Eval, loadParameters } from "braintrust";

const parameters = await loadParameters({
  projectName: "Support bot",
  slug: "retrieval-config",
  environment: "production",
});

await Eval("Support bot", {
  data: [{ input: "ping", expected: "pong" }],
  parameters,
  task: async (input, hooks) => {
    return await answerQuestion(input, hooks.parameters);
  },
  scores: [({ output, expected }) => (output === expected ? 1 : 0)],
});

​

invoke()

import { invoke } from "braintrust";
import { z } from "zod";

const result = await invoke({
  projectName: "Support bot",
  slug: "answer-question",
  input: { question: "How do I reset my password?" },
  schema: z.object({ answer: z.string() }),
});

• function_id (string): function ID.
• projectName (string): project containing the function.
• projectId (string): ID of the project containing the function. Can be used instead of projectName.
• slug (string): function slug.
• globalFunction (string): global function name. See Global Functions.
• functionType (FunctionType): global function type. Defaults to scorer for global functions.
• version (string): function version.

• input (Input, required): logged as the span input.
• messages (Message[]): additional OpenAI-style messages for LLM functions.
• metadata (Record<string, unknown>): logged as span metadata and available to the function.
• tags (string[]): logged as span tags.
• parent (Exportable | string): parent span, logger, experiment, or exported parent string.
• stream (boolean): returns a BraintrustStream when true.
• mode (StreamingMode): function streaming mode.
• strict (boolean): throws when prompt variable names do not match input keys.
• schema (z.ZodSchema): validates non-streaming output and returns the typed value.

​

initFunction()

import { Eval, initFunction } from "braintrust";

const answerQuestion = initFunction({
  projectName: "Support bot",
  slug: "answer-question",
});

await Eval("Support bot", {
  data: [{ input: { question: "ping" }, expected: "pong" }],
  task: answerQuestion,
  scores: [qualityScorer],
});

• projectName (string, required): project containing the function.
• slug (string, required): function slug.
• version (string): function version.
• state (BraintrustState): SDK state override.

​

Attachments

​

Attachment

import { Attachment, initLogger } from "braintrust";

const logger = initLogger({ projectName: "Support bot" });

logger.log({
  input: {
    document: new Attachment({
      data: "./fixtures/account-reset.pdf",
      filename: "account-reset.pdf",
      contentType: "application/pdf",
    }),
  },
});

​

ExternalAttachment

import { ExternalAttachment, initLogger } from "braintrust";

const logger = initLogger({ projectName: "Support bot" });

logger.log({
  input: {
    document: new ExternalAttachment({
      url: "s3://support-docs/account-reset.pdf",
      filename: "account-reset.pdf",
      contentType: "application/pdf",
    }),
  },
});

​

JSONAttachment

import { JSONAttachment, initLogger } from "braintrust";

const logger = initLogger({ projectName: "Support bot" });

logger.log({
  input: {
    transcript: new JSONAttachment(conversation, {
      filename: "conversation.json",
      pretty: true,
    }),
  },
});

​

ReadonlyAttachment

import { ReadonlyAttachment, type AttachmentReference } from "braintrust";

async function readAttachment(reference: AttachmentReference) {
  const attachment = new ReadonlyAttachment(reference);
  const blob = await attachment.data();
  const status = await attachment.status();

  return { blob, status };
}

• data() → Promise<Blob>: downloads the attachment as a Blob.
• asBase64Url() → Promise<string>: returns a data:<content-type>;base64,... URL for prompts.
• metadata() → Promise<AttachmentMetadata>: fetches download URL and upload status.
• status() → Promise<AttachmentStatus>: fetches current upload status.

​

Wrappers

import { initLogger } from "braintrust";

initLogger({ projectName: "Support bot" });

​

OpenAI

import OpenAI from "openai";
import { wrapOpenAI } from "braintrust";

const client = wrapOpenAI(new OpenAI());

await client.responses.create({
  model: "gpt-5-mini",
  input: "Answer in one sentence.",
});

• wrapOpenAI(openai): wraps OpenAI SDK clients and supports current v4, v5, and v6 style clients.
• wrapOpenAIv4(openai): lower-level OpenAI wrapper used by wrapOpenAI(). Use wrapOpenAI() unless you know you need this compatibility entry point.

​

Anthropic

import Anthropic from "@anthropic-ai/sdk";
import { wrapAnthropic } from "braintrust";

const anthropic = wrapAnthropic(new Anthropic());

​

Vercel AI SDK

import * as ai from "ai";
import { openai } from "@ai-sdk/openai";
import { wrapAISDK } from "braintrust";

const { generateText, streamText, Agent } = wrapAISDK(ai);

await generateText({
  model: openai("gpt-5-mini"),
  prompt: "Summarize the ticket.",
});

• wrapAISDK(ai, options?): wraps AI SDK namespace functions such as generateText, streamText, generateObject, streamObject, embed, embedMany, rerank, and agent constructors.
• wrapAgentClass(AgentClass, options?): wraps an AI SDK agent class directly.

• startAISDKBatchTrace({ batch, requests, startTime? }) → Promise<string | undefined>: records the requests of an AI SDK text batch after you submit it with experimental_startBatch(), and returns a context string to store with the batch. Returns undefined if the batch or requests aren’t supported or AI SDK instrumentation is disabled. startTime is a Unix timestamp in seconds and defaults to the batch’s createdAt, or the current time. Doesn’t call the provider.
• completeAISDKBatchTrace({ batch, requests, status, results, context?, endTime?, denyOutputPaths? }) → Promise<void>: records collected batch results and ends the batch spans. Pass the context from startAISDKBatchTrace(), including from another process. endTime is a Unix timestamp in seconds.

​

Agent SDKs

• wrapClaudeAgentSDK(sdk): wraps the Claude Agent SDK module, including query, tool, and createSdkMcpServer integration points.
• wrapOpenRouterAgent(agent): wraps @openrouter/agent clients.
• wrapGoogleADK(adkModule): wraps Google ADK modules, including runner, agent, and tool execution methods.
• wrapLiveKit(module): wraps the @livekit/agents module (v1.9.x). Sessions created with the returned module’s voice.AgentSession are traced. See LiveKit Agents.
• wrapMCPServer(server): traces the requests and notifications that an McpServer or low-level Server from @modelcontextprotocol/sdk or @modelcontextprotocol/server handles, and returns the same instance. You can wrap a server before or after you connect it to a transport. Requires v3.42.0 or later. See MCP SDK.

​

Model provider SDKs

• wrapGoogleGenAI(): wraps the entire @google/genai package export (similar to the AI SDK wrapper). Traces models.generateContent, models.generateContentStream, and models.embedContent.
• wrapOpenRouter(): pass an OpenRouter client. Traces chat, responses, embeddings, rerank, and callModel() where available.
• wrapMistral(): pass a Mistral client. Traces chat, FIM (fill-in-the-middle), agents, and embeddings.
• wrapCohere(): pass a Cohere client. Traces chat, chat streaming, embed, and rerank.
• wrapGroq(): pass a Groq client. Traces chat completions and embeddings.
• wrapHuggingFace(): pass a Hugging Face Inference module or client. Traces chat completion, text generation, streaming variants, and feature extraction.
• wrapWorkersAI(): pass a Cloudflare Workers AI binding, such as env.AI. Traces run() calls to the Clef decision models and the Clairic extraction model. Other models, and streaming or raw-response calls, pass through unchanged. See Cloudflare Workers AI.

​

Testing

​

wrapVitest()

import * as vitest from "vitest";
import { wrapVitest } from "braintrust";

const bt = wrapVitest(vitest, {
  projectName: "Support bot",
});

bt.describe("retrieval", () => {
  bt.test(
    "answers password reset",
    {
      input: "How do I reset my password?",
      expected: "Use the password reset flow.",
      scorers: [qualityScorer],
    },
    async ({ input }) => {
      const output = await answerQuestion(input);
      bt.logOutputs(output);
      bt.expect(output).toBeTruthy();
      return output;
    },
  );
});

​

initNodeTestSuite()

import { after, describe, test } from "node:test";
import assert from "node:assert/strict";
import { initNodeTestSuite } from "braintrust";

describe("retrieval", () => {
  const suite = initNodeTestSuite({
    projectName: "Support bot",
    after,
  });

  test(
    "answers password reset",
    suite.eval(
      {
        input: "How do I reset my password?",
        expected: "Use the password reset flow.",
        scorers: [qualityScorer],
      },
      async ({ input }) => {
        const output = await answerQuestion(input);
        assert.ok(output);
        return output;
      },
    ),
  );
});

​

Configuration

​

login()

import { login, loadPrompt } from "braintrust";

await login({
  apiKey: "bt_org_...", // Replace with your API key.
  orgName: "example-org",
});

const prompt = await loadPrompt({
  projectName: "Support bot",
  slug: "answer-question",
});

• apiKey (string): API key. Defaults to BRAINTRUST_API_KEY.
• appUrl (string): Braintrust app URL. Defaults to https://www.braintrust.dev.
• orgName (string): organization name, useful when credentials can access multiple orgs.
• fetch (typeof fetch): custom fetch implementation for SDK requests.
• forceLogin (boolean): log in again even if the SDK already has credentials.
• noExitFlush (boolean): disables the process exit handler that flushes pending writes.
• onFlushError ((error) => void): handles errors from the background flusher.
• disableSpanCache (boolean): disables the local span cache used by eval scorers.
• debugLogLevel ("error" | "warn" | "info" | "debug" | false): enables SDK troubleshooting output. Set false to also silence the SDK’s default warnings.

​

Environment variables

• BRAINTRUST_API_KEY: API key used to authenticate. Same as the apiKey option.
• BRAINTRUST_API_URL: Braintrust API URL. Set this for self-hosted or data plane deployments.
• BRAINTRUST_APP_URL: Braintrust app URL, used for permalinks. Defaults to https://www.braintrust.dev. Same as the appUrl option.
• BRAINTRUST_APP_PUBLIC_URL: public Braintrust app URL used when generating permalinks. Defaults to the app URL.
• BRAINTRUST_PROXY_URL: AI proxy URL. Overrides the proxy URL returned for your organization.
• BRAINTRUST_ORG_NAME: organization name, useful when credentials can access multiple orgs. Same as the orgName option.
• BRAINTRUST_DISABLE_INSTRUMENTATION: comma-separated list of integrations to skip during auto-instrumentation, for example openai,anthropic.
• BRAINTRUST_CAPTURE_ATTACHMENTS: set to 1 or true to upload files from OpenAI image and audio calls, base64 image and file inputs to OpenAI Chat Completions calls, and images and resource blobs in MCP server results, as attachments. Off by default, which records the file contents as <omitted>. In v3.41.0 and later, it doesn’t record LiveKit Agents call audio. Use the two variables that follow instead.
• BRAINTRUST_CAPTURE_USER_AUDIO_ATTACHMENTS: set to 1 or true to record the caller’s audio in LiveKit Agents sessions as attachments. Off by default. Requires v3.41.0 or later.
• BRAINTRUST_CAPTURE_AGENT_AUDIO_ATTACHMENTS: set to 1 or true to record the agent’s audio in LiveKit Agents sessions as attachments. Off by default. Requires v3.41.0 or later.
• BRAINTRUST_DEBUG_LOG_LEVEL: SDK troubleshooting output: error, warn, info, or debug. When unset, the SDK prints only warnings that need your attention, such as a client it can’t wrap. Same as the debugLogLevel option.
• BRAINTRUST_VERBOSE: set to a truthy value to enable verbose evaluation logging.
• BRAINTRUST_ENVIRONMENT_TYPE: environment type for span-origin provenance (for example, ci or server). Detected automatically from common CI and server runtimes when unset.
• BRAINTRUST_ENVIRONMENT_NAME: environment name for span-origin provenance (for example, github_actions or aws_lambda). Detected automatically when unset.

• BRAINTRUST_SYNC_FLUSH: set to a truthy value to flush logs synchronously instead of in the background.
• BRAINTRUST_FLUSH_DELAY_MS: maximum time, in milliseconds, that a logged row waits so nearby rows can be sent in one upload request. A full batch of BRAINTRUST_DEFAULT_BATCH_SIZE rows is sent right away. Defaults to 1000 in long-lived Node.js processes, and to 0 (send right away) in other runtimes and in serverless environments such as AWS Lambda, Vercel, Netlify, Google Cloud Run and Cloud Functions, App Engine standard, and Azure Functions. Requires v3.40.0 or later.
• BRAINTRUST_DEFAULT_BATCH_SIZE: number of log events sent per upload request.
• BRAINTRUST_MAX_REQUEST_SIZE: override the maximum upload request size, in bytes.
• BRAINTRUST_NUM_RETRIES: number of times to retry a failed upload.
• BRAINTRUST_MAX_GENERATOR_ITEMS: maximum number of items pulled from a generator when tracing. Defaults to 1000; 0 disables the limit.

• BRAINTRUST_CACHE_LOCATION: where prompt and parameters caches are stored: mixed, memory, disk, or none. Defaults to mixed.
• BRAINTRUST_PROMPT_CACHE_DIR: directory for the on-disk prompt cache. Defaults to ~/.braintrust/prompt_cache.
• BRAINTRUST_PROMPT_CACHE_MEMORY_MAX: maximum number of entries in the in-memory prompt cache.
• BRAINTRUST_PROMPT_CACHE_DISK_MAX: maximum number of entries in the on-disk prompt cache.
• BRAINTRUST_PARAMETERS_CACHE_DIR: directory for the on-disk parameters cache. Defaults to ~/.braintrust/parameters_cache.
• BRAINTRUST_PARAMETERS_CACHE_MEMORY_MAX: maximum number of entries in the in-memory parameters cache.
• BRAINTRUST_PARAMETERS_CACHE_DISK_MAX: maximum number of entries in the on-disk parameters cache.