> ## Documentation Index
> Fetch the complete documentation index at: https://docs.braintrust.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# TypeSafe

> Trace TypeSafe Jev calls, inspect structured decisions, and use Jev as an LLM judge to score or label AI outputs against your criteria.

[TypeSafe](https://typesafe.ai/) provides Jev, a model that makes structured decisions instead of generating text. You define the question and its allowed answers before the call, and Jev returns one of them, so nothing downstream has to find a decision inside prose or malformed JSON. You can use Braintrust with Jev in two ways:

* **[Trace Jev in your application](#tracing)**: If your application calls Jev to route support requests, assess urgency, or make other decisions, instrument those calls to inspect their inputs, answers, and probabilities in Braintrust traces.
* **[Use Jev for evaluations](#llm-as-a-judge)**: Configure Jev as the LLM judge to check AI outputs against your criteria, such as whether a support response follows a refund policy. Braintrust runs the evaluator and turns Jev's decisions into scores or labels in experiments or online scoring.

## Tracing

Instrument TypeSafe calls in your application to capture their inputs, results, timing, and errors in Braintrust decision spans.

**TypeScript**

### Setup

Requires Braintrust v3.38.0+ and `@typesafe-ai/sdk` v0.6.0 or later within v0.x. You need a TypeSafe account and API key.

**1. Install packages**

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
pnpm add braintrust@^3.38.0 @typesafe-ai/sdk@^0.6.0
```

**2. Set environment variables**

Set your [Braintrust API key](https://docs.braintrust.dev/admin/authentication.md#api-authentication) and TypeSafe API key in your shell:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
export BRAINTRUST_API_KEY="<your-braintrust-api-key>"
export TYPESAFE_API_KEY="<your-typesafe-api-key>"
```

For the EU data plane, also set `BRAINTRUST_API_URL` to `https://api-eu.braintrust.dev`. For a self-hosted deployment, use your data plane URL.

### Auto-instrumentation

Use Braintrust's import hook to trace TypeSafe calls throughout your application.

**1. Initialize Braintrust and call Jev**

Save this example as `trace-typesafe-auto.js`. It asks Jev to route a support request, assess its urgency, and identify whether it mentions a duplicate charge.

```javascript title="trace-typesafe-auto.js" theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import { initLogger } from "braintrust";
import { TypeSafeClient, choice, noul, score } from "@typesafe-ai/sdk";

const logger = initLogger({ projectName: "typesafe-example" });
const client = new TypeSafeClient();

const response = await client.systemOne({
  state: "I was charged twice. Please refund the duplicate charge today.",
  questions: {
    category: choice("Which team should handle this request?", {
      billing: "Payments and refunds",
      technical: "Software problems",
      other: "Other requests",
    }),
    urgency: score("How urgent is this request?", [
      "routine",
      "soon",
      "urgent",
    ]),
    duplicate_charge: noul("Does the customer report a duplicate charge?"),
  },
});

console.log(response.answers);
await logger.flush();
```

**2. Run with the import hook**

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
node --import braintrust/hook.mjs trace-typesafe-auto.js
```

The auto-instrumentation example uses plain JavaScript so `node --import` can run the file directly. The Braintrust APIs work the same in TypeScript projects. Compile your TypeScript to JavaScript and run the compiled file with the import hook, or [run TypeScript directly](https://docs.braintrust.dev/instrument/trace-llm-calls.md#auto-instrumentation) with a TypeScript loader.

> [!NOTE]
>
> If you're using a bundler, see [Trace LLM calls](https://docs.braintrust.dev/instrument/trace-llm-calls.md#auto-instrumentation) for plugin and loader setup.

Go to your project's [\*\* Logs\*\*](https://www.braintrust.dev/app/~/logs) and select the `typesafe.systemOne` span to [inspect the decisions](#inspect-jev-decisions).

### Manual instrumentation

Wrap individual clients with `wrapTypeSafe()` to choose which TypeSafe calls to trace.

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import { initLogger, wrapTypeSafe } from "braintrust";
import { TypeSafeClient, choice } from "@typesafe-ai/sdk";

const logger = initLogger({ projectName: "typesafe-example" });
const client = wrapTypeSafe(new TypeSafeClient());

const response = await client.systemOne({
  state: "I was charged twice. Please refund the duplicate charge today.",
  questions: {
    category: choice("Which team should handle this request?", {
      billing: "Payments and refunds",
      technical: "Software problems",
      other: "Other requests",
    }),
  },
});

console.log(response.answers.category.choice);
await logger.flush();
```

**AI SDK**: If your application calls Jev through AI SDK's `experimental_decide()` or `experimental_evaluate()`, wrap the `ai` module with `wrapAISDK()` instead. Wrapping `experimental_evaluate()` requires AI SDK v7.0.103 or later within v7. AI SDK v7.0.128 and later add `experimental_decide()`, which replaces the deprecated `experimental_evaluate()`. In your existing AI SDK application, replace the direct import with the wrapped export for your AI SDK version:

```typescript AI SDK v7.0.128+ theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import * as ai from "ai";
import { initLogger, wrapAISDK } from "braintrust";

initLogger({ projectName: "typesafe-example" });
const { experimental_decide } = wrapAISDK(ai);
```

```typescript AI SDK v7.0.103 to v7.0.127 theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import * as ai from "ai";
import { initLogger, wrapAISDK } from "braintrust";

initLogger({ projectName: "typesafe-example" });
const { experimental_evaluate } = wrapAISDK(ai);
```

Call the wrapped function with your existing decision model, state, and questions. Braintrust records an `evaluate` span with type `question` for either function, which the trace view labels as a **Decision** span. The span follows the call's `telemetry` options: `isEnabled: false` skips the span, and `recordInputs: false` or `recordOutputs: false` leaves out the input or output. The import hook also instruments these calls. Registering [`braintrustAISDKTelemetry()`](https://docs.braintrust.dev/integrations/sdk-integrations/vercel#manual-instrumentation-telemetry-registration-typescript) also records decision spans from the telemetry callbacks that AI SDK v7.0.111 and later emit. Inline images and files in the decision input are captured as attachments when `BRAINTRUST_CAPTURE_ATTACHMENTS` is set to `1` or `true`, and recorded as `<omitted>` otherwise.

### What Braintrust traces

For each `TypeSafeClient.systemOne()` call, Braintrust records a `typesafe.systemOne` span with type `decision`, which the trace view labels as a **Decision** span:

* Input state and questions, including question identifiers, instructions, and criteria
* Images in the state, instructions, and criteria (`@typesafe-ai/sdk` v0.7.0+), kept in place and captured as attachments when `BRAINTRUST_CAPTURE_ATTACHMENTS` is set to `1` or `true`. Otherwise, each image is recorded as a placeholder without its contents.
* Structured answers, including choices, scores, Noul values, and returned confidence and probabilities. When the request contains exactly one image, a box answer also records that image's location in the input as `image_path`.
* Model and provider metadata
* Token usage reported by TypeSafe and request duration
* Errors raised by the call

**Python**

### Setup

Requires Braintrust v0.41.0+ and `typesafe-sdk` v0.6.0+. You need a TypeSafe account and API key.

**1. Install packages**

```bash uv theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
uv add "braintrust>=0.41.0" "typesafe-sdk>=0.6.0"
```

```bash pip theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
pip install "braintrust>=0.41.0" "typesafe-sdk>=0.6.0"
```

**2. Set environment variables**

Set your [Braintrust API key](https://docs.braintrust.dev/admin/authentication.md#api-authentication) and TypeSafe API key in your shell:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
export BRAINTRUST_API_KEY="<your-braintrust-api-key>"
export TYPESAFE_API_KEY="<your-typesafe-api-key>"
```

For the EU data plane, also set `BRAINTRUST_API_URL` to `https://api-eu.braintrust.dev`. For a self-hosted deployment, use your data plane URL.

### Auto-instrumentation

Call `braintrust.auto_instrument()` before creating your TypeSafe clients to trace synchronous and asynchronous calls.

**1. Initialize Braintrust and call Jev**

Save this example as `trace_typesafe.py`. It asks three questions about the same support request.

```python title="trace_typesafe.py" theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import braintrust
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

logger = braintrust.init_logger(project="typesafe-example")
braintrust.auto_instrument()

with TypeSafeClient() as client:
    response = client.system_one(
        state="I was charged twice. Please refund the duplicate charge today.",
        questions={
            "category": Choice(
                instructions="Which team should handle this request?",
                criteria={
                    "billing": "Payments and refunds",
                    "technical": "Software problems",
                    "other": "Other requests",
                },
            ),
            "urgency": Score(
                instructions="How urgent is this request?",
                criteria=["routine", "soon", "urgent"],
            ),
            "duplicate_charge": Noul(
                instructions="Does the customer report a duplicate charge?",
            ),
        },
    )

print(response.answers)
logger.flush()
```

**2. Run your application**

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
python trace_typesafe.py
```

Go to your project's [\*\* Logs\*\*](https://www.braintrust.dev/app/~/logs) and select the `typesafe.systemOne` span to [inspect the decisions](#inspect-jev-decisions).

### Manual instrumentation

Wrap individual clients with `wrap_typesafe()`. This example uses the asynchronous client. The same wrapper supports `TypeSafeClient` for synchronous calls.

```python Python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import asyncio

import braintrust
from braintrust.integrations.typesafe import wrap_typesafe
from typesafe_sdk import AsyncTypeSafeClient, Choice

logger = braintrust.init_logger(project="typesafe-example")


async def main():
    async with wrap_typesafe(AsyncTypeSafeClient()) as client:
        response = await client.system_one(
            state="I was charged twice. Please refund the duplicate charge today.",
            questions={
                "category": Choice(
                    instructions="Which team should handle this request?",
                    criteria={
                        "billing": "Payments and refunds",
                        "technical": "Software problems",
                        "other": "Other requests",
                    },
                ),
            },
        )
    print(response.answers["category"].choice)


asyncio.run(main())
logger.flush()
```

### What Braintrust traces

For each synchronous or asynchronous `system_one()` call, Braintrust records a `typesafe.systemOne` span with type `decision` (`question` before v0.46.0), which the trace view labels as a **Decision** span:

* Input state and questions, including question identifiers, instructions, and criteria
* Images in the state, instructions, and criteria (`typesafe-sdk` v0.7.5+), kept in place and captured as attachments when `BRAINTRUST_CAPTURE_ATTACHMENTS` is set to `1` or `true`. Otherwise, each image is recorded as `<omitted>`. Images passed in `assets` are handled the same way wherever the request references them.
* Structured answers, including choices, scores, Noul values, boxes, and returned confidence and probabilities. When the request contains exactly one image, a box answer also records that image's location in the input as `image_path`.
* Model and provider metadata
* Token usage reported by TypeSafe and request duration
* Errors raised by the call

### Inspect Jev decisions

Select a decision span in a trace and use the **Pretty** view for its input and output. The input shows the state sent to Jev and each question's instructions and criteria. The output matches answers to questions by identifier, taken from the keys in your request, so you can inspect the decision alongside the question that produced it.

![A typesafe.systemOne span showing the question criteria sent to Jev and three rendered answers: a Choice of billing at 100 percent, a Score of 1.74 out of 2 across routine, soon, and urgent levels, and a Noul of 99 percent true.](https://mintcdn.com/braintrust/wlpOx1kzv_665aru/images/evaluate/typesafe-trace.png?fit=max\&auto=format\&n=wlpOx1kzv_665aru\&q=85\&s=169bcd93a656e8553d924c72a05106fc)

What each answer type shows, keeping in mind that model responses can vary:

* **Noul**: Probability of yes, displayed as a percentage on a scale from no to yes.
* **Choice**: The selected option, per-option probabilities, and confidence when returned.
* **Score**: The numeric result on the rubric's scale, level descriptions, per-level probabilities, and confidence when returned.

Per-option probabilities show how Jev distributes probability across the possible answers. [TypeSafe confidence](https://docs.typesafe.ai/confidence), a value between `0` and `1`, summarizes how concentrated that distribution is. It is separate from the selected option's probability and from an evaluation score. High confidence does not guarantee a correct decision.

## LLM-as-a-judge

Use Jev as the model in an [LLM-as-a-judge](https://docs.braintrust.dev/evaluate/llm-as-a-judge.md) evaluator. It reads AI output, plus any context you pass it, and returns one of the choices you define rather than text. Braintrust turns that choice into either a numeric score, such as `pass` → `1` and `fail` → `0`, or a classifying label, such as `billing` or `technical`.

Jev works best for frequent, focused checks with a clear answer, such as whether a support reply follows your refund policy. For broad or nuanced criteria like tone or helpfulness, where the useful output is a judgment rather than a selection, a general-purpose LLM is the better judge. See [Choose a judge model](https://docs.braintrust.dev/evaluate/llm-as-a-judge.md#choose-a-judge-model).

### Configure access

Braintrust runs the evaluator, so you give it access to Jev in your settings rather than in your application. There are two ways:

* **Braintrust's built-in Jev**, which needs no TypeSafe account of your own. Usage draws down your organization's [model credits](https://docs.braintrust.dev/plans-and-limits.md#model-credits) like other built-in models. It is not available through the AI gateway. Go to ** Settings** > [** Models**](https://www.braintrust.dev/app/~/configuration/models/organization), click **Enable Jev**, and authorize Braintrust to engage TypeSafe as a subprocessor.

  > [!NOTE]
  >
  > This requires [built-in models](https://docs.braintrust.dev/admin/ai-providers.md#manage-built-in-models) to be allowed for your organization, and only members of the **Owners** [permission group](https://docs.braintrust.dev/admin/access-control.md), or a custom permission group with the **Manage settings** organization permission, can change the setting. Disabling Jev can take up to a minute to take effect.

  > [!NOTE]
  >
  > Braintrust's built-in Jev is not available for [BYOC](https://docs.braintrust.dev/admin/deployment/byoc.md) or [self-hosted](https://docs.braintrust.dev/admin/self-hosting.md) deployments. On those deployments, use a key from your own TypeSafe account instead.

* **A key from your own TypeSafe account**, if you already have one. [Add TypeSafe as an AI provider](https://docs.braintrust.dev/admin/ai-providers.md#organization-providers) at organization or project scope. This requires data plane v2.15.0 or later.

### Create an evaluator

**1. Create the evaluator**

Go to [\*\* Scorers\*\*](https://www.braintrust.dev/app/~/scorers) and click  **Scorer**. Enter a name and slug, select **LLM judge**, and choose **Jev** under the provider you configured above. Jev appears in the model picker for scorers and classifiers, not in the picker for generating outputs in prompts or playgrounds.

Selecting Jev hides the model parameter and tool controls, because it takes no sampling parameters and selects from your choices without a tool schema.

**2. Write the prompt**

Write a prompt that includes the context needed to make the decision and explains what each choice means. Jev judges text, so media and tool message content are rejected. To judge a trace that contains tool calls, pass the relevant context as text.

For example:

```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
Evaluate whether the support response follows this refund policy:
Unused items can be refunded within 30 days of purchase.
Used items and purchases older than 30 days are not eligible.

Customer request: {{input}}
Support response: {{output}}

Choose pass if the response applies the policy correctly to the request.
Choose fail if it contradicts the policy or promises an ineligible refund.
```

The prompt defines the criteria. The output configuration in the next step defines what Jev's decision becomes.

**3. Configure the output**

Both output types require at least one non-empty choice. Jev returns its decision directly rather than reasoning first, so the **Use chain of thought (CoT)** setting doesn't appear.

> [tab] Scorer
>
> Set **Output type** to **Score**. Under **Choice scores**, add each choice and the number it maps to, such as `pass` with score `1` and `fail` with score `0`. Choices and scores must be unique.
>
> If your criteria do not apply to every input, enable **Allow skip** and explain when to skip in the prompt. Jev gains a reserved `Skip` choice, and selecting it returns a `null` score so the case is excluded from score aggregates rather than counted as `0`.

> [tab] Classifier
>
> Set **Output type** to **Classification**. Under **Classifications**, add each label Jev can choose. Labels must be unique, and Jev selects exactly one. The selected label becomes both the `id` and the `label` of the resulting classification.
>
> Reach for a classifier when the interesting answer is which failure occurred, not how good the output was. Grading a support reply for whether it can be sent, for example, distinguishes cases a single number would flatten together:
>
> * `Incorrect, unsafe, or invents information`
> * `Accurate, but does not answer the customer's request`
> * `Answers the request, but misses a necessary detail or next step`
> * `Accurate, answers the request, and clearly explains the next step`
>
> To let Jev return no label when none of yours fit, enable **Allow "No match"**. The result is then `no_match`.

**4. Test known cases**

In the  **Run** section, enter test values for `input` and `output` in the editor, then click **Test**.

For the scorer above, set `input` to `I bought this item 10 days ago and have not used it. Can I get a refund?` and `output` to `Yes, your unused item is eligible for a refund within 30 days.` The intended result is `pass`, with score `1`.

Repeat with `input` set to `I bought this item 45 days ago and have used it. Can I get a refund?` and the same output. The intended result is `fail`, with score `0`. Compare Jev's actual decisions with these judgments and refine the prompt if they disagree.

**5. Save the evaluator**

Click **Save as custom scorer**. Select the saved evaluator when you [run an experiment](https://docs.braintrust.dev/evaluate/run-evaluations.md) or [score production traces](https://docs.braintrust.dev/evaluate/score-online.md) to apply the same criteria to new outputs.

### Interpret the result

Jev returns one of the choices you define, which Braintrust maps to a number for a scorer or keeps as a label for a classifier. The mapping ignores confidence, so a `pass` choice scored `1` returns `1` even when Jev is unsure. Both scorers and classifiers carry the same metadata:

| Field | Meaning |
| - | - |
| `metadata.choice` | The selected choice. |
| `metadata.typesafe.model` | The resolved Jev model. |
| `metadata.typesafe.confidence` | How concentrated the probability distribution is. |
| `metadata.typesafe.probabilities` | The probability assigned to each choice. |

Use confidence and probabilities to investigate uncertain judgments. Compare confidence with correctness on human-reviewed examples before choosing a threshold for further review. Routing uncertain results to a person or another judge requires your own logic.

An evaluator's model call appears as a `TypeSafe Jev` decision span. Open its **Pretty** view to inspect the rendered evaluator state, choice criteria, selected answer, confidence, and probabilities together.

## Resources

**Tracing**

* [TypeSafe quickstart](https://docs.typesafe.ai/introduction/quickstart)
* [TypeSafe client SDKs](https://docs.typesafe.ai/sdk)
* [Customize traces](https://docs.braintrust.dev/instrument/trace-application-logic.md) beyond what the integration records

**LLM-as-a-judge**

* [Run experiments](https://docs.braintrust.dev/evaluate/run-evaluations.md) with your saved evaluator
* [Configure online scoring](https://docs.braintrust.dev/evaluate/score-online.md) to evaluate production traces
* [Develop LLM-as-a-judge evaluators](https://docs.braintrust.dev/evaluate/llm-as-a-judge.md) for span, trace, or group scoring


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.