> ## Documentation Index
> Fetch the complete documentation index at: https://docs.braintrust.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Dataset pipelines

> Transform spans or traces from your project logs into dataset rows in bulk with declarative, version-controlled pipelines.

A dataset pipeline transforms spans or traces from your project logs into dataset rows in bulk. It automates the same flow as [promoting individual traces from logs](https://docs.braintrust.dev/annotate/datasets/create.md#promote-traces-from-logs), but applies it to many spans or traces at once and keeps the logic in version-controlled code.

You define a pipeline with `DatasetPipeline(...)` in a TypeScript or Python file, then run it with the [`bt datasets pipeline`](https://docs.braintrust.dev/reference/cli/datasets#bt-datasets-pipeline) CLI command.

> [!NOTE]
>
> Dataset pipelines require `bt` CLI v0.10.0 or later, plus the `braintrust` SDK for the language you write the pipeline in: TypeScript SDK v3.16.0 or later, or Python SDK v0.23.0 or later.

## Define a pipeline

A pipeline definition has three parts:

* **`source`**: Which project to read from, an optional [SQL](https://docs.braintrust.dev/reference/sql.md) filter, and whether to operate on individual spans or entire traces (`scope`).
* **`transform`**: A function that receives a source span or trace and returns one or more dataset rows, or nothing to skip it.
* **`target`**: The dataset to write to. Braintrust creates the dataset if it does not exist.

```typescript pipeline.ts theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import { DatasetPipeline } from "braintrust";

DatasetPipeline({
  name: "support-cases", // Optional; required only when a file defines multiple pipelines
  source: {
    projectName: "Customer Support", // Replace with your source project
    filter: "metadata.flagged = true", // Optional SQL filter; omit to consider all spans
    scope: "span", // "span" (default) or "trace"
  },
  transform: ({ id, input, output, expected, metadata, trace }) => {
    return {
      input,
      expected: output, // Map the production output to the expected value
      metadata,
    };
  },
  target: {
    projectName: "Customer Support", // Replace with your target project
    datasetName: "Support cases", // Created if it does not exist
  },
});
```

```python pipeline.py theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
from braintrust import DatasetPipeline

def to_row(id=None, input=None, output=None, metadata=None, expected=None, trace=None):
    return {
        "input": input,
        "expected": output,  # Map the production output to the expected value
        "metadata": metadata,
    }

DatasetPipeline(
    name="support-cases",  # Optional; required only when a file defines multiple pipelines
    source={
        "project_name": "Customer Support",  # Replace with your source project
        "filter": "metadata.flagged = true",  # Optional; omit to consider all spans
        "scope": "span",  # "span" (default) or "trace"
    },
    transform=to_row,
    target={
        "project_name": "Customer Support",  # Replace with your target project
        "dataset_name": "Support cases",  # Created if it does not exist
    },
)
```

The `transform` function returns a single row, a list of rows, or `null`/`None` to skip the source. A row accepts the standard dataset fields: `input`, `expected`, `metadata`, `tags`, and `id`. When you omit `id`, the row inherits the source span or trace ID.

When `scope` is `"span"`, the transform receives the span's `id`, `input`, `output`, `expected`, and `metadata`, along with the full `trace`. When `scope` is `"trace"`, it receives only the `trace`.

## Run a pipeline

Run the full pipeline in one shot. The `--limit` flag controls how many source spans or traces to discover, and `--window` sets how far back to look, which defaults to the last day (`1d`). For the full list of flags, see the [`bt datasets pipeline`](https://docs.braintrust.dev/reference/cli/datasets#bt-datasets-pipeline) CLI reference.

```bash TypeScript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
bt datasets pipeline run ./pipeline.ts --limit 100 --window 30d
```

```bash Python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
bt datasets pipeline run ./pipeline.py --limit 100 --window 30d
```

`run` discovers source refs, transforms them, and inserts the resulting rows into the target dataset.

## Staged workflow

For larger jobs, or when you want to inspect or edit rows before writing them, split the run into three stages:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
# 1. Discover source refs and write them to pulled.jsonl
bt datasets pipeline pull ./pipeline.ts --limit 500

# 2. Apply the transform and write the results to transformed.jsonl
bt datasets pipeline transform ./pipeline.ts

# 3. Push the transformed rows to the target dataset
bt datasets pipeline push ./pipeline.ts
```

Each stage writes its artifacts to `bt-sync/` by default: `pull` writes `pulled.jsonl` and `transform` writes `transformed.jsonl`. You can inspect or edit `transformed.jsonl` before running `push`. Use `--root` to change the directory, or `--out` and `--in` to override individual artifact paths.

For the full set of flags, including source and target overrides and concurrency controls, see the [`bt datasets pipeline`](https://docs.braintrust.dev/reference/cli/datasets#bt-datasets-pipeline) CLI reference.

## Next steps

* Browse the [`bt datasets pipeline`](https://docs.braintrust.dev/reference/cli/datasets#bt-datasets-pipeline) command reference for every flag.
* Learn other ways to [create datasets](https://docs.braintrust.dev/annotate/datasets/create.md) from uploads, the SDK, or production logs.
* [Use datasets in evaluations](https://docs.braintrust.dev/annotate/datasets/use-in-evaluations.md) once your rows are in place.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.