> ## Documentation Index
> Fetch the complete documentation index at: https://docs.braintrust.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# What Loop can do

> Query logs, diagnose traces, write scorers, run experiments, and build dashboards through natural language, with example prompts for every task.

Loop can find a failure in your logs, write a scorer for it, and open that scorer for review in a single turn. It combines tools for playgrounds, custom views, dataset snapshots, and the workspace with the [Braintrust MCP](https://docs.braintrust.dev/integrations/developer-tools/mcp.md) server's read and write tools.

Explore its capabilities and example prompts by task: investigate data, build objects, run evaluations and automations, or get help.

## Investigate

Ask questions about your data, from a single trace up to every project in your organization.

###  Explore your data

Loop queries logs, experiments, and datasets with SQL and inspects their schemas. For large or complex traces, it can read, search, and list sandbox files or run commands. Open any trace it resolves or links to in the workspace beside the thread.

Loop can also:

* **Hand off to the Query page.** Loop can write and run a query on the [** Query**](https://www.braintrust.dev/app/~/sql) page, then [refine or fix it](https://docs.braintrust.dev/reference/sql.md#loop) on request. The query stays there for you to edit.
* **Start from the logs table.** Loop can [turn a question into a SQL filter](https://docs.braintrust.dev/observe/filter.md#ask-questions-with-loop) and [find traces similar](https://docs.braintrust.dev/observe/filter.md#ask-questions-with-loop) to your selection.
* **Resolve identities.** Loop resolves `span_attributes.created_by_user_id` to organization member names and emails, and `span_attributes.created_by_api_key_id` to API key and service token names, so you can ask about people and keys by name instead of by ID.

> [!NOTE]
>
> Fields you define in `metadata`, including ones named like `metadata.user_id`, describe your own application's users. Loop doesn't read them as Braintrust members.

Example prompts:

> [accordion] Summarize logs and suggest next steps
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Look through my logs, explain what you find, and recommend the best next steps for my project.
> ```

> [accordion] Break down cost by model
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Show me the cost of traces named X over the last week, broken down by model. Include total, average, p50, p95, and p99 cost.
> ```

> [accordion] Graph caching and usage metrics
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Find the metrics related to prompt caching and LLM usage, then graph cache-hit rate and the number of LLM calls per trace.
> ```

> [accordion] Hand off a query to the Query page
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Write that query into the Query page and run it there so I can keep refining it.
> ```

> [accordion] Attribute traces to teammates
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Which teammates logged the most traces this week, and what did each one cost?
> ```

###  Analyze traces

Loop can diagnose trace failures and summarize work across LLM and tool calls, or help [troubleshoot online scoring failures](https://docs.braintrust.dev/evaluate/score-online.md#scoring-failures).

To ask about a trace in a Loop thread, paste its permalink or ID. Loop resolves it and loads its spans.

Example prompts:

> [accordion] Understand what happened
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Summarize this trace. What was the total latency, and where was the time spent?
> ```

> [accordion] Find related failures
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Are there other traces with similar errors? Show me what they have in common.
> ```

###  Find patterns

Loop searches logs and raw traces for recurring problems and trends, even without a scorer measuring them, and records them as [patterns](https://docs.braintrust.dev/observe/patterns.md).

Each pattern includes supporting evidence. Loop updates an existing pattern when it finds related evidence, rather than creating a duplicate.

Example prompts:

> [accordion] Record a failure you suspect
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> I think our tool calls are failing more often since last week. Dig into it, and if you find something, record it as a pattern.
> ```

> [accordion] Turn an investigation into a tracked pattern
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Look through this project's recent logs, find a recurring failure, and record it as a pattern with the supporting traces.
> ```

> [accordion] Ask how a pattern was found
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> How did you derive this pattern, and what would make it more accurate?
> ```

> [!NOTE]
>
> Loop can also find patterns on a schedule, without being asked. See [Enable Patterns](https://docs.braintrust.dev/observe/patterns/enable.md).

###  Use external tools and context

Connect [external MCP servers](https://docs.braintrust.dev/loop/mcp.md) (public preview) so Loop can combine trace evidence with known issues, internal documentation, and other business context. Depending on the server's tools, Loop can also create follow-up tasks from its findings.

Example prompts:

> [accordion] Investigate failures with Linear context
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Investigate recurring tool-call failures from the past week. Search Linear for related issues, and draft a bug report for any failure that isn't already tracked. Include example traces.
> ```

> [accordion] Check responses against internal guidance
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Use the connected knowledge base to find our refund policy. Compare it with recent traces where the agent answered refund questions, and flag responses that conflict with the policy. Include the policy references and trace links.
> ```

###  Investigate failures in source code

With the [GitHub integration](https://docs.braintrust.dev/loop/github.md) (public preview), Loop can investigate trace failures alongside your source code. With [write access](https://docs.braintrust.dev/loop/github.md#give-loop-write-access), it can implement a fix and regression test, push a branch, and open a pull request for review.

Example prompts:

> [accordion] Trace a failure back to the code
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Compare a failed tool call with a successful one, then find the relevant implementation in the connected repository. Explain the likely cause and suggest a fix and a regression test. Include trace links and file paths.
> ```

> [accordion] Find test coverage gaps
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Review recurring failures in this project's traces from the past week and the tests in the connected repository. Identify failure cases the tests don't cover, and suggest regression tests using representative trace inputs.
> ```

> [accordion] Turn a failure into a pull request
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Find the cause of the most common tool-call failure in this project's traces from the past week. Fix it in the connected repository, add a regression test, push the change to a new branch, and open a pull request that links to example traces.
> ```

###  Analyze projects

Loop can investigate a whole project or compare projects across your organization.

For a single project, open  **Loop agent** at the bottom right of its ** Home** page and enter a prompt. Loop can inspect the project's logs, recent experiments, datasets, prompts, and score progress chart.

Example prompts:

> [accordion] Track a score over time
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> What's the trend in my accuracy score over the last 7 days? Which scorer has been improving the most?
> ```

> [accordion] Find the experiments that moved
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Which experiments had the biggest score changes?
> ```

> [accordion] Check recent activity
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Show me recent logs with errors, and tell me what datasets were created recently.
> ```

> [accordion] Watch token usage
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> How has my project's token usage changed over time?
> ```

To compare projects, select  **Loop agent** from the organization-level [projects list](https://www.braintrust.dev/app). Ask about project metrics, usage, or performance trends.

Example prompts:

> [accordion] Rank projects by usage
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Which projects have the highest token usage, and which have the most experiments?
> ```

> [accordion] Compare performance across projects
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Compare latency metrics across my top 5 projects.
> ```

> [accordion] Spot trends in volume
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Show me trends in log volume across projects, and which projects have declining usage.
> ```

> [accordion] Audit recent projects
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> What projects were created in the last month?
> ```

## Build

Turn findings into saved Braintrust objects. Loop asks for approval before creating or changing them.

###  Author prompts and evaluators

Loop can suggest, create, and save evaluators based on your application's behavior. A [scorer](https://docs.braintrust.dev/evaluate/write-scorers.md) returns a number, and a classifier returns a label. New and updated evaluators open in the workspace for review.

Loop can also create [human review scores](https://docs.braintrust.dev/annotate/human-review.md#configure-review-scores), including categorical choices, sliders, and free-form inputs.

Loop can also create and update prompts, including [revising playground prompts from your annotations](https://docs.braintrust.dev/evaluate/playgrounds.md#annotate-outputs) on individual outputs.

Example prompts:

> [accordion] Suggest scorers for my project
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Suggest useful scorers for my project.
> ```

> [accordion] Write a scorer for known errors
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Write a scorer that detects the errors I just identified, then create and test it on a few traces.
> ```

> [accordion] Create a helpfulness judge
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Create an LLM-as-a-judge scorer for helpfulness based on these logs.
> ```

> [accordion] Set up online scoring
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Set up online scoring with the scorer I just created, and verify it runs automatically on new traces.
> ```

###  Curate datasets

Loop can [generate datasets](https://docs.braintrust.dev/annotate/datasets/create.md#generate-with-loop) from log patterns and edit existing rows. Edits open in a read-only workspace table that refreshes as Loop works. Select a row to inspect its trace inline.

Ask Loop to find and copy project log rows into a dataset in the same project. Each log row becomes a dataset row with a link to its source, without loading its full contents into context. This copies individual rows, not [full-trace references](https://docs.braintrust.dev/annotate/datasets/create.md#promote-traces-from-logs). See [Copy behavior and limits](https://docs.braintrust.dev/integrations/developer-tools/mcp.md#run-evals-and-edit-datasets) for details.

Requires data plane v2.18.0+ (Self-hosted customers must upgrade to use this feature.)

Loop can also manage dataset versions and the environments that point at them:

* **Snapshots.** Save a [snapshot](https://docs.braintrust.dev/annotate/datasets/manage.md#save-snapshots) before editing, or restore one to undo edits. Loop can list snapshots, their environment tags, and the current head version. Restoring uses the dataset UI's confirmation flow and can save the current state first.
* **Environments.** Point an [environment](https://docs.braintrust.dev/deploy/environments.md), such as `production` or `staging`, at a snapshot or clear its mapping. Loop can list your organization's environment slugs and create missing environments.

> [!WARNING]
>
> Restoring a dataset to a snapshot replaces its current rows, and retagging an environment changes which dataset state your applications resolve. Review what Loop proposes before you approve either action.

Example prompts:

> [accordion] Generate a dataset from errors
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Generate a dataset from logs with errors.
> ```

> [accordion] Add traces to a dataset
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Add these traces to my regression dataset and set the expected output.
> ```

> [accordion] Copy logs into a regression dataset
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Find log rows with a correctness score below 0.5 and copy them into my regression dataset.
> ```

> [accordion] Snapshot before editing
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Snapshot this dataset before you change anything, then remove the rows that no longer reflect our current behavior.
> ```

> [accordion] Review and restore a version
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> What snapshots exist for this dataset, and which one is production pointing at? Restore the one from before yesterday's edits.
> ```

> [accordion] Promote a dataset version
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Point the staging environment at the snapshot you just saved, and tell me which environments exist first.
> ```

###  Build views and dashboards

Loop can create and edit [custom views](https://docs.braintrust.dev/annotate/custom-views.md) for traces and dataset rows, and [build dashboard charts](https://docs.braintrust.dev/observe/dashboards/build-charts.md). Open new or updated dashboards in the workspace to review them beside the thread.

Example prompts:

> [accordion] Suggest trace visualizations
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> What are some useful trace visualizations you can make for me? List the options and I'll tell you which ones to create as custom trace views.
> ```

> [accordion] Build a custom dataset view
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Build a custom view for this dataset that renders the input and expected output side by side.
> ```

> [accordion] Build a daily cost dashboard
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Create a dashboard for daily cost analysis.
> ```

> [accordion] Investigate an error spike
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> This topic or error spike looks interesting. Show me which customers and interactions are affected, inspect representative traces, and explain the likely root cause.
> ```

> [accordion] Share a dashboard with your team
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Turn this Loop analysis into a shareable dashboard my team and executives can use to understand the result and monitor it over time.
> ```

###  Configure Topics

Give Loop a goal to configure the [Topics](https://docs.braintrust.dev/observe/topics.md) pipeline:

* **Check the preprocessor.** Loop tests your project's effective default [preprocessor](https://docs.braintrust.dev/observe/topics/enable.md#write-a-custom-preprocessor) on real traces for usable conversation text. If it returns nothing, Loop writes and tests a custom preprocessor for your trace shape and can make it the project default.
* **Confirm the scope.** Loop looks for a grouping key, such as `metadata.conversation_id`, and recommends analyzing a span, trace, or group of related traces.
* **Validate the facets.** Loop tests a facet on up to ten representative traces and shows results inline for your feedback. Reuse that trace set as you refine the prompt. For larger sets, Loop runs an eval.
* **Evaluate candidates.** Loop compares candidate facet prompts in evals before saving them.
* **Enable the automation.** Loop creates the facets, enables the Topics automation, and offers to process a historical window.

> [!NOTE]
>
> Changing the project default preprocessor affects every function that uses it, including built-in Topics facets. Loop asks for approval before changing it.

> [!NOTE]
>
> Rewinding a Topics automation processes historical traces and draws from your monthly [model credits](https://docs.braintrust.dev/plans-and-limits.md#model-credits). Loop estimates the volume and asks for approval before it rewinds.

Example prompts:

> [accordion] Set up Topics from scratch
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Set up Topics for my project.
> ```

> [accordion] Configure Topics with a custom preprocessor
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Help me configure Topics with a custom preprocessor. My traces don't store conversation text on LLM spans.
> ```

> [accordion] Discover error categories
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> I want to discover the kinds of errors my users hit. Build a facet for that, validate it on real traces, and enable it.
> ```

## Run and automate

Test changes with evaluations, then schedule recurring checks or alerts.

###  Run evaluations

Loop runs evaluations two ways:

* **In a playground or experiment.** Loop can edit the task prompt or code, add or modify scorers, and re-run the evaluation to show the effect of a change.
* **As a new experiment.** Loop can run an experiment from a dataset, from rows it supplies inline, or from a prior experiment, using a saved or inline task and any scorers you choose. When a prior experiment supplies the data, its outputs become the expected values.

Evaluation runs open in the workspace's **Experiments** panel, where you can compare progress and scores without leaving the chat.

Example prompts:

> [accordion] Build a dataset and run an experiment
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Build a dataset from the traces where my agent gave a wrong answer, then run an experiment on it with my current prompt and scorers, and show me which cases still fail.
> ```

> [accordion] Evaluate an agent decision
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Evaluate whether this agent made the right decision, but only use the relevant messages, reasoning, and tool calls from these very large traces.
> ```

> [accordion] Compare a cheaper model
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Evaluate whether we can switch this workflow to a cheaper candidate model. Compare quality, cost, and latency, and tell me whether it is safe to roll out.
> ```

> [!NOTE]
>
> Loop runs evaluations in Braintrust rather than by writing an `Eval()` script. To keep evaluations in your own repository and run them in CI, see [Run evaluations in code](https://docs.braintrust.dev/evaluate/run-in-code.md).

###  Manage automations and alerts

Turn an investigation into an alert or a scheduled check:

* **Alerts.** Create an [alert](https://docs.braintrust.dev/observe/alerts.md) for matching logs, aggregates crossing a threshold over a recent window, or environment updates. Loop generates the filter from your description.
* **Automations.** Create a [scheduled Loop job](https://docs.braintrust.dev/loop/automations.md) with its own instructions and schedule. Loop can also list existing online scoring rules, alerts, exports, retention policies, and Topics automations with their full configurations.
* **Status.** Pause an automation or make it active again.
* **Updates.** Rewrite an alert or Loop automation under its existing name. Scheduled exports and retention policies are inspect-only.

> [!WARNING]
>
> Rewriting an automation replaces its entire configuration. Ask Loop to read the current configuration first, then review the proposed replacement before approving it.

You can also edit any automation directly in ** Settings**, on the page for its type: [** Loop**](https://www.braintrust.dev/app/~/configuration/automations/loop), [** Topics**](https://www.braintrust.dev/app/~/configuration/automations/topics), [** Online scoring**](https://www.braintrust.dev/app/~/configuration/automations/online-scoring), or [** Alerts**](https://www.braintrust.dev/app/~/configuration/automations/alerts).

Example prompts:

> [accordion] Alert on the errors you just found
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Create an alert that notifies me when the error you just found shows up in my logs again.
> ```

> [accordion] Alert on a quality regression
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Alert me when the average score for this scorer drops below 0.8 over any six hour window.
> ```

> [accordion] Alert on cost growth
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Watch my daily spend and alert me when it goes more than 20% above last week's average.
> ```

> [accordion] Audit what's configured
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> What automations are configured in this project, and which ones are paused?
> ```

> [accordion] Work out why a scheduled run found nothing
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Show me the configuration for my pattern discovery automation, then explain why yesterday's run didn't record anything.
> ```

## Get help

Find documentation and objects, or contact support through Loop.

###  Search docs and find objects

Loop can search the Braintrust documentation, resolve object names and URLs, list recent objects, summarize an experiment's metrics against a baseline, and generate shareable links to objects it creates.

Example prompts:

> [accordion] Look up how to set up human review
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> How do I set up human review?
> ```

> [accordion] Compare experiments
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Summarize my latest experiment compared to the previous one.
> ```

> [accordion] Get a shareable link
>
> ```text wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> Give me a shareable link to the dashboard you just made.
> ```

###  Request support

Ask Loop for help, or type `/` and choose the support command, to file a ticket with Braintrust. Loop guides you through the request and includes relevant context from your page and conversation.

## Next steps

* [Run Loop on a schedule](https://docs.braintrust.dev/loop/automations.md) to turn any of this into recurring work.
* [Enable Patterns](https://docs.braintrust.dev/observe/patterns/enable.md) for automatic discovery of recurring problems.
* [Manage threads and limits](https://docs.braintrust.dev/loop/manage.md) for sandbox behavior and cleanup.
* [What Loop can't do](https://docs.braintrust.dev/loop.md#what-loop-cant-do) for the boundaries it works inside.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.