Trace scorers rerun due to metrics.end updates

Applies to:

Plan - Any
Deployment - Any
Data plane - v2.11.0 or later
Use case - Explain why trace-level scorers may run multiple times for the same trace, show how to diagnose using timeline + score spans, and provide workarounds and the shipped fix.

Summary

Trace-level scorers can run more than once after post-hoc updates to metrics.end or asynchronous scoring failures and retries. Increase the trace automation’s Idle timeout (seconds) to reduce update-driven retriggers, or add a SQL exclusion to block subsequent scoring eligibility once the named automation span exists. The idle timeout and automation-span exclusion affect scoring eligibility. Neither prevents failure retries.

What is happening

Asynchronous scorer failures and retries can also cause repeated scorer executions if Braintrust cannot attribute the failure to a single scorer.

Fix or suggestion

Option 1: Increase the trace scorer idle window

Edit the trace-level automation.
Increase Idle timeout (seconds) to cover delayed updates. A value of 300 allows 5 minutes for updates and may reduce duplicate scoring, but does not guarantee prevention.
Save the automation and monitor scoring volume.

Cons: Larger idle windows delay initial scoring.

Option 2: Migrate root-span scorers to trace-level scorers

Create a trace-level scorer that evaluates the whole trace.
To restrict filter matching to root spans, add an is_root condition to the trace automation’s Filter. The trace scorer still receives the whole trace, with input, output, and metadata populated from the root span.
Disable the span-level root scorer after the trace-level scorer is exercising the same logic.
Test on a small project before rolling out organization-wide.

Pros: Trace scoring supports an idle timeout and avoids the identified ingestion-timing race. Cons: Requires changes to scorer configurations and testing. Migration does not guarantee that all duplicate executions stop.

Option 3: Add a SQL exclusion to block subsequent scoring eligibility

Use a filter that excludes traces containing the named trace-level scorer automation span. Replace the example automation name with the exact name shown on your trace’s automation span, including spaces. Example SQL filter fragment:

NOT any_span(
  span_attributes.type = "automation"
  AND span_attributes.purpose = "scorer"
AND span_attributes.name = "[M2] Trace level Scorers" -- replace with the exact name from your automation span
)

Add other any_span(...) exclusions if needed for agent-specific scorers. Pros: Blocks subsequent scoring eligibility once the named automation span exists. Cons: More brittle and requires maintaining the exclusion logic.

How to confirm it worked

Open the trace timeline and enable Include score spans in timeline from the timeline menu. Verify each scorer appears once at the expected offset.
Monitor scoring volume and model call counts for the affected scorers for several hours after the change.

Notes

For self-hosted deployments running a data plane version earlier than v2.11.0, upgrade the data plane to v2.11.0 or later.
If you still see duplicates after trying these suggestions, collect a redacted trace with scorer spans and contact support for further investigation.

​Summary

​What is happening

​Fix or suggestion

​Option 1: Increase the trace scorer idle window

​Option 2: Migrate root-span scorers to trace-level scorers

​Option 3: Add a SQL exclusion to block subsequent scoring eligibility

​How to confirm it worked

​Notes