Online scorers not finding production traces

Applies to:

Plan - Starter, Pro, Enterprise
Deployment - Braintrust-hosted, Self-hosted
Use case - Online scorers with time-based SQL filters in production environments

Summary

Issue: Online scorers configured with time-based filters like created > now() - 1 day fail to score production traces, showing missing scores in the UI. Cause: Time-based filters in online scorer queries can interfere with trace discovery due to timing mismatches between trace creation and scorer evaluation. Resolution: Remove time-based filters from online scorer SQL queries, as scorers automatically apply to new traces without needing time conditions.

Resolution Steps

Remove Time-Based Filters

Step 1: Edit scorer rule

Navigate to the Online Scoring modal and remove any created > now() - X conditions from your SQL filter.

-- Before
WHERE span_attributes.name IN ('agent_run', 'turn_start')
AND created > now() - 1 day

-- After
WHERE span_attributes.name IN ('agent_run', 'turn_start')

Step 2: Set sampling to 100%

Temporarily increase sampling rate to 100% to verify the scorer works on all matching traces. The sampling rate determines what percentage of incoming traces or spans that match the rule’s filter are selected for scoring. For example, a 50% sampling rate selects approximately half of the matching traces, while 100% selects all matching traces or spans.

Step 3: Test the automation

Click Test automation to confirm your SQL filter matches recent spans.

Step 4: Verify scoring

Create new traces and confirm scores appear. Some lag (seconds to minutes) is normal.

If No Results Appear After Testing

Step 1: Simplify filter

Temporarily remove span_attributes.name IN (...) to check if any spans match.

Step 2: Verify scorer output

Run the scorer against a recent trace in the scorer page to confirm it returns a numeric value. Check that your filter selects spans containing the content the scorer needs. If an execution returns score: null, inspect its input and any skip reason. Increasing sampling does not fix missing input data.

Step 3: Check LLM-judge scores

For LLM-judge scorers, ensure each choice maps to a unique numeric score in [0,1] range.

# Incorrect - both choices set to 0
clear: 0
showed_frustration: 0

# Correct
clear: 0
showed_frustration: 1

Additional Context

Online scorers automatically apply to new traces as they are created. Time-based filters are unnecessary and can cause the scorer to miss traces due to timing issues between trace creation and query evaluation. Some delay in score appearance is expected behavior.

​Summary

​Resolution Steps

​Remove Time-Based Filters

​Step 1: Edit scorer rule

​Step 2: Set sampling to 100%

​Step 3: Test the automation

​Step 4: Verify scoring

​If No Results Appear After Testing

​Step 1: Simplify filter

​Step 2: Verify scorer output

​Step 3: Check LLM-judge scores

​Additional Context

Summary

Resolution Steps

Remove Time-Based Filters

Step 1: Edit scorer rule

Step 2: Set sampling to 100%

Step 3: Test the automation

Step 4: Verify scoring

If No Results Appear After Testing

Step 1: Simplify filter

Step 2: Verify scorer output

Step 3: Check LLM-judge scores

Additional Context