Applies to:
- Plan - Any
- Deployment - Braintrust-hosted
- Use case - How to investigate and reconcile discrepancies between OpenAI billed requests and Braintrust llm-cost reporting for online scorer traffic, including failed calls, retries, and cross-project API key usage.
Summary
Braintrustllm-cost only counts token usage that was recorded in spans. OpenAI can bill HTTP attempts that never produced recorded usage. Run project-level SQL counts, compare them to OpenAI GET /v1/organization/usage/completions by api_key_id, and then rotate or re-scope the API key and add logging to close the gap.
What is happening
Braintrust records LLM token usage when a span withspan_attributes.type = 'llm' contains metrics. If a scorer call fails before that span or before usage is logged, llm-cost will not include the billed tokens. wrap_openai() creates one span per SDK call and records the usage returned by the response. SDK-internal retries can produce multiple billed OpenAI HTTP attempts while producing a single span. That can make OpenAI num_model_requests exceed Braintrust span counts. Failed scorer runs still consume compute and can count toward execution charges even if token counts are missing.
Fix or suggestion
Option 1: reconcile counts across projects (recommended)
- Run these SQL counts for the UTC day you are reconciling in each candidate project, replacing
<PROJECT_ID>and the timestamps.
- Sum the
scorer_purpose_llm_spansacross all projects that might inherit the org-level key. - Compare that total to OpenAI usage from
GET /v1/organization/usage/completionsgrouped byapi_key_idand thenum_model_requestsfield. - Where OpenAI shows billed requests but Braintrust has no matching spans, inspect functions that load the org-level secret and any projects that invoke those functions.
Option 2: eliminate cross-project bleed by scoping or rotating the key
- Create a new OpenAI key dedicated to the project or scope the existing key to the project level. In Braintrust, prefer a project-scoped env var named differently from the org var, for example
SCORER_OPENAI_API_KEY. - Set the project env var to the new key and update your scorer code to read
api_key = os.environ["SCORER_OPENAI_API_KEY"](or the equivalent in your runtime). Do not remove the old key until you confirm no residual usage. - To write a project-scoped env var for a Braintrust-hosted deployment, use the following curl command. Replace
<BRAINTRUST_API_KEY>,<PROJECT_ID>, and<OPENAI_API_KEY>with your values.
- Monitor the OpenAI
api_key_idusage for the old key. If it drops to zero, the cross-project bleed was the cause. If it remains nonzero, continue searching for projects or functions that inherit the org-level key.
Instrumentation and scorer changes to capture billed attempts
- Log each OpenAI attempt and the response presence and returned usage. Record retry counts. Use those fields in the span metadata.
- To reduce duplicate scorer runs from merges, delay scoring until the export window stabilizes. Test a delayed window (for example 1–3 hours) on a sample set before rolling out.
How to confirm it worked
- Compare the OpenAI
GET /v1/organization/usage/completionsnum_model_requestsfor the key to the summedscorer_purpose_llm_spansacross projects for the same UTC day. They should match within expected retry error rates. - After scoping or rotating the key, confirm the old key shows zero usage in the OpenAI dashboard or API for at least one full UTC day.
Notes
- Use SQL when querying
project_logs(...). Thewrap_openai()helper records usage returned by the response. If no usage is returned, spans have no token counts. - Do not remove an org-level key until you have monitored the old key and validated that no other projects rely on it.