Skip to main content
Applies to:
  • Plan - Any
  • Deployment - Braintrust-hosted
  • Use case - How to investigate and reconcile discrepancies between OpenAI billed requests and Braintrust llm-cost reporting for online scorer traffic, including failed calls, retries, and cross-project API key usage.

Summary

Braintrust llm-cost only counts token usage that was recorded in spans. OpenAI can bill HTTP attempts that never produced recorded usage. Run project-level SQL counts, compare them to OpenAI GET /v1/organization/usage/completions by api_key_id, and then rotate or re-scope the API key and add logging to close the gap.

What is happening

Braintrust records LLM token usage when a span with span_attributes.type = 'llm' contains metrics. If a scorer call fails before that span or before usage is logged, llm-cost will not include the billed tokens. wrap_openai() creates one span per SDK call and records the usage returned by the response. SDK-internal retries can produce multiple billed OpenAI HTTP attempts while producing a single span. That can make OpenAI num_model_requests exceed Braintrust span counts. Failed scorer runs still consume compute and can count toward execution charges even if token counts are missing.

Fix or suggestion

  1. Run these SQL counts for the UTC day you are reconciling in each candidate project, replacing <PROJECT_ID> and the timestamps.
  1. Sum the scorer_purpose_llm_spans across all projects that might inherit the org-level key.
  2. Compare that total to OpenAI usage from GET /v1/organization/usage/completions grouped by api_key_id and the num_model_requests field.
  3. Where OpenAI shows billed requests but Braintrust has no matching spans, inspect functions that load the org-level secret and any projects that invoke those functions.

Option 2: eliminate cross-project bleed by scoping or rotating the key

  1. Create a new OpenAI key dedicated to the project or scope the existing key to the project level. In Braintrust, prefer a project-scoped env var named differently from the org var, for example SCORER_OPENAI_API_KEY.
  2. Set the project env var to the new key and update your scorer code to read api_key = os.environ["SCORER_OPENAI_API_KEY"] (or the equivalent in your runtime). Do not remove the old key until you confirm no residual usage.
  3. To write a project-scoped env var for a Braintrust-hosted deployment, use the following curl command. Replace <BRAINTRUST_API_KEY>, <PROJECT_ID>, and <OPENAI_API_KEY> with your values.
  1. Monitor the OpenAI api_key_id usage for the old key. If it drops to zero, the cross-project bleed was the cause. If it remains nonzero, continue searching for projects or functions that inherit the org-level key.

Instrumentation and scorer changes to capture billed attempts

  • Log each OpenAI attempt and the response presence and returned usage. Record retry counts. Use those fields in the span metadata.
  • To reduce duplicate scorer runs from merges, delay scoring until the export window stabilizes. Test a delayed window (for example 1–3 hours) on a sample set before rolling out.

How to confirm it worked

  • Compare the OpenAI GET /v1/organization/usage/completions num_model_requests for the key to the summed scorer_purpose_llm_spans across projects for the same UTC day. They should match within expected retry error rates.
  • After scoping or rotating the key, confirm the old key shows zero usage in the OpenAI dashboard or API for at least one full UTC day.

Notes

  • Use SQL when querying project_logs(...). The wrap_openai() helper records usage returned by the response. If no usage is returned, spans have no token counts.
  • Do not remove an org-level key until you have monitored the old key and validated that no other projects rely on it.