> ## Documentation Index
> Fetch the complete documentation index at: https://docs.braintrust.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Reconcile OpenAI billing with Braintrust llm-cost

> Reconcile Braintrust llm-cost reporting with OpenAI billing by checking failed scorer calls, SDK retries, and API keys shared across projects.

> [!NOTE]
>
> **Applies to:**
>
> * Plan - Any
> * Deployment - Braintrust-hosted
> *
> * Use case - How to investigate and reconcile discrepancies between OpenAI billed requests and Braintrust llm-cost reporting for online scorer traffic, including failed calls, retries, and cross-project API key usage.

## Summary

Braintrust `llm-cost` only counts token usage that was recorded in spans. OpenAI can bill HTTP attempts that never produced recorded usage. Run project-level SQL counts, compare them to OpenAI `GET /v1/organization/usage/completions` by `api_key_id`, and then rotate or re-scope the API key and add logging to close the gap.

## What is happening

Braintrust records LLM token usage when a span with `span_attributes.type = 'llm'` contains metrics. If a scorer call fails before that span or before usage is logged, `llm-cost` will not include the billed tokens. `wrap_openai()` creates one span per SDK call and records the usage returned by the response. SDK-internal retries can produce multiple billed OpenAI HTTP attempts while producing a single span. That can make OpenAI `num_model_requests` exceed Braintrust span counts. Failed scorer runs still consume compute and can count toward execution charges even if token counts are missing.

## Fix or suggestion

### Option 1: reconcile counts across projects (recommended)

1. Run these SQL counts for the UTC day you are reconciling in each candidate project, replacing `<PROJECT_ID>` and the timestamps.

```sql theme={null}
-- all logged llm spans
SELECT COUNT(*) AS all_llm_spans
FROM project_logs('<PROJECT_ID>', shape => 'spans')
WHERE span_attributes.type = 'llm'
  AND created >= '2026-10-25T00:00:00Z'
  AND created <  '2026-10-26T00:00:00Z';

-- llm spans matching the prompt_cache_key filter
SELECT COUNT(*) AS cache_key_llm_spans
FROM project_logs('<PROJECT_ID>', shape => 'spans')
WHERE span_attributes.type = 'llm'
  AND metadata.extra_body.prompt_cache_key LIKE 'scorer:%'
  AND created >= '2026-10-25T00:00:00Z'
  AND created <  '2026-10-26T00:00:00Z';

-- llm spans marked explicitly as scorer usage
SELECT COUNT(*) AS scorer_purpose_llm_spans
FROM project_logs('<PROJECT_ID>', shape => 'spans')
WHERE span_attributes.type = 'llm'
  AND span_attributes.purpose = 'scorer'
  AND created >= '2026-10-25T00:00:00Z'
  AND created <  '2026-10-26T00:00:00Z';
```

2. Sum the `scorer_purpose_llm_spans` across all projects that might inherit the org-level key.
3. Compare that total to OpenAI usage from `GET /v1/organization/usage/completions` grouped by `api_key_id` and the `num_model_requests` field.
4. Where OpenAI shows billed requests but Braintrust has no matching spans, inspect functions that load the org-level secret and any projects that invoke those functions.

### Option 2: eliminate cross-project bleed by scoping or rotating the key

1. Create a new OpenAI key dedicated to the project or scope the existing key to the project level. In Braintrust, prefer a project-scoped env var named differently from the org var, for example `SCORER_OPENAI_API_KEY`.
2. Set the project env var to the new key and update your scorer code to read `api_key = os.environ["SCORER_OPENAI_API_KEY"]` (or the equivalent in your runtime). Do not remove the old key until you confirm no residual usage.
3. To write a project-scoped env var for a Braintrust-hosted deployment, use the following curl command. Replace `<BRAINTRUST_API_KEY>`, `<PROJECT_ID>`, and `<OPENAI_API_KEY>` with your values.

```bash theme={null}
curl --fail-with-body --silent --show-error \
  --request PUT \
  --url 'https://api.braintrust.dev/v1/env_var' \
  --header 'Authorization: Bearer <BRAINTRUST_API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{
    "object_type": "project",
    "object_id": "<PROJECT_ID>",
    "name": "SCORER_OPENAI_API_KEY",
    "value": "<OPENAI_API_KEY>"
  }'
```

4. Monitor the OpenAI `api_key_id` usage for the old key. If it drops to zero, the cross-project bleed was the cause. If it remains nonzero, continue searching for projects or functions that inherit the org-level key.

### Instrumentation and scorer changes to capture billed attempts

* Log each OpenAI attempt and the response presence and returned usage. Record retry counts. Use those fields in the span metadata.
* To reduce duplicate scorer runs from merges, delay scoring until the export window stabilizes. Test a delayed window (for example 1–3 hours) on a sample set before rolling out.

## How to confirm it worked

* Compare the OpenAI `GET /v1/organization/usage/completions` `num_model_requests` for the key to the summed `scorer_purpose_llm_spans` across projects for the same UTC day. They should match within expected retry error rates.
* After scoping or rotating the key, confirm the old key shows zero usage in the OpenAI dashboard or API for at least one full UTC day.

## Notes

* Use SQL when querying `project_logs(...)`. The `wrap_openai()` helper records usage returned by the response. If no usage is returned, spans have no token counts.
* Do not remove an org-level key until you have monitored the old key and validated that no other projects rely on it.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.