Skip to main content
Applies to:
  • Plan - Any
  • Deployment - Any
  • Use case - Planning evaluation timelines when using provider batch APIs (OpenAI) with Braintrust workflow evaluations

Summary

The OpenAI Batch API has a 24-hour processing window per batch. This is a deadline, not a guarantee that every request succeeds. Plan evaluation timelines around the full window, allow time to handle failures or expiry, and use batch mode for non-urgent jobs.

What is happening

When you send work through the OpenAI Batch API, the provider processes requests within a 24-hour completion window. Batches can finish sooner, but batches that do not finish in time expire and unfinished requests are cancelled. Your integration must handle failed, expired, and cancelled outcomes as well as successful completion. See OpenAI’s batch expiration guidance.

Fix or suggestion

Braintrust workflow evaluations are in public preview and can change before reaching general availability.
Workflow evaluations require TypeScript SDK v3.37.0 or later, or Python SDK v0.39.0 or later. The TypeScript API is marked experimental and is imported from braintrust. For Python, import from braintrust.workflow_eval, or from braintrust with v0.45.0 or later.
  • Use batch pricing for large, non-urgent evaluation suites such as nightly regression runs, model comparisons, or bulk scoring.
  • Schedule runs so results are not needed before the 24-hour window elapses, with additional time to recover from failed or expired requests.
  • Poll the provider or your evaluation job status periodically. If the provider offers webhooks, register them to receive completion notifications.
  • Use Braintrust workflow evaluations in the SDK that support batch APIs. See TypeScript workflow evaluations and Python workflow evaluations for implementation details.

Option 2: Reduce latency at the cost of higher cost

  • Use synchronous model calls instead of batch APIs for time-sensitive cases.
  • Split work into smaller batches and run them in parallel to reduce per-batch wall-clock time.
  • Trade off cost and throughput by adjusting batch size and concurrency.

How to confirm it worked

  • Verify each batch’s terminal status and inspect its output and error files for successful, failed, or expired requests.
  • Check the experiment or workflow status in the Braintrust UI or via the SDK.
  • Confirm downstream systems receive the results through polling or webhook delivery.

Notes

  • OpenAI’s 24-hour window applies to processing each batch. It does not guarantee that your entire evaluation succeeds within 24 hours.
  • Use batch mode when cost savings outweigh the added latency. Refer to the workflow evaluations SDK pages above for examples and polling patterns.