> ## Documentation Index
> Fetch the complete documentation index at: https://docs.braintrust.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Workflow evaluations

> Submit evaluation tasks and scorers to asynchronous provider APIs, then collect results through polling or webhooks in one Braintrust experiment.

> [!WARNING]
>
> TypeScript workflow evaluations are in [public preview](https://docs.braintrust.dev/feature-lifecycle.md) and can change before reaching general availability.

> [!NOTE]
>
> Requires TypeScript SDK v3.37.0 or later. Import from the top-level `braintrust` package. The API is marked experimental in the SDK.

Workflow evaluations let you submit tasks or scorers to an asynchronous provider API, such as the [OpenAI Batch API](https://developers.openai.com/api/docs/guides/batch), and collect the results later. Use them for evaluations that run through a batch API, which providers often offer at a discount, or that need to continue across process exits.

You write functions that submit requests, check whether they're done, and fetch the results. The SDK calls them, saves progress in a store, and records the outputs and scores in one Braintrust experiment.

## How workflow evaluations work

A workflow evaluation has three stages:

1. **[Define an evaluation](#define-an-evaluation)**: Call [`defineWorkflowEval()`](https://docs.braintrust.dev/sdks/typescript/api-reference#defineworkfloweval) with your data, task, scorers, and a store. It works like [`Eval()`](https://docs.braintrust.dev/sdks/typescript/api-reference#eval), except that the task or scorers can send their work to a provider's asynchronous API. It returns an object with [methods](https://docs.braintrust.dev/sdks/typescript/api-reference.md#workflow-evaluation-methods) that you use to start and resume runs.
2. **[Start the run](#start-the-run)**: Call [`start()`](https://docs.braintrust.dev/sdks/typescript/api-reference#workflow-evaluation-methods) to submit the requests. It returns a `runId` right away, without waiting for the provider.
3. **[Resume until completion](#resume-until-completion)**: Pass the `runId` to [`poll()`](https://docs.braintrust.dev/sdks/typescript/api-reference#workflow-evaluation-methods) on a schedule, or to [`processSubmissionResult()`](https://docs.braintrust.dev/sdks/typescript/api-reference#workflow-evaluation-methods) from a webhook handler. Each call collects the results that are ready and scores them. The run is done when its status is `"completed"`.

The following example walks through each of these stages.

## Example: OpenAI Batch API

This example sends questions about capital cities through the [OpenAI Batch API](https://developers.openai.com/api/docs/guides/batch) and scores each answer against the expected city. Here's what the script does:

* Submits the cases to OpenAI together, in one batch.
* Polls OpenAI until the batch finishes, then collects each answer.
* Scores each answer with `exactMatch()`, a plain scorer function, and logs the results to Braintrust.

To get started, install the `braintrust` and `openai` packages, and set your `BRAINTRUST_API_KEY` and `OPENAI_API_KEY` environment variables:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
npm install "braintrust@>=3.37.0" openai
export BRAINTRUST_API_KEY="your-braintrust-api-key"
export OPENAI_API_KEY="your-openai-api-key"
```

Then, save the evaluation and its callbacks in `evaluation.ts`. They live in their own module so the Redis and webhook examples later on this page can import them:

```typescript evaluation.ts expandable theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import OpenAI, { toFile } from "openai";
import {
  WorkflowBatchTask,
  defineWorkflowEval,
  type EvalScorerArgs,
  type WorkflowEvalStore,
} from "braintrust";

export const client = new OpenAI();

// Data your submit callback returns and the SDK saves between calls.
// The SDK passes it back to the polling and collection callbacks.
// Here it holds the OpenAI batch ID. You choose the fields. Use a type
// alias rather than an interface, because TypeScript can't confirm that an
// interface is JSON-serializable.
export type Submission = { batchId: string };

// One request in a batch. customId is the ID the SDK gives each item in a
// batch. It goes to OpenAI as custom_id, so results can be matched to items.
export type BatchRequest = { customId: string; prompt: string };

// One entry per request, as read from the batch's output and error files.
export type BatchText =
  | { customId: string; text: string }
  | { customId: string; error: unknown };

// The fields this example reads from one line of a batch's output or error
// file.
type BatchResultLine = {
  custom_id: string;
  error?: unknown;
  response?: { status_code: number; body: OpenAI.Responses.Response };
};

// Checks that a parsed line has the custom_id that matches it to an item.
function isBatchResultLine(value: unknown): value is BatchResultLine {
  return (
    typeof value === "object" &&
    value !== null &&
    "custom_id" in value &&
    typeof value.custom_id === "string"
  );
}

// Sends the prompts to OpenAI as one batch.
export async function submitBatch(
  requests: BatchRequest[],
): Promise<Submission> {
  // The Batch API reads its requests from an uploaded JSONL file.
  const lines = requests.map(({ customId, prompt }) =>
    JSON.stringify({
      custom_id: customId,
      method: "POST",
      url: "/v1/responses",
      body: { model: "gpt-5-mini", input: prompt },
    }),
  );
  const batchFile = await client.files.create({
    file: await toFile(Buffer.from(lines.join("\n") + "\n"), "requests.jsonl"),
    purpose: "batch",
  });
  const batch = await client.batches.create({
    input_file_id: batchFile.id,
    endpoint: "/v1/responses",
    completion_window: "24h",
  });
  // The SDK saves this object and passes it to the polling and collect
  // callbacks.
  return { batchId: batch.id };
}

// The polling callback. Called during each poll() until the batch reports
// complete. It reports complete once OpenAI stops working on the batch,
// including when the batch fails or expires, so collect can record the
// requests that have no result as failed and the run can finish.
export async function pollBatch({ batchId }: Submission) {
  const batch = await client.batches.retrieve(batchId);
  switch (batch.status) {
    case "completed":
    case "failed":
    case "expired":
    case "cancelled":
      return { status: "complete" } as const;
    default:
      return { status: "pending" } as const;
  }
}

// Reads each request's answer, or its error, from the batch's output and
// error files.
export async function* collectText({
  batchId,
}: Submission): AsyncGenerator<BatchText> {
  const batch = await client.batches.retrieve(batchId);
  for (const fileId of [batch.output_file_id, batch.error_file_id]) {
    if (!fileId) {
      continue;
    }
    const content = await client.files.content(fileId);
    for (const line of (await content.text()).split("\n")) {
      if (!line.trim()) {
        continue;
      }
      const result: unknown = JSON.parse(line);
      // Stops on a line that can't be matched to an item. The SDK calls
      // collect again on the next poll() or webhook delivery.
      if (!isBatchResultLine(result)) {
        throw new Error(`Unexpected line in batch ${batchId} results: ${line}`);
      }
      if (result.error || result.response?.status_code !== 200) {
        yield {
          customId: result.custom_id,
          error: result.error ?? result.response?.body,
        };
        continue;
      }
      const text = result.response.body.output
        .flatMap((item) => (item.type === "message" ? item.content : []))
        .flatMap((part) => (part.type === "output_text" ? [part.text] : []))
        .join("");
      yield { customId: result.custom_id, text };
    }
  }
}

// A plain scorer function, like the ones you pass to Eval(). The SDK runs
// it on each case's collected output.
function exactMatch({
  output,
  expected,
}: EvalScorerArgs<string, string, string>) {
  return output.trim().toLowerCase() === expected.toLowerCase() ? 1 : 0;
}

// The store and completion mode are parameters so the Redis and webhook
// examples later on this page can reuse this function.
export function createEvaluation(
  store: WorkflowEvalStore,
  completionMode: "poll" | "webhook",
) {
  // The type arguments are the input, output, and expected output types.
  // TypeScript uses them to infer the parameter types of the callbacks below.
  return defineWorkflowEval<string, string, string>("capital-cities-eval", {
    store,
    // Each case needs a stable, unique id so the run can resume.
    data: [
      { id: "france", input: "Capital of France?", expected: "Paris" },
      { id: "japan", input: "Capital of Japan?", expected: "Tokyo" },
    ],
    task: new WorkflowBatchTask({
      // Puts up to 100 cases in each OpenAI batch, so this evaluation's two
      // cases go in one batch. Add trialCount to run each case more than once.
      batching: { maxSize: 100 },
      // Called once for each batch, with one item for each case in it.
      submit: (items) =>
        submitBatch(
          items.map(({ customId, input }) => ({
            customId,
            prompt: `${input} Reply with only the city name.`,
          })),
        ),
      // How the SDK learns that a batch finished: by polling, or from a
      // webhook that the SDK matches to the batch ID getExternalId returns.
      completion:
        completionMode === "poll"
          ? { mode: "poll", poll: pollBatch }
          : { mode: "webhook", getExternalId: ({ batchId }) => batchId },
      // Called after the batch completes. Returns one entry for each item,
      // matched by customId. The SDK records an item without an entry as
      // failed.
      collect: async function* (submission) {
        for await (const entry of collectText(submission)) {
          yield "error" in entry
            ? entry
            : { customId: entry.customId, result: { output: entry.text } };
        }
      },
    }),
    // Plain scorer functions, WorkflowScorer, and WorkflowBatchScorer
    // instances can go here.
    scores: [exactMatch],
  });
}
```

Next, save the script that runs the evaluation in `workflow-eval.ts`:

```typescript workflow-eval.ts theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import { setTimeout as sleep } from "node:timers/promises";
import { WorkflowEvalMemoryStore } from "braintrust";
import { createEvaluation } from "./evaluation";

async function main() {
  // The memory store works in this example because the whole run
  // happens in one process.
  const evaluation = createEvaluation(new WorkflowEvalMemoryStore(), "poll");
  // Submits the cases in one batch and returns right away with status
  // "waiting".
  let result = await evaluation.start();
  console.log(result.runId, result.status);
  // In an application, schedule poll() instead of sleeping in one process.
  while (result.status === "waiting") {
    await sleep(60_000);
    result = await evaluation.poll({ runId: result.runId });
    // pending counts submitted batches that haven't been collected.
    console.log(result.status, result.pending);
  }
  console.log(result.summary);
}

main();
```

Run it with `npx tsx workflow-eval.ts`. OpenAI batches can take minutes or hours, so the script can run for a while. After each `poll()`, it prints the run's status and how many batches are still pending. When it finishes, the Braintrust experiment contains two rows, each with an `exactMatch` score.

The rest of this section walks through the example's three stages.

### Define an evaluation

First, `evaluation.ts` defines the evaluation in `createEvaluation()`. In outline, the call looks like this:

```typescript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
const evaluation = defineWorkflowEval("capital-cities-eval", {
  data: [/* ... */],
  task: new WorkflowBatchTask({ batching, submit, completion, collect }),
  scores: [/* ... */],
  store: new WorkflowEvalMemoryStore(),
});
```

The following sections describe each argument. For every option, see [`defineWorkflowEval()`](https://docs.braintrust.dev/sdks/typescript/api-reference#defineworkfloweval).

> [accordion] Project and experiment
>
> The first argument is the project name, `"capital-cities-eval"` in the example. Braintrust creates the project if it doesn't exist. Each run logs to a new experiment unless you set `experimentName`.

> [accordion] Data
>
> The `data` argument accepts the same cases as `Eval()`. Each case needs a unique `id`, so the SDK can match results to cases when the run resumes. Cases can also set `metadata`, `tags`, and `trialCount`.
>
> Case data must be JSON-serializable, so the store can save it.

> [accordion] Task
>
> The `task` argument accepts a plain task function, a `WorkflowBatchTask`, or a `WorkflowTask`. Use a `WorkflowBatchTask` when the provider accepts many requests at once and returns their results later, as batch APIs do. The example's task, with polling completion, looks like this:
>
> ```typescript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> new WorkflowBatchTask({
>   batching: { maxSize: 100 },
>   submit: (items) =>
>     submitBatch(
>       items.map(({ customId, input }) => ({
>         customId,
>         prompt: `${input} Reply with only the city name.`,
>       })),
>     ),
>   completion: { mode: "poll", poll: pollBatch },
>   collect: async function* (submission) {
>     for await (const entry of collectText(submission)) {
>       yield "error" in entry
>         ? entry
>         : { customId: entry.customId, result: { output: entry.text } };
>     }
>   },
> })
> ```
>
> * **`batching`**: `maxSize` sets the most items in one batch. `start()` submits every case and trial right away, in batches of up to `maxSize` items.
> * **`submit`**: Sends one batch to the provider. Receives an array of items, one for each case and trial, with the case's fields and a `customId` that identifies the item within its batch, such as `"item-0"`. Use the `customId` as the provider's request ID, such as OpenAI's `custom_id`. Returns data, such as the provider's batch ID. The SDK saves it and passes it to the `completion` and `collect` callbacks when it calls them.
> * **`completion`**: How the SDK learns the batch is done: by polling with a function such as the example's `pollBatch()`, or from a webhook. See [completion options](https://docs.braintrust.dev/sdks/typescript/api-reference.md#completion-configuration).
> * **`collect`**: Fetches the finished batch's results and returns one `{ customId, result }` or `{ customId, error }` entry per item, as an array, an iterable, or an async iterable. `result` holds the task's `output`. The SDK records an item that has an error or no entry as failed: its row shows the error, and its scorers don't run.
>
> The store saves what `submit` and `collect` return, so both must be JSON-serializable. In the example, those are `{ batchId: ... }` and the model's answers.
>
> The example passes type arguments to `defineWorkflowEval<string, string, string>()` for the input, output, and expected output. TypeScript uses them to check the data and to infer the parameter types of callbacks written inline, so `input` is a `string` and `submission` is a `Submission` without annotations. For the fields of the submitted items and the collected entries, see [`WorkflowBatchTask`](https://docs.braintrust.dev/sdks/typescript/api-reference#workflowbatchtask-and-workflowbatchscorer).
>
> To send each case as its own request instead, use a `WorkflowTask`. See [Submit one request per case](#submit-one-request-per-case).

> [accordion] Scores
>
> The `scores` argument accepts [plain scorer functions](https://docs.braintrust.dev/evaluate/write-scorers.md), like the example's `exactMatch()`, `WorkflowBatchScorer` instances, and `WorkflowScorer` instances. Use a `WorkflowBatchScorer` when scoring needs requests that finish later, such as LLM-as-a-judge calls through a batch API. It has this shape:
>
> ```typescript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> new WorkflowBatchScorer({
>   name: "llmJudge",
>   batching: { maxSize: 100 },
>   submit: async (items) => {/* ... */},
>   completion: { mode: "poll", poll: pollBatch },
>   collect: async function* (submission) {/* ... */},
> })
> ```
>
> It takes the same callbacks as a `WorkflowBatchTask`, plus a `name`. Its items include the task's `output` along with the case's fields, and each `collect` entry's `result` has a `score`. A failed item omits that score and records the error in the row's `scorer_errors` metadata. See [`WorkflowBatchScorer`](https://docs.braintrust.dev/sdks/typescript/api-reference#workflowbatchtask-and-workflowbatchscorer).
>
> A `WorkflowBatchScorer` submits a batch once `maxSize` items have task outputs, or once no more items can get one. To submit partial batches sooner, set `batching.maxWaitMs`. The SDK submits a partial batch that has waited that long on the next `start()`, `poll()`, or `processSubmissionResult()` call.
>
> **Example: Score with a batch request**
>
> The following `llmJudge` scorer asks a model to grade each answer through the OpenAI Batch API. It reuses the example's `submitBatch()`, `collectText()`, and `pollBatch()`. To try it, add `WorkflowBatchScorer` to the `braintrust` import in `evaluation.ts`, then add this code right before `createEvaluation()`:
>
> ```typescript expandable theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> const llmJudge = new WorkflowBatchScorer({
>   // Names the score in Braintrust. Keep it the same across resumes.
>   name: "llmJudge",
>   // Waits for up to 100 task outputs, or until no more can arrive, then
>   // submits them in one grading batch.
>   batching: { maxSize: 100 },
>   // A scorer defined outside defineWorkflowEval() has no type arguments to
>   // infer from, so annotate the items with EvalScorerArgs.
>   submit: (
>     items: (EvalScorerArgs<string, string, string> & { customId: string })[],
>   ) =>
>     submitBatch(
>       items.map(({ customId, input, output, expected }) => ({
>         customId,
>         prompt: [
>           "Does the answer name the expected city? Reply with only 1 for yes or 0 for no.",
>           `Question: ${input}`,
>           `Expected: ${expected}`,
>           `Answer: ${output}`,
>         ].join("\n"),
>       })),
>     ),
>   // Reuses the task's polling callback, since both are OpenAI batches.
>   completion: { mode: "poll", poll: pollBatch },
>   // Called after pollBatch() reports the grading batch complete.
>   collect: async function* (submission) {
>     for await (const entry of collectText(submission)) {
>       if ("error" in entry) {
>         yield entry;
>         continue;
>       }
>       const verdict = entry.text.trim();
>       // Records an unexpected reply as a failed score for that case.
>       yield verdict === "0" || verdict === "1"
>         ? { customId: entry.customId, result: { score: Number(verdict) } }
>         : {
>             customId: entry.customId,
>             error: `Unexpected judge response: ${verdict}`,
>           };
>     }
>   },
> });
> ```
>
> Then change `scores: [exactMatch]` to `scores: [exactMatch, llmJudge]` and run `workflow-eval.ts` again. Your `exactMatch()` scorer runs as soon as the answers arrive. The same `poll()` submits one grading batch for both answers, and a later `poll()` collects the grades.

> [accordion] Store
>
> The `store` argument saves the run's progress between calls:
>
> * **[`new WorkflowEvalMemoryStore()`](https://docs.braintrust.dev/sdks/typescript/api-reference#workflowevalstore)**: Keeps progress in memory. Use it when the whole run happens in one process, as in the example.
> * **[`new WorkflowEvalRedisStore({ client })`](https://docs.braintrust.dev/sdks/typescript/api-reference#workflowevalstore)**: Keeps progress in Redis, using a connected `redis`, `ioredis`, or `@upstash/redis` client. Use it when the run must survive process exits or be resumed by another process. Records expire after seven days by default.
>
> For Redis options and custom stores, see [`WorkflowEvalStore`](https://docs.braintrust.dev/sdks/typescript/api-reference#workflowevalstore). Every process that works on a run must use the same store and the same evaluation definition.

### Start the run

Next, `main()` in `workflow-eval.ts` starts a run by calling `start()`, which submits the first batches and returns a result with the `runId`, without waiting for the provider.

**Save the `runId`.** You pass it to `poll()`, `processSubmissionResult()`, and [`status()`](https://docs.braintrust.dev/sdks/typescript/api-reference#workflow-evaluation-methods).

> [!WARNING]
>
> Calling `start()` again creates a new run and submits every request to the provider again. To continue an existing run, pass its `runId` to `poll()` or `processSubmissionResult()`.

### Resume until completion

Finally, the example calls `poll()` in a loop until every case is scored. A waiting run can resume in either of two ways, or both:

* **Polling**: Call `evaluation.poll({ runId })` on a schedule.
* **Webhooks**: Call `evaluation.processSubmissionResult()` when the provider sends a completion event.

Each task and scorer chooses its own method. If the task uses webhooks and a scorer uses polling, you still need to schedule `poll({ runId })` to collect the scorer results.

**Check progress.** Each call returns a result whose `status` stays `"waiting"` until the run completes, and whose `pending` field has `poll` and `webhook` counts of the batches not yet collected. To check without advancing the run, call `evaluation.status({ runId })`.

**Get the results.** When the status is `"completed"`, `result.summary` has the scores and a link to the experiment.

> [accordion] Resume with polling
>
> The main example runs in one process that waits until the run finishes, and it keeps progress in memory.
>
> For runs that take hours, an alternative is a separate script, such as the following `resume-eval.ts`, that runs in place of `workflow-eval.ts`. Each time it runs, it does one step and exits, and it saves progress in Redis so the next run continues where the last one stopped. It imports `createEvaluation()` from `evaluation.ts` so it doesn't repeat the evaluation. It requires the `redis` package (`npm install redis`) and a `REDIS_URL` environment variable, such as `redis://localhost:6379`:
>
> ```typescript resume-eval.ts expandable theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> import { createClient } from "redis";
> import { WorkflowEvalRedisStore } from "braintrust";
> import { createEvaluation } from "./evaluation";
>
> async function main() {
>   const redis = await createClient({ url: process.env.REDIS_URL }).connect();
>   try {
>     // Saves run state in Redis, so any process with the same Redis URL
>     // and evaluation definition can resume the run.
>     const evaluation = createEvaluation(
>       new WorkflowEvalRedisStore({ client: redis }),
>       "poll",
>     );
>     const runId = process.argv[2];
>     // With no argument, starts a new run and prints its run ID.
>     // With a run ID, checks the run's pending batches once and advances
>     // any cases whose results are ready.
>     const result = runId
>       ? await evaluation.poll({ runId })
>       : await evaluation.start();
>     console.log(result.runId, result.status);
>   } finally {
>     await redis.close();
>   }
> }
>
> main();
> ```
>
> To use it:
>
> * Running `npx tsx resume-eval.ts` with no arguments submits the requests and prints the run ID.
> * Running `npx tsx resume-eval.ts <runId>` advances the run once. A scheduled job can run it periodically until it prints `completed`.

> [accordion] Resume from provider webhooks
>
> Use [webhook completion](https://docs.braintrust.dev/sdks/typescript/api-reference.md#completion-configuration) when your provider sends an event when a request finishes. For OpenAI, configure a [webhook endpoint](https://developers.openai.com/api/docs/guides/webhooks) for `batch.completed`, `batch.failed`, `batch.expired`, and `batch.cancelled` events, and use a shared Redis store. The following `webhook-handler.ts` reuses `createEvaluation()` and `client` from `evaluation.ts`:
>
> ```typescript webhook-handler.ts expandable theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> import { createClient } from "redis";
> import { WorkflowEvalRedisStore } from "braintrust";
> import { client, createEvaluation } from "./evaluation";
>
> const redis = createClient({ url: process.env.REDIS_URL });
> // Uses webhook completion. After submit, the SDK saves the batch ID, so a
> // later webhook can find the submission.
> const evaluation = createEvaluation(
>   new WorkflowEvalRedisStore({ client: redis }),
>   "webhook",
> );
>
> async function connectRedis() {
>   if (!redis.isOpen) {
>     await redis.connect();
>   }
> }
>
> // Starts a run that completes through webhooks.
> export async function startRun() {
>   await connectRedis();
>   return evaluation.start();
> }
>
> // Call this from your HTTP endpoint with the incoming request.
> export async function handleWebhook(request: Request) {
>   await connectRedis();
>   // Checks that the request came from OpenAI, using the signing secret in
>   // OPENAI_WEBHOOK_SECRET, and parses the event.
>   const event = await client.webhooks.unwrap(
>     await request.text(),
>     request.headers,
>   );
>   // Handles the events OpenAI sends when it stops working on a batch, and
>   // ignores the rest.
>   switch (event.type) {
>     case "batch.completed":
>     case "batch.failed":
>     case "batch.expired":
>     case "batch.cancelled":
>       break;
>     default:
>       return;
>   }
>   try {
>     // The SDK finds the run and the submission by batch ID, then calls the
>     // task's collect callback. It records each request without a result,
>     // such as in an expired batch, as failed.
>     return await evaluation.processSubmissionResult({
>       externalId: event.data.id,
>     });
>   } catch (error) {
>     // Skips batches that this evaluation didn't submit, and batches that use
>     // polling, such as llmJudge's grading batches.
>     if (
>       error instanceof Error &&
>       error.message === "No submission matches this result"
>     ) {
>       return;
>     }
>     throw error;
>   }
> }
> ```
>
> Set `OPENAI_WEBHOOK_SECRET` to the endpoint's [signing secret](https://developers.openai.com/api/docs/guides/webhooks#verifying-webhook-signatures) from OpenAI. Pass the incoming [`Request`](https://developer.mozilla.org/en-US/docs/Web/API/Request) from your HTTP endpoint, such as a Next.js route handler, to `handleWebhook()`. It reads the unparsed body, which signature verification requires. Return an error status if it throws, so OpenAI redelivers the event. Start the run with `startRun()`.
>
> Keep in mind:
>
> * **Failed batches**: The example passes `batch.failed`, `batch.expired`, and `batch.cancelled` events to `processSubmissionResult()` along with `batch.completed`. The SDK records each request without a result as failed, so the run still completes.
> * **Run ID**: The example passes only the batch ID. That works when each batch ID belongs to one run of the evaluation, as OpenAI batch IDs do. If several runs can share an external ID, also pass `runId`.
> * **Security**: `processSubmissionResult()` doesn't check that the event came from the provider. In the example, `unwrap()` does.
> * **Repeated events**: The same event can arrive more than once. Make your `collect` callbacks safe to run more than once.
>
> For how the SDK matches an event to a batch, see [`processSubmissionResult()`](https://docs.braintrust.dev/sdks/typescript/api-reference#workflow-evaluation-methods).

> [accordion] Handle failures and concurrency
>
> * **Concurrency**: `maxConcurrency` limits how many of your `submit`, `getExternalId`, polling, and `collect` callbacks the SDK runs at once during each call. It defaults to `10`. For a batch task or scorer, each `submit` call sends one batch. See [`defineWorkflowEval()`](https://docs.braintrust.dev/sdks/typescript/api-reference#defineworkfloweval).
> * **Provider failures**: When your polling callback returns [`{ status: "failed", error }`](https://docs.braintrust.dev/sdks/typescript/api-reference#completion-configuration), `evaluation.poll()` throws the error. The SDK checks the batch again on the next `poll()`, so a permanent failure makes every later `poll()` throw, and the run never completes. The example's `pollBatch()` avoids this by returning `{ status: "complete" }` for failed, expired, and cancelled batches, so `collect` records the requests without results as failed. The SDK doesn't resubmit or cancel requests that the provider accepted, so retries and cancellation are up to your integration.
> * **Errors in your functions**: For polling, the SDK retries your polling callback and your `collect` function on the next `poll()`. For webhooks, `collect` runs again only when `processSubmissionResult()` is called again, such as when the provider redelivers the event. If your `submit` function throws, the next `poll()` calls it again with the same items and `submissionId`, for both polling and webhook completion. If your provider supports idempotency keys, use the `submissionId` as the key so a retry doesn't submit twice. A `submit` that throws during `start()` makes `start()` throw before it returns the `runId`, so save `context.runId` in `submit` if you need to resume. The SDK runs your plain task, scorer, and classifier functions only once per case, so if one of them fails, that case never finishes. Fix the problem and start a new run.

## Submit one request per case

A `WorkflowTask` or `WorkflowScorer` submits one request for each case and trial instead of batching them. Use one when your provider's asynchronous API takes one request per job, or when each case's scorers should start as soon as that case's output is ready.

The following task sends each case to OpenAI as its own single-request batch. To try it, add `WorkflowTask` to the `braintrust` import in `evaluation.ts`, and replace the `WorkflowBatchTask` in `createEvaluation()` with this task:

```typescript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
task: new WorkflowTask({
  // Called once for each case in data, or once per trial if you set
  // trialCount. Sends the case as a single-request batch.
  submit: ({ input }, context) =>
    submitBatch(
      [
        {
          customId: context.submissionId,
          prompt: `${input} Reply with only the city name.`,
        },
      ],
    ),
  completion: { mode: "poll", poll: pollBatch },
  // Returns the batch's only result. Throwing makes the SDK call collect
  // again on the next poll().
  collect: async (submission) => {
    for await (const entry of collectText(submission)) {
      if ("error" in entry) {
        throw new Error(`Request failed: ${JSON.stringify(entry.error)}`);
      }
      return { output: entry.text };
    }
    throw new Error(`Batch ${submission.batchId} has no result`);
  },
})
```

It differs from a `WorkflowBatchTask` in these ways:

* **Items**: `submit` receives one item, without a `customId`. The `context.submissionId` is unique to each case and trial, so the example uses it as OpenAI's `custom_id`.
* **Results**: `collect` returns one result, such as `{ output }` for a `WorkflowTask` or `{ score }` for a `WorkflowScorer`, instead of entries keyed by `customId`.
* **Failures**: If `collect` throws, the SDK calls it again on the next `poll()`, so a request that can't succeed, such as one in an expired batch, keeps the run waiting. Start a new run to recover.
* **Scorers**: A `WorkflowScorer` submits each case's request as soon as that case's task output is ready.

See [`WorkflowTask` and `WorkflowScorer`](https://docs.braintrust.dev/sdks/typescript/api-reference#workflowtask-and-workflowscorer).

## Next steps

* Review [`defineWorkflowEval()`](https://docs.braintrust.dev/sdks/typescript/api-reference#defineworkfloweval) and the [workflow evaluation methods](https://docs.braintrust.dev/sdks/typescript/api-reference.md#workflow-evaluation-methods).
* Learn how to [run experiments in code](https://docs.braintrust.dev/evaluate/run-in-code.md).
* Add [custom scorers](https://docs.braintrust.dev/evaluate/write-scorers.md) to measure task quality.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.