Run multi-turn agent evaluations

​

Summary

A custom evaluation task is a function that runs for each dataset row. It can send predefined user messages sequentially, preserve each generated agent response, and return the full conversation as one evaluation result. This pattern works with agents that use MCP tools, other tools, or no tools. Expose the task as a remote eval if you want to launch it from a playground while the code runs in your environment.

​

What is happening

When you prepopulate multiple user messages in a playground prompt, they are sent together before the model responds. Adding separate message fields does not create separate execution steps, so this setup does not generate an assistant response between the user messages. The custom task controls the conversation loop. It sends the first message, waits for the agent and any tool calls to finish, then appends the next user message to the conversation. Remote evals make that task callable from a playground and stream its results back.

​

Fix or suggestion

​

Option 1: Prefill both messages in the playground (if sequencing is not required)

1. Put shared instructions in the System message. Keep them outside the dataset.
2. Store per-row fields such as first_message and second_message in your dataset row.
3. In the prompt editor, add two separate user messages and reference {{input.first_message}} and {{input.second_message}}.
4. Run the evaluation. The trace shows one eval row with both user messages sent up front.

Use this approach when the model should receive both user messages together. If the agent must respond before receiving the second message, use Option 2. This applies even when the second message is predefined.

​

Option 2: Send messages sequentially in a custom evaluation task

1. Prepare the dataset row. Make one row represent a full conversation, with first_message and second_message in its input. Keep shared instructions outside the row.
2. Implement the conversation task. Run one conversation per dataset row. Inside the task:
   • Initialize a fresh agent conversation with your shared instructions and any tools it needs.
   • Send the first predefined user message from first_message.
   • Wait for the agent to finish, including any tool calls.
   • Record the agent response, tool history, and conversation state.
   • Send the second predefined user message from second_message in the same conversation.
   • Wait for the agent to finish and then return the complete transcript as the task output.
   The following TypeScript pseudocode illustrates this sequence. Agent and its methods are placeholders for your agent framework, not Braintrust SDK APIs:
   // Replace with your agent framework imports and configuration
   async function runConversation(evalRow) {
     const agent = new Agent();
   
     await agent.startConversation();
     await agent.sendUser(evalRow.first_message);
     await agent.waitForReply();
   
     await agent.sendUser(evalRow.second_message);
     await agent.waitForReply();
   
     return { transcript: agent.getTranscript() };
   }
   
   Adjust for your language and agent framework. Preserve the generated assistant messages and any tool history when sending the next user message.
3. Trace the conversation. Make sure every turn and tool call appears in the eval row’s trace. Returning a transcript does not create child spans on its own. How you do this depends on where the agent runs:
   • The agent runs inside the task. Model calls made with a wrapped client, and functions wrapped with traced(), nest under the eval row automatically. See Trace and debug your eval tasks.
   • The task calls a separate agent service for each turn. Export the task span’s context and pass it with each request, so the service’s spans attach to the eval row instead of starting new traces. See Trace distributed systems.
   In both cases, keep the conversation history in your task and pass the full message history on each call. Each model span then records the conversation up to that turn.
4. Choose how to run the task. Register your conversation function as the Eval(...) task and run it with the SDK. To launch it from a playground, follow the remote eval guide. The task controls the conversation sequence in either case.

For a concrete example of preserving generated assistant responses between turns, see the multi-turn task example.

​

Optional: Configure MCP tools

If your agent uses MCP tools, configure the server connection, authentication, and tool execution in your agent code. Your task code does not automatically inherit the MCP connection attached to a playground prompt. See Add external MCP servers for the distinction between prompt connections in the UI and SDK invocation. Include MCP requests and results in the conversation history, and instrument the calls as child spans. The message sequence stays the same whether your agent uses MCP tools or another tool integration.

​

Confirm it worked

• One eval row contains the full user and assistant transcript with the agent reply between the two user messages.
• The trace shows child spans for each model call and any tool calls under the same eval row.