Run multi-turn agent evaluations with MCP tools

​

Summary

A custom SDK evaluation task can send predefined user messages sequentially, preserve each generated agent response, and return the full conversation as one evaluation result. Expose that task as a remote eval to launch it from Playground while the code runs in your environment.

​

What is happening

When you prepopulate multiple User messages in a Playground prompt, they are sent together before the model responds. Adding separate message fields does not create separate execution steps, so this setup does not generate an assistant response between the user messages. The custom task controls the conversation loop. It sends the first message, waits for the agent and any tool calls to finish, then appends the next user message to the conversation. Remote evals make that task callable from Playground and stream its results back. If your task uses MCP tools, configure their connection and authentication in the agent code. The remote process does not automatically inherit the MCP connection attached to the Playground prompt.

​

Fix or suggestion

​

Option 1: Prefill both messages in Playground (if sequencing is not required)

1. Put shared instructions in the System message. Keep them outside the dataset.
2. Store per-row fields such as first_message and second_message in your dataset row.
3. In the prompt editor add two separate User messages and reference {{input.first_message}} and {{input.second_message}}.
4. Run the evaluation. The trace will show one eval row with both user messages sent up front.

This is the simplest approach. Use it only when the second user turn does not depend on the agent response.

​

Option 2: Remote eval that emits turns step-by-step (recommended when you need a reply between turns)

1. Prepare the dataset row. Make one row represent a full conversation, with first_message and second_message in its input. Keep shared instructions outside the row.
2. Implement the conversation task. Run one conversation per dataset row. Inside the task:
   • Initialize a fresh agent conversation with your MCP tools and authenticate the MCP connection in your code.
   • Send the first predefined user message from first_message.
   • Wait for the agent to finish and for any MCP tool calls to complete.
   • Record the agent response, tool history, and conversation state.
   • Send the second predefined user message from second_message in the same conversation.
   • Wait for the agent to finish and then return the complete transcript as the task output.
   The following TypeScript pseudocode illustrates this sequence. Agent and its methods are placeholders for your agent framework, not Braintrust SDK APIs:
   // Replace with your SDK and MCP client imports
   async function runConversation(evalRow) {
     const agent = new Agent({ mcpConfig: { apiKey: process.env.MCP_API_KEY } });
   
     await agent.startConversation();
     await agent.sendUser(evalRow.first_message);
     const firstReply = await agent.waitForReply();
     // Persist firstReply and any tool calls as spans
   
     await agent.sendUser(evalRow.second_message);
     const finalReply = await agent.waitForReply();
   
     return { transcript: agent.getTranscript(), spans: agent.getSpans() };
   }
   
   Adjust for your language and SDK. Configure MCP authentication and tool execution explicitly in your agent code. Do not rely on Playground to forward MCP credentials.
3. Trace the conversation. Instrument model and MCP calls as child spans under the same eval trace so the entire conversation and tool history appear in one eval row. Returning a spans field alone does not log them.

For a concrete example of preserving generated assistant responses between turns, see the multi-turn task example. For setup, follow the remote eval guide to register your conversation function as the Eval(...) task, serve the eval, and select it in Playground. Your task controls the conversation sequence; remote evals let Playground launch it and display the results.

​

Confirm it worked

• One eval row contains the full user and assistant transcript with the agent reply between the two user messages.
• The trace shows child spans for each model call and each MCP tool call under the same eval row.

​

Notes

• Use the Playground prefill approach only when the second user message does not need to wait for the agent.
• Use a remote eval when the test must preserve the agent response between turns.