Custom code scorers with internet access

​

Summary

On Braintrust SaaS, custom code scorers in the UI editor can make outbound HTTP requests. Use an external search or web API to fetch snippets, compute relevance, and return a numeric score between 0 and 1. For self-hosted deployments, confirm your data plane allows outbound egress.

​

What is happening

A custom code scorer runs your handler code in an isolated runtime. The scorer does not include a built-in web crawler or search engine. The scorer can call external HTTP APIs and use the responses to compute a score. You must provide any search or web API yourself.

​

Fix or suggestion

For either option, store the provider API key under Environment variables on Settings > Runtime as SEARCH_API_KEY or OPENAI_API_KEY rather than embedding secrets in code.

​

Option 1: UI editor scorer (recommended)

1. Create a custom code scorer in Scorers and open the code editor.
2. Implement the handler with the correct signature def handler(input, output, expected, metadata): and call the search API. Process snippets and map a relevance value to 0 to 1. Example minimal Python scorer:
   import os
   import requests
   
   API_KEY = os.environ.get("SEARCH_API_KEY")  # Replace with your env var name
   
   def handler(input, output, expected, metadata):
       # Replace with your search endpoint and query params
       resp = requests.get(
           "https://api.example.com/search",
           params={"q": output["text"]},
           headers={"Authorization": f"Bearer {API_KEY}"},
           timeout=10,
       )
       data = resp.json()
       relevance = float(data.get("relevance", 0.0))
       # Map or clamp relevance to 0 to 1
       if relevance < 0.0:
           relevance = 0.0
       if relevance > 1.0:
           relevance = 1.0
       return {"score": relevance, "metadata": {"source": "search_api"}}
   
3. Save the scorer and attach it to your experiment or evaluation.

​

Option 2: Use an LLM responses API with web search enabled

1. Use a provider that offers web-enabled responses, for example, OpenAI Responses with web search enabled.
2. Call the provider from handler, extract cited content or snippets, then compute and return a 0 to 1 score.

​

How to confirm it worked

• Run a test row and verify the scorer returns a numeric score and optional metadata in the evaluation results.
• Check the scorer run logs for error messages.

​

Notes

• For implementation details, see Custom code scorers and the security documentation.
• Mind provider rate limits and cost when calling broad web search APIs. For production, use request timeouts and retries with exponential backoff.