> ## Documentation Index
> Fetch the complete documentation index at: https://docs.braintrust.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Autoevals Python API

> Python API reference for Autoevals v0.3.0

AutoEvals is a tool to quickly and easily evaluate AI model outputs.

## Installation

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
pip install autoevals
```

## LLM Evaluators

### Battle

Compare if a solution performs better than a reference solution.

### ClosedQA

Evaluate answer correctness using the model's knowledge.

### Factuality

Check factual accuracy against a reference.

### Humor

Rate the humor level in text.

### LLMClassifier

High-level classifier for evaluating text using LLMs.

**name** `Any` *required*

**prompt\_template** `Any` *required*

**choice\_scores** `Any` *required*

**model** `Any`

**use\_cot** `Any`

**max\_tokens** `Any`

**temperature** `Any`

**reasoning\_effort** `Any`

**reasoning\_enabled** `Any`

**reasoning\_budget** `Any`

**use\_responses\_api** `Any`

**engine** `Any`

**api\_key** `Any`

**base\_url** `Any`

**client** `Any`

### Possible

Evaluate if a solution is feasible and practical.

### Security

Evaluate if a solution has security vulnerabilities.

### Sql

Compare if two SQL queries are equivalent.

### Summary

Evaluate text summarization quality.

### Translation

Evaluate translation quality.

## String Evaluators

### EmbeddingSimilarity

String similarity scorer using embeddings.

**prefix** `Any`

**model** `Any`

**expected\_min** `Any`

**api\_key** `Any`

**base\_url** `Any`

**client** `Any`

### ExactMatch

A scorer that tests for exact equality between values.

### Levenshtein

String similarity scorer using edit distance.

## Numeric Evaluators

### NumericDiff

Numeric similarity scorer using normalized difference.

## JSON Evaluators

### JSONDiff

Compare JSON objects for structural and content similarity.

**string\_scorer** `Scorer`

**number\_scorer** `Scorer`

**preserve\_strings** `bool`

### ValidJSON

Validate if a string is valid JSON and optionally matches a schema.

**schema** `Any`

## List Evaluators

### ListContains

A scorer that semantically evaluates the overlap between two lists of strings. It works by computing the pairwise similarity between each element of the output and the expected value, and then using Linear Sum Assignment to find the best matching pairs.

**pairwise\_scorer** `Any`

**allow\_extra\_entities** `Any`

## RAGAS Evaluators

### AnswerCorrectness

Evaluates how correct the generated answer is compared to the expected answer.

**pairwise\_scorer** `Any`

**model** `Any`

**factuality\_weight** `Any`

**answer\_similarity\_weight** `Any`

**answer\_similarity** `Any`

**embedding\_model** `Any`

**client** `Any`

### AnswerRelevancy

Evaluates how relevant the generated answer is to the input question.

**model** `Any`

**strictness** `Any`

**temperature** `Any`

**embedding\_model** `Any`

**client** `Any`

### AnswerSimilarity

Evaluates how semantically similar the generated answer is to the expected answer.

**pairwise\_scorer** `Any`

**model** `Any`

**client** `Any`

### ContextEntityRecall

Measures how well the context contains the entities mentioned in the expected answer.

**pairwise\_scorer** `Any`

**model** `Any`

**client** `Any`

### ContextPrecision

Measures how precise and focused the context is for answering the question.

**pairwise\_scorer** `Any`

**model** `Any`

**client** `Any`

### ContextRecall

Measures how well the context supports the expected answer.

**pairwise\_scorer** `Any`

**model** `Any`

**client** `Any`

### ContextRelevancy

Evaluates how relevant the context is to the input question.

**pairwise\_scorer** `Any`

**model** `Any`

**client** `Any`

### Faithfulness

Evaluates if the generated answer is faithful to the given context.

**model** `Any`

**client** `Any`

## Moderation

### Moderation

A scorer that evaluates if AI responses contain inappropriate or unsafe content.

**threshold** `Any`

**api\_key** `Any`

**base\_url** `Any`

**client** `Any`

## Other

### DefaultModelConfig

Configuration for default models used by Autoevals.

### LLMClient

A client wrapper for LLM operations that supports both OpenAI SDK v0 and v1.

## Source Code

For the complete Python source code and additional examples, visit the [autoevals GitHub repository](https://github.com/braintrustdata/autoevals).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.