> ## Documentation Index
> Fetch the complete documentation index at: https://docs.braintrust.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Brainstore OOM errors and retry-induced rate limits

> [!NOTE]
>
> **Applies to:**
>
> * Plan - Enterprise
> * Deployment - Self-hosted (GCP)
> *
> * Use case - Self-hosted deployments on GCP experiencing rate limiting on GCS object storage and hitting 10 of 10 retries.

## Summary

**Issue:** Self-hosted brainstore pods restart or fail, followed by cascading rate limit failures with `429` errors on `.managed.json` objects in GCS. Multiple retry attempts per item exhaust the GCS write limit of 1 write/second per object.

**Cause:** Excessive retry attempts on individual objects trigger GCS rate limits, creating a cascading failure loop.

**Resolution:** Deploy the fix from commit SHA `40fda545aed99a12b1d04064c6205f8450d209ec` which prevents multiple per-item retry attempts.

## Resolution Steps

### Deploy the fix using commit SHA

#### Step 1: Update Helm configuration

Set the `image.tag` value in your Helm `values.yaml` to the fix commit SHA where a fix has been applied (alternatively, this will be included in a future official release):

```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
image:
    tag: "40fda545aed99a12b1d04064c6205f8450d209ec"

```

#### Step 2: Re-apply deployment

Apply the updated Helm configuration using Terraform or directly with helm upgrade.

#### Step 3: Monitor for rate limit errors

Check logs to verify `429` errors on `.managed.json` objects no longer appear.

### Alternative: Wait for official release

#### Step 1: Monitor changelog

The fix will be included in a future official data plane release.

#### Step 2: Update to latest version

Update your `image.tag` to the latest semantic version when available.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.