> ## Documentation Index > Fetch the complete documentation index at: https://docs.braintrust.dev/llms.txt > Use this file to discover all available pages before exploring further. # Scaling and storage > Size the API and Brainstore compute in a self-hosted data plane with Terraform variables and Helm resource settings. These settings size the API and Brainstore compute for your workload. The defaults are suitable for most deployments. Tune them for high ingestion volume, heavy eval workloads, or cluster capacity constraints. ## Configure AWS API ECS services On AWS, Terraform module v6.0 introduces ECS as the target runtime for the API, split into three services by workload type. Each service scales automatically through Application Auto Scaling, adjusting its running task count between a minimum and maximum based on multiple metrics, including CPU target tracking and event loop timing. The module exposes optional variables to tune per-task size and task counts. The defaults are suitable for most deployments. See [Upgrade to Terraform module v6](https://docs.braintrust.dev/admin/self-hosting/upgrade/v6.md) for the migration. 1024 CPU units equal 1 vCPU. **`braintrust-api`** handles general API traffic. | Variable | Default | | - | - | | `braintrust_api_cpu` | 1024 | | `braintrust_api_memory` | 8192 | | `braintrust_api_min_count` | 3 | | `braintrust_api_max_count` | 50 | **`braintrust-api-ingest`** handles ingestion paths (`/logs3`, `/logs3/overflow`, `/otel/v1/*`). On Terraform module v6.9.0 and later, it also handles `POST` requests to `/logs`, `/logs2`, `/v1/insert`, `/v1/*/*/insert`, and `/v1/*/*/feedback`. | Variable | Default | | - | - | | `braintrust_api_ingest_cpu` | 1024 | | `braintrust_api_ingest_memory` | 8192 | | `braintrust_api_ingest_min_count` | 6 | | `braintrust_api_ingest_max_count` | 200 | **`braintrust-api-background`** handles background paths (evals, function invocation, proxy). | Variable | Default | | - | - | | `braintrust_api_background_cpu` | 1024 | | `braintrust_api_background_memory` | 8192 | | `braintrust_api_background_min_count` | 3 | | `braintrust_api_background_max_count` | 50 | For example, to raise the ingestion service's floor for a high-volume deployment: ```hcl theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} braintrust_api_ingest_cpu = 2048 braintrust_api_ingest_min_count = 12 ``` Raise `braintrust_api_ingest_min_count` for deployments with spiky ingestion volume or if you expect a launch that will suddenly increase log traffic, and `braintrust_api_background_min_count` for heavy eval workloads. ## Prune AWS quarantine function versions On AWS, the data plane creates Lambda functions and function versions at runtime to run custom scorers and tools in the quarantine VPC. Terraform doesn't manage them, so old versions accumulate. On Terraform module v7.0.0 and later, you can deploy a Lambda function that runs hourly and deletes versions of quarantine functions that nothing references. It's off by default and requires `enable_quarantine_vpc = true`: ```hcl theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} enable_quarantine_pruner = true quarantine_pruner_mode = "dry-run" ``` `quarantine_pruner_mode` controls what each run does: * `off` (default): The function runs and exits without checking versions. * `dry-run`: The function logs the versions it would delete. * `on`: The function deletes unreferenced versions. Start with `dry-run`, review the function's logs, then set `on`. The function's IAM role can list and delete only versions of functions tagged as quarantine functions, and can't delete a whole function. ## Configure Helm API workload isolation > [!NOTE] > > This feature is available to Kubernetes deployments using Helm chart 6.13.0+. AWS ECS deployments use the Terraform-managed routing described above. Setting `api.workloadIsolation.enabled: true` creates dedicated `braintrust-api-ingest` and `braintrust-api-background` Deployments and Services alongside the default `braintrust-api` pool. The pools share the same image and base configuration while allowing independent replica counts, resources, probes, rollout settings, and disruption budgets, so classified ingestion or background load does not consume the default API pool's capacity. Your ingress or gateway must keep `braintrust-api` as its default backend and route the following paths to the specialized pools: | Pool | Paths | | - | - | | `braintrust-api-ingest` | `POST /logs3`, `POST /otel/v1/traces`, `POST /attachment`, `POST /attachment/status` | | `braintrust-api-background` | `POST /v1/eval` (exact) and `POST /v1/eval/` (prefix), `POST /function/eval`, `POST /function/sandbox`, `POST /function/use`, `POST /function/invoke-async-batch`, `POST /function/insert-functions`, `POST /automation/logs/trigger`, `POST /automation/trigger` (chart 6.20.0+), and all methods for `/v1/proxy/chat/completions` and `/v1/proxy/responses` | | `braintrust-api` (default) | All requests not matched by an ingest or background route | If your ingress cannot match requests by HTTP method (for example, GKE Ingress), route the listed paths for all methods instead. Pools use fixed replica counts by default. Configure them with `api.replicas`, `api.workloadIsolation.ingest.replicas`, and `api.workloadIsolation.background.replicas`. On GKE, you can scale each pool automatically instead. See [Configure GKE API autoscaling](#configure-gke-api-autoscaling). ### Roll out workload isolation for an existing deployment For an existing deployment, stage the rollout so you can verify each pool before routing traffic to it: 1. Create the pools without routing traffic to them or switching Brainstore's internal AI proxy. Set `api.workloadIsolation.brainstoreAiProxyToBackground: false` and apply: ```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} api: workloadIsolation: enabled: true brainstoreAiProxyToBackground: false ``` 2. Verify the ingest and background pools are ready. Then update your ingress or gateway to route the classified paths to the new Services, and set `brainstoreAiProxyToBackground: true` to switch Brainstore's internal AI proxy to the background pool: ```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} api: workloadIsolation: enabled: true brainstoreAiProxyToBackground: true ``` To roll back, first return the classified paths and `brainstoreAiProxyToBackground` to the default API Service and verify it is serving them, then disable workload isolation in the chart. The chart cannot update an external ingress or gateway on its own, so disabling isolation before rerouting would leave the classified paths pointed at Services that no longer exist. ### Integrate with the Istio VirtualService When using the chart-managed Istio VirtualService, set `virtualService.workloadIsolation.enabled: true` after the isolated pools are healthy. The chart then renders the ingest and background route contract before your existing `virtualService.http` rules. The classified routes take precedence, so do not use `virtualService.http` to override a classified path. > [!WARNING] > > Enable this only after the ingest and background pools are running and ready. This option requires both `virtualService.enabled: true` and `api.workloadIsolation.enabled: true`. The chart fails to render if either is missing. ```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} virtualService: workloadIsolation: enabled: true ``` ### Configuration reference | Key | Default | Description | | - | - | - | | `api.workloadIsolation.enabled` | `false` | Create dedicated ingest and background API pools. | | `api.workloadIsolation.brainstoreAiProxyToBackground` | `true` | Route Brainstore's internal AI proxy to the background pool. Set to `false` while staging pools for an existing deployment. | | `api.workloadIsolation.ingest.name` | `braintrust-api-ingest` | Name of the ingest Deployment. | | `api.workloadIsolation.ingest.replicas` | `3` | Number of ingest pod replicas. | | `api.workloadIsolation.ingest.service.name` | `braintrust-api-ingest` | Name of the ingest Service. | | `api.workloadIsolation.ingest.podDisruptionBudget.enabled` | `true` | Enable a PodDisruptionBudget for the ingest pool. | | `api.workloadIsolation.ingest.podDisruptionBudget.minAvailable` | `1` | Minimum available ingest pods during voluntary disruptions. | | `api.workloadIsolation.ingest.topologySpread.enabled` | `true` | Enable topology spread constraints for the ingest pool. | | `api.workloadIsolation.background.name` | `braintrust-api-background` | Name of the background Deployment. | | `api.workloadIsolation.background.replicas` | `3` | Number of background pod replicas. | | `api.workloadIsolation.background.service.name` | `braintrust-api-background` | Name of the background Service. | | `api.workloadIsolation.background.podDisruptionBudget.enabled` | `true` | Enable a PodDisruptionBudget for the background pool. | | `api.workloadIsolation.background.podDisruptionBudget.minAvailable` | `1` | Minimum available background pods during voluntary disruptions. | | `api.workloadIsolation.background.topologySpread.enabled` | `true` | Enable topology spread constraints for the background pool. | | `virtualService.workloadIsolation.enabled` | `false` | Render the ingest and background route contract in the chart-managed VirtualService. Requires `virtualService.enabled: true` and `api.workloadIsolation.enabled: true`. | ### Default pool availability settings Helm chart 6.13.0 or later also exposes availability settings for the default `braintrust-api` pool. The defaults preserve the chart's previous behavior, so no configuration changes are required when upgrading. The ingest and background pools inherit these values as their base configuration. | Key | Default | Description | | - | - | - | | `api.strategy.type` | `RollingUpdate` | Rollout strategy for the API Deployment. | | `api.strategy.rollingUpdate.maxSurge` | `100%` | Maximum number of extra pods created during a rollout. | | `api.strategy.rollingUpdate.maxUnavailable` | `0` | Maximum number of pods that can be unavailable during a rollout. | | `api.podDisruptionBudget.enabled` | `false` | Enable a PodDisruptionBudget for the default API pool. | | `api.podDisruptionBudget.minAvailable` | `1` | Minimum available default-pool pods during voluntary disruptions. | | `api.topologySpread.enabled` | `false` | Enable topology spread constraints for the default API pool. | | `api.topologySpread.maxSkew` | `1` | Maximum pod-count skew across topology domains. | | `api.topologySpread.topologyKey` | `topology.kubernetes.io/zone` | Node label that defines the topology domain. | | `api.topologySpread.whenUnsatisfiable` | `ScheduleAnyway` | Whether the scheduler places pods when the constraint cannot be met. | ## Configure GKE API autoscaling > [!WARNING] > > This feature is available to GKE deployments using Helm chart 6.15.0 or later. It is built on GKE's `AutoscalingMetric` resource, which Google classifies as Preview (Pre-GA), so Google may change or discontinue it. Setting `api.autoscaling.enabled: true` renders a HorizontalPodAutoscaler and an `AutoscalingMetric` for each API pool. Each pool then scales on three signals: * CPU utilization scoped to the `api` container, so sidecars and `extraContainers` do not skew the measurement. * Node.js event-loop utilization, as a ratio from 0 to 1. * Mean Node.js event-loop delay, in seconds. These are the same three signals AWS uses through Application Auto Scaling. Default CPU (50%) and event-loop utilization (40%) match. Event-loop delay is the exception: AWS uses step scaling, which Kubernetes HPA does not support, so GKE uses target tracking at 50 ms instead. See [Configure AWS API ECS services](#configure-aws-api-ecs-services). ### Prerequisites * Data plane v2.9.0 or later, which serves the Prometheus metrics endpoint on the API health server. * GKE 1.35.1-gke.1396000 or later. * The Performance HPA profile enabled on the cluster. * The Autoscaling API enabled on the cluster. * `roles/autoscaling.metricsWriter` granted to every node service account. * The Autoscaling API included in your service perimeter, if you use VPC Service Controls. `helm install` and `helm upgrade` fail fast if the chart detects a missing prerequisite, rather than rendering a partial configuration: * Setting `cloud` to anything other than `google` fails, because autoscaling is supported only on GKE. * A cluster without the `autoscaling.gke.io/v1beta1` API fails. Verify with `kubectl api-resources | grep autoscalingmetric`. ### Enable autoscaling ```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} api: autoscaling: enabled: true minReplicas: 4 maxReplicas: 50 ``` When autoscaling is enabled for a pool, the chart omits `replicas` from that pool's Deployment and the HorizontalPodAutoscaler controls the replica count instead. The chart also sets `ENABLE_PROMETHEUS_METRICS` to `true` and exposes `api.healthServer.port` (`8001` by default) on the `api` container so the `AutoscalingMetric` can scrape it. Account for that port in any network policy that restricts traffic to the API pods. ### Override autoscaling per pool With `api.workloadIsolation.enabled: true`, the ingest and background pools inherit every value under `api.autoscaling` and can override any of them under `api.workloadIsolation..autoscaling`, including the metric targets and the scaling behavior: ```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} api: autoscaling: enabled: true minReplicas: 3 maxReplicas: 40 workloadIsolation: enabled: true ingest: autoscaling: minReplicas: 3 maxReplicas: 60 eventLoopUtilization: targetAverageValue: "0.5" behavior: scaleUp: stabilizationWindowSeconds: 30 background: autoscaling: minReplicas: 3 maxReplicas: 50 ``` This example turns on workload isolation at the same time as autoscaling, which suits a new deployment. On an existing deployment, stage the pools first. The example leaves `api.workloadIsolation.brainstoreAiProxyToBackground` at its default `true`, which switches Brainstore's internal AI proxy to the background pool as soon as you apply it. See [Roll out workload isolation for an existing deployment](#roll-out-workload-isolation-for-an-existing-deployment). Setting `api.workloadIsolation..autoscaling.enabled: false` keeps that pool at a fixed replica count while its siblings continue to scale. That pool's `replicas` value is honored again. > [!TIP] > > The Helm chart ships [`examples/google-api-isolation-autoscaling/values.yaml`](https://github.com/braintrustdata/helm/blob/main/braintrust/examples/google-api-isolation-autoscaling/values.yaml) as a starting point for this pattern. Combine it with an Autopilot or Standard values file, which supply the rest of the configuration. For background on the GKE resource behind this feature, see Google's [Expose custom metrics for autoscaling](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/expose-custom-metrics-autoscaling). ### Configuration reference | Key | Default | Description | | - | - | - | | `api.autoscaling.enabled` | `false` | Autoscale every API pool. Requires `cloud: google`. | | `api.autoscaling.minReplicas` | `4` | Minimum replicas for the default pool. Other pools inherit this value unless overridden. | | `api.autoscaling.maxReplicas` | `50` | Maximum replicas for the default pool. Other pools inherit this value unless overridden. | | `api.autoscaling.metricsPath` | `/metrics` | Prometheus metrics path on the API health server. | | `api.autoscaling.cpu.targetAverageUtilization` | `50` | Target CPU utilization percentage for the `api` container. | | `api.autoscaling.eventLoopUtilization.targetAverageValue` | `"0.4"` | Target event-loop utilization, as a ratio from 0 to 1. `"0.4"` is 40%. | | `api.autoscaling.eventLoopDelayMean.targetAverageValue` | `"0.05"` | Target mean event-loop delay in seconds. `"0.05"` is 50 ms. | | `api.autoscaling.behavior.scaleDown.stabilizationWindowSeconds` | `300` | Stabilization window before scaling down. Passed through to the HorizontalPodAutoscaler. | | `api.autoscaling.behavior.scaleUp.stabilizationWindowSeconds` | `60` | Stabilization window before scaling up. Passed through to the HorizontalPodAutoscaler. | | `api.annotations.hpa` | `{}` | Annotations added to each generated HorizontalPodAutoscaler. | | `api.annotations.autoscalingMetric` | `{}` | Annotations added to each generated `AutoscalingMetric`. | | `api.workloadIsolation.ingest.autoscaling.minReplicas` | `3` | Minimum replicas for the ingest pool. | | `api.workloadIsolation.ingest.autoscaling.maxReplicas` | `200` | Maximum replicas for the ingest pool. | | `api.workloadIsolation.background.autoscaling.minReplicas` | `3` | Minimum replicas for the background pool. | | `api.workloadIsolation.background.autoscaling.maxReplicas` | `50` | Maximum replicas for the background pool. | ## Configure Brainstore Fast reader settings apply to every cloud. Resource, rollout, and startup gate settings apply to GCP and Azure deployments using the Helm chart. ### Set resources > [!WARNING] > > Upgrading from Helm chart 6.25.0 or earlier to 6.26.0 or later doubles the default memory requests and limits for all Brainstore roles. Pods can stay `Pending` if your nodes lack capacity. Before upgrading, confirm that your nodes have sufficient memory or explicitly set the memory requests and limits under `brainstore..resources` to retain your earlier values. > [!NOTE] > > This section applies to GCP and Azure deployments using the Helm chart (v5.0.1+). AWS deployments manage Brainstore resources automatically. Starting in Helm chart v5.0.1, the `resources` block for each Brainstore component (`brainstore.reader`, `brainstore.writer`, `brainstore.fastreader`) is passed through as-is to the Kubernetes pod spec. You can omit `limits` entirely, set them to `{}`, or supply any valid Kubernetes resource spec. The examples below use the [recommended production sizing](https://docs.braintrust.dev/admin/self-hosting/index.md#hardware-requirements): 16 vCPUs and 64 GiB of memory for readers and fast readers, and 32 vCPUs and 128 GiB for writers. Helm chart 6.26.0 and later use these values as defaults. On earlier versions, set them explicitly in `values.yaml`. > [!TIP] > > Omitting `limits` sets the pod QoS class to Burstable, which prevents CPU throttling and allows pods to use available node capacity. This can improve query performance on nodes with spare capacity, but increases the risk of resource contention if multiple pods compete for the same node. ```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} # With limits (Guaranteed QoS) brainstore: reader: resources: requests: cpu: "16" memory: "64Gi" limits: cpu: "16" memory: "64Gi" # Without limits (Burstable QoS, allows CPU and memory bursts) brainstore: reader: resources: requests: cpu: "16" memory: "64Gi" writer: resources: requests: cpu: "32" memory: "128Gi" fastreader: resources: requests: cpu: "16" memory: "64Gi" ``` > [accordion] Auto-derived environment variables (Helm v5.1.0+) > > As of Helm chart v5.1.0, `BRAINSTORE_RESPONSE_CACHE_URI` and `BRAINSTORE_CODE_BUNDLE_URI` are automatically populated from your `objectStorage` configuration and do not need to be set manually. The chart derives these values as follows: > > | Cloud | Source key | `BRAINSTORE_RESPONSE_CACHE_URI` | `BRAINSTORE_CODE_BUNDLE_URI` | > | - | - | - | - | > | AWS | `objectStorage.aws.responseBucket` / `objectStorage.aws.codeBundleBucket` | `s3:///brainstore-cache` | `s3://` | > | Azure | `objectStorage.azure.responseContainer` / `objectStorage.azure.codeBundleContainer` | `az:///brainstore-cache` | `az://` | > | GCP | `objectStorage.google.apiBucket` | `gs:///brainstore-cache` | `gs://` | > > If you previously configured these via `extraEnvVars`, remove those overrides after upgrading to v5.1.0 to avoid conflicts. ### Configure fast readers Fast readers are isolated Brainstore nodes dedicated to serving predictable UI queries (paginated viewers, span and trace lookups), preventing resource-intensive ad-hoc queries from making the UI unresponsive. * **GCP and Azure**: Fast readers are enabled by default starting in Helm chart v5.0.0. See the configuration reference below. * **AWS**: Fast readers are enabled by default (2 nodes) starting in Terraform module v5.5.0. On earlier module versions they are disabled by default. Set `brainstore_fast_reader_instance_count` in your Terraform configuration to control the node count, or set it to `0` to opt out (recommended for sandbox or non-production deployments). > [!WARNING] > > Upgrading to Helm chart v5.0.0 from an earlier version automatically creates 2 fast reader nodes. Plan for these additional nodes and explicitly configure each fast reader with the recommended 16 vCPUs and 64 GiB of memory. Verify that your cluster has capacity before upgrading. > [!WARNING] > > If you have custom `brainstore.readinessProbe` overrides pointing to `/status`, remove them before upgrading to Helm chart v5.0.0+. The `/status` readiness endpoint has a bug where it never recovers after a failure, which can permanently mark Brainstore nodes as not ready. Remove any `brainstore.readinessProbe` or `brainstore.fastreader.readinessProbe` customizations and rely on the chart defaults. **Configuration reference** Fast readers are configured under the `brainstore.fastreader` key in your `values.yaml`. If you have customized `brainstore.reader` settings, mirror those customizations to `brainstore.fastreader`. The memory values below match the [production defaults in Helm chart 6.26.0 and later](#set-resources). Set them explicitly on earlier versions. | Key | Value | Description | | - | - | - | | `brainstore.fastreader.name` | `brainstore-fastreader` | Name of the Deployment, Service, and ConfigMap | | `brainstore.fastreader.replicas` | `2` | Number of fast reader pod replicas | | `brainstore.fastreader.service.port` | `4000` | Service port | | `brainstore.fastreader.service.type` | `ClusterIP` | Kubernetes service type | | `brainstore.fastreader.resources.requests.cpu` | `16` | CPU request | | `brainstore.fastreader.resources.requests.memory` | `64Gi` | Recommended production memory request | | `brainstore.fastreader.resources.limits.cpu` | `16` | CPU limit | | `brainstore.fastreader.resources.limits.memory` | `64Gi` | Recommended production memory limit | | `brainstore.fastreader.objectStoreCacheMemoryLimit` | `1Gi` | Object store memory cache limit | | `brainstore.fastreader.objectStoreCacheFileSize` | `1000Gi` | Object store file cache size | | `brainstore.fastreader.cacheDir` | `/mnt/tmp/brainstore` | Local cache mount path | | `brainstore.fastreader.volume.size` | `""` | Ephemeral storage size; required for Azure ACS | | `brainstore.fastreader.extraEnvVars` | `[]` | Additional environment variables | | `brainstore.fastreader.nodeSelector` | `{}` | Node selector for scheduling | | `brainstore.fastreader.tolerations` | `[]` | Pod tolerations | | `brainstore.fastreader.affinity` | `{}` | Pod affinity rules | > [tab] Azure > > Azure users must explicitly set `brainstore.fastreader.volume.size` when using Azure Container Storage (`enableAzureContainerStorageDriver: true`): > > ```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} > brainstore: > fastreader: > volume: > size: "100Gi" > ``` > [tab] GCP Autopilot > > On GKE Autopilot, set `brainstore.fastreader.volume.size` to configure ephemeral storage requests. Match the resource profile of your standard reader nodes: > > ```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} > brainstore: > fastreader: > resources: > requests: > cpu: "16" > memory: "64Gi" > limits: > cpu: "16" > memory: "64Gi" > objectStoreCacheFileSize: "900Gi" > volume: > size: "1000Gi" > ``` ### Pace rollouts > [!NOTE] > > This section applies to GCP and Azure deployments using the Helm chart (6.17.1+). AWS deployments manage Brainstore rollout through Terraform and do not expose these settings. Brainstore readers, fast readers, and writers each have independently configurable rollout strategy and readiness dwell settings. These controls let you limit how many replacement pods start together and require replacements to stay ready for a minimum duration before a rollout continues. This can reduce simultaneous cache warm-up, object-storage, scheduling, and compaction pressure on cache-heavy or high-throughput deployments. The defaults preserve the rollout behavior from earlier chart versions and apply to all three Brainstore roles: | Key | Default | Description | | - | - | - | | `brainstore..strategy.type` | `RollingUpdate` | Rollout strategy for this Brainstore role's Deployment. | | `brainstore..strategy.rollingUpdate.maxSurge` | `100%` | Maximum number of extra pods created during a rollout. | | `brainstore..strategy.rollingUpdate.maxUnavailable` | `0` | Maximum pods that can be unavailable during a rollout. | | `brainstore..minReadySeconds` | `0` | Minimum seconds a replacement pod must stay ready before the rollout proceeds. | | `brainstore..progressDeadlineSeconds` | `600` | Maximum seconds the Deployment controller waits for progress. Must be greater than `minReadySeconds`. | Replace `` with `reader`, `fastreader`, or `writer`. Upgrading without overriding these values does not change rollout pacing. These settings do not change the pod template and do not start a rollout on their own. Kubernetes uses updated settings for an active rollout and future rollouts. For cache-heavy or high-throughput deployments, use a longer readiness dwell: ```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} brainstore: writer: minReadySeconds: 300 progressDeadlineSeconds: 900 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0 reader: minReadySeconds: 120 progressDeadlineSeconds: 600 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0 fastreader: minReadySeconds: 120 progressDeadlineSeconds: 600 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0 ``` `progressDeadlineSeconds` must be greater than `minReadySeconds`. The chart rejects an invalid pairing. Keep enough deadline margin for scheduling, image pulls, startup, and the readiness dwell. Longer readiness dwell reduces rollout pressure but does not confirm that a pod's local cache is fully warm. Monitor workload health until the rollout converges. `maxUnavailable: 0` requires enough capacity to schedule the configured surge. On managed clusters, even `maxSurge: 1` can require an additional node and local storage. Before upgrading, verify regional compute quota and the availability of the required node type and storage, and account for the temporary increase in compute and storage cost. > [!WARNING] > > Setting `strategy.type: Recreate` stops all pods in a Brainstore role before creating replacements. This causes a complete role outage, and for the writer (which defaults to a single replica) pauses background processing until the replacement becomes ready. > [!WARNING] > > Slow rollouts extend the period during which old and new Brainstore versions run together. Do not change `brainstoreWalFooterVersion` in the same upgrade as an image version bump, except where the documented [data plane 2.0 upgrade sequence](https://docs.braintrust.dev/admin/self-hosting/upgrade/v2.md) explicitly permits it. The chart does not create PodDisruptionBudgets for Brainstore, so these rollout controls do not limit voluntary disruptions such as node drains or protect against involuntary pod or node failures. ### Brainstore startup gate > [!NOTE] > > This section applies to GCP and Azure deployments using Helm chart 6.20.0+. On AWS, Terraform module v6.9.0 and later gate API updates on Brainstore during `terraform apply`. See [Review and apply](https://docs.braintrust.dev/admin/self-hosting/upgrade/routine.md#2-review-and-apply) in the AWS upgrade steps. A Helm upgrade updates the API and Brainstore Deployments at the same time, so new API pods can start serving traffic while Brainstore is still rolling out, against an older Brainstore release or with fewer Brainstore pods than it needs. The Brainstore startup gate prevents this by holding each new API pod until Brainstore can serve it. A `wait-for-brainstore` init container waits until every reader, writer, and fast reader pod runs a compatible release and each role has its desired number of available pods. Existing API pods keep serving traffic while new pods wait. The gate is enabled by default and applies to every API pool. A Brainstore release is compatible when it has the same major version as the API image and is the same version or newer. For example, an API on `v2.17.0` waits for Brainstore `v2.17.0` or a later 2.x release. | Key | Default | Description | | - | - | - | | `api.brainstoreStartupGate.enabled` | `true` | Set to `false` to disable the gate. | | `api.brainstoreStartupGate.minimumVersion` | `""` | Minimum compatible Brainstore release, such as `v2.17.0`. When empty, the gate uses each API pool's `api.image.tag`. | | `api.brainstoreStartupGate.timeoutSeconds` | `1200` | Seconds to wait before the init container fails. Kubernetes then retries it, so the pod stays stopped until Brainstore is compatible. Must be greater than `pollIntervalSeconds`. | | `api.brainstoreStartupGate.pollIntervalSeconds` | `15` | Seconds between checks. | | `api.brainstoreStartupGate.image` | `python:3.13-alpine` (pinned by digest) | Init container image. For air-gapped or mirrored registries, point this at a mirror of the same image. | When the gate is enabled, the chart fails to render if: * `api.image.tag` is not a release tag in `vX.Y.Z` format and `api.brainstoreStartupGate.minimumVersion` is not set. * An API pool doesn't use the `RollingUpdate` strategy with `maxUnavailable: 0`. * An API pool sets `maxSurge` to `0`. * `api.brainstoreStartupGate.timeoutSeconds` is less than or equal to `api.brainstoreStartupGate.pollIntervalSeconds`. > [!WARNING] > > The gate identifies Brainstore versions by image tag. If `brainstore.image.tag` is a commit SHA, a digest without a release tag, or a pre-release tag, the gate never passes and new API pods don't start. Set `api.brainstoreStartupGate.enabled: false` before you deploy such an image. If you slow Brainstore rollouts with [rollout controls](#pace-rollouts), raise `timeoutSeconds` so the gate outlasts a full Brainstore rollout. The chart creates a Role and RoleBinding that let the API service accounts get the Brainstore Deployments and list pods in the namespace. API pods set `automountServiceAccountToken: false`, and only the init container mounts a token, so a sidecar that needs Kubernetes API access must project its own. ## External attachments Use `ExternalAttachment` to reference files in S3 without uploading them to Braintrust. Before logging external attachments, [grant the API access to your attachment buckets](https://docs.braintrust.dev/admin/self-hosting/configure/security.md#grant-access-to-external-attachment-buckets). The following example logs a reference to a PDF in your bucket: ```typescript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} import { ExternalAttachment, initLogger } from "braintrust"; const logger = initLogger({ projectName: "My Project" }); logger.log({ input: { document: new ExternalAttachment({ url: "s3://my-bucket/path/to/file.pdf", filename: "file.pdf", contentType: "application/pdf", }), }, output: "Document processed", }); ``` ```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} from braintrust import ExternalAttachment, init_logger logger = init_logger(project="My Project") logger.log( input={ "document": ExternalAttachment( url="s3://my-bucket/path/to/file.pdf", filename="file.pdf", content_type="application/pdf", ), }, output="Document processed", ) ``` > [!WARNING] > > **Custom view PDF limitation:** PDF files referenced as `ExternalAttachment` objects cannot be rendered in custom views due to browser-level security restrictions in the sandboxed iframe environment. While images and videos work correctly, browsers block PDF rendering in this context. > > To display PDFs in custom views, upload them as standard `Attachment` objects or host them externally and link to them instead of attempting inline rendering. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.