Skip to main content
GoModel wraps every upstream provider call with two resilience layers:
  1. Retry with exponential backoff — repeats a failed request against the same provider with growing delays.
  2. Circuit breaker — short-circuits calls to a provider that has been failing repeatedly, then probes once the timeout elapses.
Retries apply to the selected target. Circuit breakers apply per provider by default, or per model with scope: model. They do not switch to a different model or provider on failure. For cross-model failover, see Failover.

​
Defaults

The defaults are tuned to be safe for most deployments. Override only what you need. To disable retries, set max_retries: 0 — every request gets exactly one attempt. To disable the circuit breaker, set circuit_breaker.enabled: false (or CIRCUIT_BREAKER_ENABLED=false); the thresholds are kept but never consulted, so flipping it back on restores your tuning. Setting failure_threshold: 0 also disables the breaker. Both switches work globally or per provider, and a per-provider enabled: true re-enables the breaker for that provider when it is off globally. Turn the breaker off when something in front of the provider already sheds load (a provider-side load balancer, a service mesh) or when a short burst of 5xx responses must never pause traffic to that provider. Keep it on otherwise: it is what stops a dead upstream from tying up every request for the full timeout.

​
What counts as a failure

Retries fire on transport errors (connection refused, resets, DNS failures) and, by default, on 429, 502, 503, 504, 522, and 524 responses. Other statuses — including 500 — are returned to the caller without retrying. The following paths retry less than the table suggests:
  • Local HTTP client timeouts return immediately without retrying. An expired overall request context also prevents further attempts.
  • Streaming requests are never retried once dispatched, because partial data may already have been sent.
  • Passthrough requests are retried only when they are replay-safe: GET, HEAD, OPTIONS, PUT, or any request carrying an Idempotency-Key header.
Idempotency keys reach the provider. When a client sends an Idempotency-Key header, GoModel forwards it unchanged on the provider call it makes for that request and on every retry of that call, so a provider that deduplicates by the key can recognize a replayed request. A failover attempt to another target sends a different request body, so it goes out without the key, and so do gateway-internal calls such as guardrail checks. Keys must be visible ASCII of at most 255 characters; any other value is ignored. Errors hidden behind HTTP 200 are unmasked before any of this applies. Some providers (OpenRouter is the best-known) answer 200 OK with a bare {"error": ...} JSON body. GoModel detects such payloads and handles them as the error they really are: a status embedded in error.code is preserved (so a hidden 429 behaves like a real one), anything else maps to 502. The mapped status then drives retries, the circuit breaker, and failover exactly like a genuine error status, and the response is never cached as a success. Passthrough endpoints are exempt: they forward the provider’s response byte-for-byte. Streams that stop early end with an explicit error instead of a silent EOF. When a provider stream closes or fails before its terminal event (data: [DONE] or a finish_reason for chat completions, a terminal response.* event for /v1/responses, message_stop for /v1/messages), GoModel relays everything received so far and then sends one error event in the endpoint’s format:
  • chat completions: data: {"error":{"code":"stream_incomplete",...}}
  • /v1/responses: an error event with "code":"stream_incomplete"
  • /v1/messages: an Anthropic error event with error.type set to api_error
This covers malformed chunks, a connection dropped in the middle of a tool call, and similar truncations, so a client never mistakes a partial answer for a complete one. A stream that already sent its terminal event stays complete even if the connection fails afterwards. The request log records a truncated stream as a stream_error. The circuit breaker counts transport errors and, by default, 429 and all 5xx responses as failures. One exhausted retry sequence counts as one failure; individual HTTP retries do not increment the breaker. A successful sequence records success. Client cancellation is ignored; local client timeouts count as failures. Set retry.retry_on_statuses and circuit_breaker.failure_on_statuses independently. Both accept exact codes and classes such as 5xx. Omitted lists use defaults; explicit lists replace them; [] disables status-based triggers while retaining transport-error handling. For example, failure_on_statuses: [5xx] excludes rate limits from breaker failures, including during recovery probes. Invalid codes or scopes fail configuration loading. The default now includes 429 breaker failures. Deployments that previously relied on rate limits leaving the breaker closed can set failure_on_statuses: [5xx] to retain that behavior. While the circuit is open, requests fail fast with a 503 and the message circuit breaker is open - provider <name> temporarily unavailable, where <name> is the configured provider name (openai-eu, not its type). After timeout elapses, a single probe request is let through while concurrent requests keep failing fast; success_threshold consecutive successful probes close the circuit, and a failure matching the breaker policy reopens it. Probes make only one HTTP attempt, without retries. The circuit breaker is in-memory and per gateway process: each provider gets its own breaker by default. With scope: model, each model within a provider instance gets an independent breaker; requests without a model, such as discovery, use a separate provider breaker. Multipart audio uploads use their explicit model names. Each provider retains at most 1,024 model breakers. Idle closed entries expire after 10 minutes when a new model is looked up; the oldest idle closed entry is evicted sooner when the limit is reached. Active and open breakers are preserved. If every slot is protected, a new model receives a failover-eligible 503 until a slot becomes available. State resets on restart and nothing is shared between replicas.

​
Trip rules

trip_on rules open the circuit breaker instantly when an upstream error message matches a Go regular expression — without counting failures toward failure_threshold. Use them for errors that mean “stop calling this provider”, such as exhausted quota or a disabled billing key, where retrying only burns requests.
  • match is a Go regexp tested against the provider’s error message and error code.
  • ttl is how long the breaker stays open. Omit it to reuse timeout (default 30s).
  • The group name only matters for env overrides; resolution evaluates rules in group-name order.
  • Rules are inert until configured: without them the breaker behaves exactly as described above. trip_on: {} disables them explicitly.
  • A provider’s trip_on map replaces the global map entirely; omit the key to inherit the global rules.

​
Environment variables

The same rules are settable without a config file — one MATCH (plus optional TTL) variable per named group:
Env groups override config rules by name only (case-insensitive): a group named like an existing config rule replaces it; config rules with other names survive; env-only names append. A malformed group (missing MATCH, invalid duration) fails config load, matching YAML behavior.
  • Trip rules match on translated (non-streaming) and streaming errors; raw passthrough routes rely on status-based breaker behavior instead.
  • An empty match, a pattern that does not compile, or a negative ttl fails configuration loading with a circuit_breaker.trip_on[i] error.
Override per provider like any other breaker setting:
A tripped breaker behaves like an ordinary open one: requests fail fast with a 503 before reaching the provider, failover virtual models route around it, and after ttl the half-open probe decides whether traffic resumes. If the probe sees the same error, the trip window restarts.

​
Resetting a tripped breaker

Once the quota or key is fixed, you do not have to wait out the ttl. Use the Reset breaker button on the provider’s dashboard card (enabled while the breaker is open or half-open), or call the admin API:
204 resumes traffic immediately. 404 means the provider name is unknown, 400 that the provider’s adapter cannot reset its breaker, and 503 that the reset hook is unavailable on the gateway.

​
Why did my provider pause?

A provider card showing Circuit Open while requests fail with 503 before reaching the provider means the breaker is open — either failure_threshold failures accumulated or a trip_on rule matched. GET /admin/providers/status reports the live state per provider as circuit_state (closed, open, or half-open; empty until the provider has served traffic) alongside the effective resilience.circuit_breaker settings, including trip_on. Providers configured in the dashboard can edit their trip rules directly in the credential editor (“Quota breaker”). Dashboard-managed providers inherit the global rules unless they define their own list; clearing all rule rows restores the inherited list (yaml’s explicit trip_on: [] to disable has no dashboard-managed equivalent). Providers declared in config.yaml or environment variables show their rules read-only there; change them in the YAML file. Trip rules have no environment-variable equivalent.

​
Environment Variables

These set the global defaults that apply to every provider unless overridden in YAML.

​
Retry

​
Circuit Breaker

​
YAML

The same fields are available under the global resilience: block, and can be overridden per provider:
Only fields explicitly listed under a provider’s resilience: block are overridden. Everything else inherits from the global section, which in turn inherits from the built-in defaults.
Per-provider tuning must come from YAML. Environment variables set global defaults only — RETRY_MAX_RETRIES cannot target a single provider. See config.yaml gotchas.

​
Worked example

Given the YAML above, the effective per-provider settings are: anthropic, ollama, and vllm inherit every field they did not explicitly override. With the breaker disabled, vllm also reports no breaker state on the dashboard or in the gomodel_circuit_breaker_state metric.

​
Circuit breaker and dashboard provider health

The dashboard’s provider status combines two independent signals:
  • Model discovery — whether the provider’s model inventory could last be fetched (details below).
  • Request health — a 10-minute sliding window of real request outcomes per provider and model, including the live circuit breaker state.
The circuit breaker’s state is shown on each provider card: expanding a card reveals a Breaker State chip, an open breaker turns the provider’s status pill to Circuit Open, and a half-open breaker shows Recovering. With scope: model, the provider-level breaker display and metric reflect the breaker of the most recently completed request, rather than an aggregate of all model breakers. A model whose recent requests keep failing (at least 3 errors making up half or more of its windowed requests) marks the provider Degraded even while model discovery still succeeds — this catches upstreams that list models fine but fail real calls, e.g. with misreported 4xx errors that deliberately never trip the breaker. Request-health signals only ever worsen the discovery-based status, never improve it, and the tracking is in-memory per gateway process. The breaker still recovers on its own within timeout (default 30s) once the provider is back; when metrics are enabled its state is also exported as the gomodel_circuit_breaker_state gauge (0 = closed, 1 = half-open, 2 = open). Model discovery is re-checked:
  • at startup,
  • on every model registry refresh, controlled by CACHE_REFRESH_INTERVAL (seconds, default 3600 — hourly),
  • on the fast recheck loop, which re-probes only providers whose latest refresh failed, controlled by PROVIDER_RECHECK_INTERVAL (cache.model.recheck_interval in config.yaml; seconds, default 60; 0 disables), and
  • on demand, when a request asks for a provider-qualified model (provider/model) that is missing from the registry.

​
What happens while a provider is down

When a provider’s refresh fails, its previously discovered models are marked stale, and the dashboard shows the provider as Offline:
  • Model listings (GET /v1/models and the dashboard model list) hide the provider’s models until it recovers, so clients are not offered models that cannot currently be served.
  • Direct requests to its models (provider/model) still resolve and are sent to the provider, so callers get an honest 502/503 (and a shadowing virtual model’s failover chain can fire) instead of a misleading “model not found”.
  • Virtual-model redirects skip the provider’s targets, so a load-balanced redirect keeps working through its healthy targets.
Exception: when every provider is failing at once (for example, a control-plane-only outage where model discovery is unreachable but inference still works), the previous inventory is kept as-is and stays listed — hiding everything would only turn provider errors into “model not found”. The fast recheck loop re-probes the provider every PROVIDER_RECHECK_INTERVAL seconds, updating “Last checked” and restoring normal routing typically within a minute of the provider coming back. A provider that was already down at startup (nothing discovered yet) shows as Unhealthy until its first successful fetch.

​
Failover vs. Resilience

Retries stay on the selected target; the circuit breaker tracks the configured provider or model scope. If you also want GoModel to try a different model or provider when the primary keeps failing, give the model a virtual model with more than one target — the remaining targets are its failover chain. See Failover.

​
Retry Cloudflare timeouts, then switch models

For non-streaming translated requests, combine retries with a failover virtual model. Merge the following settings into your existing configuration, using your configured provider and model names:
Send model: resilient-chat. While model1’s breaker is closed, a 524 response triggers up to two retries against model1. If all three attempts fail, GoModel tries model2 with its own retry budget. Model1’s open breaker skips its upstream calls without blocking model2. With the default scope: provider, both models share a breaker instead. The caller’s deadline must allow time for the attempts and backoff. Streaming requests do not get this retry sequence; a partially delivered response cannot be restarted transparently.
Last modified on September 19, 2026