Runtime settings
These settings take effect as soon as you save them in the admin console, without a restart. Environment variables are separate and require a restart after changes.
Two-layer settings §
Timeouts, blacklisting thresholds, and similar runtime parameters support two settings layers:
- System Level — Configured on the Settings page as the global default.
- Group-level — Configured on the Group's Settings tab; Enable an override to edit it, or restore defaults to inherit the system value; changes take effect after saving.
Upstreams behave differently: some are slow but stable, while others are fast but occasionally unreliable. One timeout and blacklisting threshold does not suit them all.
Settings that support group overrides inherit the system value when no override is enabled.
The Group's Settings tab marks values inherited from the system and values overridden for this Group. Check this first when one Group behaves differently.
Three types of timeouts §
The three timeouts cover different stages of a request. Match the timeout to the symptom:
| Parameters | Manage what | Symptoms to watch |
|---|---|---|
| First byte timeout | From sending the request to receiving the first byte. | Upstream queue is long, request hasn't started responding and is judged failed |
| Per-attempt upstream timeout | Time limit for each upstream attempt, not the total duration across retries | Long output tasks get interrupted midway |
| Stream idle timeout | Maximum interval between two data blocks in a streaming response | Streaming output freezes for a long time, but the connection is not broken |
Special attention needed for first-byte timeout of inference models — They may spend a long time reasoning before output starts; a tight default can misclassify them as failed.
Concurrency limits §
Under Settings → Routing, set the global, default access-key, and default Group concurrency limits. All default to 0, meaning unlimited. These limit active requests, not requests per minute.
Access keys can override their default, or leave the field blank to inherit it. Enable an override in Group runtime settings for a Group-specific limit. An explicit 0 removes only that layer’s limit; other limits still apply.
At capacity, new requests are rejected immediately without queuing; HTTP requests return 429 with concurrency_limit_exceeded. Streaming requests occupy a slot while active. Lowering a limit does not forcibly interrupt requests already admitted.
Concurrency counts apply only within the current GPT-Load process, not across multiple instances. JEV decision calls also obey Group concurrency limits; guardrail fail-open behavior does not bypass concurrency admission.
Retry and blacklist §
- Retry count — a global additional-retry budget, without Group overrides. Increasing it may improve success rates but also delay failing requests.
- Blacklist threshold — Consecutive failures before automatic blacklisting. A lower value isolates bad credentials faster but can react to transient failures.
See How scheduling works for the complete behavior of both settings.
Empty response retry is off by default and can be overridden per group. When enabled, a streaming conversation tries the next candidate if upstream ends naturally without output before anything is delivered to the client. After the budget is exhausted, the original empty response is delivered. It excludes WebSocket, non-streaming, prewarm, and historical-resource-reference requests, as well as empty results with an explicit reason such as refusal, content filtering, or output-budget exhaustion.
Retried empty responses may already be billable upstream, but the gateway records usage and cost only for the attempt ultimately delivered. Accept this estimation limit before enabling the feature.
Upstream error rules §
Add global rules under Settings → Routing → Error handling rules. Group runtime settings can replace the entire global list. Without an override, the group inherits global rules; an empty override uses only built-in handling. Unmatched errors also use the built-in policy. Both Modern and Classic provide these controls.
Rules are matched from top to bottom; the first matching rule applies. Status codes must be integers from 200 to 599, and any listed code can match. Keywords match substrings in recognized error codes, types, or messages without case sensitivity; any listed keyword can match. If both condition types are specified, both must match. Specify at least one condition type.
Each rule independently selects whether to retry and one action: keep state, cool down the current model for this credential, cool down the entire credential, record failures toward blacklisting, or skip the group for this request. Cooldown durations are positive integer seconds. Blacklisting still uses the failure threshold; model cooldown applies only to operations that support it.
Changes take effect after saving. A list can contain at most 100 rules and 65,535 encoded bytes. Rules use only recognized upstream errors, including errors already recognized under HTTP 200; they do not scan normal answer text. Credential probing and recovery are unchanged.
Rules preserve the protections for client request errors, authentication recovery, and request-scoped rate limits. They cannot force retries after output has reached the client or when execution results are unknown. Retries still share the global budget.
Routing strategy §
Choose the global routing strategy under Settings → Routing. Native First is the default and prefers native routes within the same priority tier. Weighted Mix lets native and converted candidates in that tier compete by effective weight. This setting does not support group overrides. Traffic still depends on credential availability, affinity, and protocol capabilities.
Responses WebSocket is enabled by default; Groups may inherit or override it. Disabling it interrupts affected connections without changing HTTP/SSE. A global disable does not override a Group explicitly enabled.
Session affinity §
When enabled, the gateway derives a soft-affinity key from the AccessKey, client protocol, and the request instructions or the prefix of the first user input. Requests with the same stable prefix prefer the same credential among candidates allowed by the current priority tier and routing strategy. It has three settings:
- Switch — Enables or disables the feature.
- TTL — How long to retain an affinity record.
- Capacity — Maximum remembered sessions (10,000 by default).
Soft affinity only prefers credential reuse; it does not guarantee resource ownership. Responses previous_response_id uses an independent continuation binding unaffected by the affinity toggle. This guarantee does not cover conversation or other resource references.
Log retention §
Request log retention days controls log retention. The default is 7 days, after which logs are deleted automatically.
Increase it to inspect older history, but the database will keep growing — pay attention to disk usage for high request volumes. Take extra care with SQLite.
Original request bodies enabled per access key use the same retention period and are cleaned up with their logs; there is no separate global recording switch. Disabling recording for a key affects future requests only and does not delete saved content. Recording increases database and backup size, so limit its scope and duration to troubleshooting needs.
Other settings §
Select the global upstream proxy in the Upstream proxy field on the Settings page. Add, bulk-import, edit, and test proxy addresses on the Proxies sidebar page for reuse by global, group, and credential settings. Tests access the test URL from the server through the selected proxy and do not automatically enable or disable it.
Request redaction has its own settings section. Automatic model selection and JEV AI Guardrails are under Experimental features. Voice mode is under Connections and timeouts and can be overridden per Codex group. See the detailed guides:
- Request redaction: fixed replacement and reversible encryption
- JEV AI Guardrails: rules, warnings, and blocking
- Connect Codex live voice
- Configure automatic model selection
- Check Interval — How often to validate credential availability.
- Model pricing automatic sync — whether to sync prices from a public data source; see Model management. Note that this option can be taken over by the environment variable
MODELS_DEV_AUTO_SYNC_ENABLED— once that variable is set, this becomes read-only; see Environment variables - Upstream request headers, downstream response headers, and CORS — for special requirements; see Proxy and request headers.
When to adjust §
The default value applies to most situations; don’t change it if there are no clear symptoms. Blindly increasing timeouts and retries only makes failures take longer.
Match the setting to the symptom:
- Inference models frequently timeout → Increase time to first byte, preferably only for that Group.
- Long text tasks are truncated → Increase the request timeout.
- Streaming output disconnects midway → Increase the stream idle timeout.
- Good credentials get blacklisted frequently → Increase the blacklisting threshold.
- Isolating bad credentials is too slow → Decrease the blacklisting threshold.
- Stateful request reports context not found → check whether the response ID is registered and its original credential is available. Other resource references need a single credential or upstream sharing.
Confirm the symptom in Monitoring and troubleshooting, only change the item related to the symptom, and observe the result before changing another setting.