Runtime settings
Changing these parameters in the management UI takes effect immediately and does not require a restart. This is different from environment variables—which need a restart after changes.
Two-layer settings §
The same parameters exist at two levels:
- System Level — Configured on the Settings page as the global default.
- Group-level — Configured on the Group's Settings tab; Filling it will override the system value; leaving it blank will inherit.
This design is because the characteristics of different upstream services vary greatly: some providers respond slowly but steadily, while others are fast but occasionally unstable. Using the same timeout and retry parameters is not suitable.
When there is no individual configuration in the Group, the value here is used.
The Group's Settings tab marks values inherited from the system and values overridden for this Group. Check this first when one Group behaves differently.
Three types of timeouts §
The three timeouts cover different stages of a request. Match the timeout to the symptom:
| Parameters | Manage what | Symptoms to watch |
|---|---|---|
| First byte timeout | From sending the request to receiving the first byte. | Upstream queue is long, request hasn't started responding and is judged failed |
| Request timeout | The total maximum duration of the entire request | Long output tasks get interrupted midway |
| Stream idle timeout | Maximum interval between two data blocks in a streaming response | Streaming output freezes for a long time, but the connection is not broken |
Special attention needed for first-byte timeout of inference models — They may spend a long time reasoning before output starts; a tight default can misclassify them as failed.
Retry and blacklist §
- Retry count — Maximum credential switches after a failure. More retries may improve success rate but lengthen failed requests.
- Blacklist threshold — Consecutive failures before automatic blacklisting. A lower value isolates bad credentials faster but can react to transient failures.
See How scheduling works for the complete behavior of both settings.
Session affinity §
When enabled, the gateway derives a soft-affinity key from the AccessKey, client protocol, and the request instructions or the prefix of the first user input. Requests with the same stable prefix prefer the same credential. It has three settings:
- Switch — Enables or disables the feature.
- TTL — How long to retain an affinity record.
- Capacity — Maximum remembered sessions (10,000 by default).
The affinity mechanism does not read previous_response_id, conversation, or any other upstream resource ID. It helps ordinary requests with the same prompt prefix reuse a credential, but it cannot guarantee that a stateful resource returns to the credential that created it. Use one credential for these resources, or confirm that the upstream supports sharing them across credentials.
Log retention §
Request log retention days controls log retention. The default is 7 days, after which logs are deleted automatically.
Increase it to inspect older history, but the database will keep growing — pay attention to disk usage for high request volumes. Take extra care with SQLite.
Other settings §
- Check Interval — How often to validate credential availability.
- Model pricing automatic sync — whether to sync prices from a public data source; see Model management. Note that this option can be taken over by the environment variable
MODELS_DEV_AUTO_SYNC_ENABLED— once that variable is set, this becomes read-only; see Environment variables - Request header rules and Usage options injection — For non-standard scenarios; see Proxy and request headers.
When to adjust §
The default value applies to most situations; don’t change it if there are no clear symptoms. Blindly increasing timeouts and retries only makes failures take longer.
Match the setting to the symptom:
- Inference models frequently timeout → Increase time to first byte, preferably only for that Group.
- Long text tasks are truncated → Increase the request timeout.
- Streaming output disconnects midway → Increase the stream idle timeout.
- Good credentials get blacklisted frequently → Increase the blacklisting threshold.
- Isolating bad credentials is too slow → Decrease the blacklisting threshold.
- Stateful request reports context not found → Use one credential, or confirm that the upstream supports sharing resources across credentials
Confirm the symptom in Monitoring and troubleshooting, only change the item related to the symptom, and observe the result before changing another setting.