How scheduling works
This page explains the internal mechanisms. You can use it normally without understanding, but it is very useful when troubleshooting 'why this credential was used.'
The complete path of a single request §
After the request comes in, the gateway performs these steps sequentially:
- Authentication — Verify that the AccessKey is valid and enabled.
- Protocol Check — Verify that the AccessKey allows the current protocol.
- Filter groups — find groups authorized by the access key that can serve the requested model.
- Select Credential — first select a tier by group priority, then choose an available credential using the routing strategy and weights.
- Forward — Convert protocols when needed, then send the request upstream.
- Retry if failed — Switch credentials until the request succeeds or retries are exhausted.
Route inspection explains why an AccessKey, Group, or credential cannot be a candidate. Its reason_code differs from client-response code and request-log error_code; see the final section.
Candidate groups §
Candidate Groups must meet all of the following:
- Within the authorization scope of this AccessKey
- Enabled
- The requested model is exposed
- Effective weight is greater than 0
Group priority defaults to 0 and accepts negative values. Higher values take precedence. Among eligible groups with available credentials, the highest priority tier is selected first. If a higher tier is unavailable, a lower available tier is used directly.
Available credentials §
Then filter the credentials in these groups:
- Status is available (not disabled, cooling down, or blacklisted)
- Subscription accounts also need to have normal authorization status
- Effective weight is greater than 0
Within a priority tier, the gateway first filters by routing strategy, then uses weighted round-robin with group weight × credential weight. Soft affinity reuses only candidates allowed in that tier. Actual traffic is not guaranteed to match weight ratios exactly.
Weight §
Group and credential weights both default to 50 and can be set to 1–100. Among candidates in the same tier, higher weights receive a larger relative traffic share. Weights are no longer calculated automatically from success rates.
The management UI and management API do not accept a manual weight of 0. To stop traffic temporarily, disable the Group or credential; its configuration and historical statistics remain available.
Session affinity §
When enabled, the gateway derives a soft-affinity key from the AccessKey, client protocol, and the request instructions or the prefix of the first user input. Later requests with the same stable prefix prefer the previously successful credential among candidates allowed by the current priority tier and routing strategy.
Chat Completions and Responses can also use a valid prompt_cache_key for soft affinity. An affinity hit does not guarantee an upstream cache hit. Responses ID continuation is independent of soft affinity; see the protocol page for its boundaries.
View Responses continuation boundaries →
Affinity records have a TTL and capacity limit (10,000 by default) and evict the oldest entries. Affinity is not absolute: if the remembered credential is cooling down or blacklisted, the gateway selects another available credential.
See Runtime settings for configuration.
After failure §
When it is safe to retry a different target, the gateway moves to the next lower available tier before exhausting credentials in the current tier. If no lower tier is available, it tries other credentials in the current tier. Once it moves down, that request does not return to higher tiers. Switching groups or tiers still shares the global retry budget.
The following describes the built-in defaults. For custom handling, see upstream error rules:
- Retry — Upstream rate limits, server errors, and network timeouts are issues that may be resolved by changing credentials.
- Does not retry — explicit parameter errors, context limits, or content-policy rejection. Changing credentials does not help.
Explicit model or capability rejection may allow another candidate within replay safety boundaries. Do not blindly replay after output or when execution is uncertain. Response ID continuation never switches credentials.
Once a streaming response is committed to the client, candidates cannot be switched. With empty response retry enabled, the gateway may briefly buffer leading events without output, so receiving an upstream chunk does not necessarily mean it has been sent downstream. See Runtime settings for scope and billing limits.
Cooldown and blacklist §
The two-level protection mechanism prevents bad credentials from repeatedly slowing down requests:
- Cooldown — A credential is temporarily skipped after an error, then restored automatically when cooldown expires. This is normal after upstream rate limiting.
- Blacklist — removed after consecutive failures exceed the threshold. API keys may recover automatically after successful validation; subscription credentials require manual handling.
Model-level rate limits cool down only that model on the credential; other models remain eligible. Explicit credential-wide restrictions still affect the whole credential. Restoring a credential clears its model cooldowns.
Subscription quota recovery can release built-in quota cooldowns early; successfully redeeming a quota reset also triggers recovery. Historical states without a recorded cooldown cause keep their original deadlines and can be cleared with an explicit credential restoration.
Cooldown is temporary avoidance, assuming the problem is temporary. Blacklisting means the credential is considered broken, such as when a key is revoked or an account is unpaid.
A successful attempt resets the consecutive failure count. Transient failures therefore do not accumulate toward blacklisting.
Configure thresholds in Runtime settings and inspect current status on the Health tab in Monitoring and troubleshooting.
Reason codes for routing failures §
Route inspection returns layered reason_code values for the AccessKey, Groups, and credentials. The error reference is the single catalog for reasons, levels, and remediation.