How scheduling works
This page explains the internal mechanisms. You can use it normally without understanding, but it is very useful when troubleshooting 'why this credential was used.'
The complete path of a single request §
After the request comes in, the gateway performs these steps sequentially:
- Authentication — Verify that the AccessKey is valid and enabled.
- Protocol Check — Verify that the AccessKey allows the current protocol.
- Select Group — Find authorized Groups that expose the requested model.
- Select Credential — Choose an available credential from the Group's pool.
- Forward — Convert protocols when needed, then send the request upstream.
- Retry if failed — Switch credentials until the request succeeds or retries are exhausted.
Route inspection explains why an AccessKey, Group, or credential cannot be a candidate. Its reason_code differs from client-response code and request-log error_code; see the final section.
Select the Group first §
Candidate Groups must meet all of the following:
- Within the authorization scope of this AccessKey
- Enabled
- The requested model is exposed
- Effective weight is greater than 0
When multiple Groups qualify, choose by weight. This is how multiple sources for the same model provide automatic failover: another Group can serve the model when one Group has no working credentials.
Select a credential §
After selecting a Group, filter in its credential pool:
- Status is available (not disabled, cooling down, or blacklisted)
- Subscription accounts also need to have normal authorization status
- Effective weight is greater than 0
Choose randomly from the remaining candidates by weight. This is not simple polling: round-robin can repeatedly select a failing credential, while weighted selection plus cooldown is more resilient.
Weight §
Weight determines relative traffic share. Both Groups and credentials support automatic and manual modes:
- Group automatic — uses the default weight
- Credential automatic — calculated dynamically from recent successes and failures
- Manual — ranges from 1 to 100; higher values receive proportionally more traffic
The management UI and management API do not accept a manual weight of 0. To stop traffic temporarily, disable the Group or credential; its configuration and historical statistics remain available.
Session affinity §
When enabled, the gateway derives a soft-affinity key from the AccessKey, client protocol, and the request instructions or the prefix of the first user input. Later requests with the same stable prefix prefer the credential that previously succeeded.
The affinity mechanism does not read previous_response_id, conversation, or any other upstream resource ID, and it does not guarantee that a stateful resource returns to its original credential. For reliable stateful-resource access, keep only one credential in the Group or confirm that the upstream permits resource sharing across credentials.
Affinity records have a TTL and capacity limit (10,000 by default) and evict the oldest entries. Affinity is not absolute: if the remembered credential is cooling down or blacklisted, the gateway selects another available credential.
See Runtime settings for configuration.
After failure §
When a request fails, the gateway retries with a different credential until it succeeds or reaches the retry limit.
The key point is "what counts as a failure":
- Retry — Upstream rate limits, server errors, and network timeouts are issues that may be resolved by changing credentials.
- Does not retry — Request errors such as invalid parameters or a missing model fail on every credential, so retrying only wastes time.
Once streaming output starts, the request will not be retried: the client already has partial content, so replaying on another credential would corrupt the response. The gateway switches safely only before the first data block arrives; later errors are returned as-is.
Cooldown and blacklist §
The two-level protection mechanism prevents bad credentials from repeatedly slowing down requests:
- Cooldown — A credential is temporarily skipped after an error, then restored automatically when cooldown expires. This is normal after upstream rate limiting.
- Blacklist — Automatically removed after consecutive failures exceed the threshold. It requires manual verification and does not recover automatically.
Cooldown is temporary avoidance, assuming the problem is temporary. Blacklisting means the credential is considered broken, such as when a key is revoked or an account is unpaid.
A successful attempt resets the consecutive failure count. Transient failures therefore do not accumulate toward blacklisting.
Configure thresholds in Runtime settings and inspect current status on the Health tab in Monitoring and troubleshooting.
Reason codes for routing failures §
Route inspection returns layered reason_code values for the AccessKey, Groups, and credentials. The error reference is the single catalog for reasons, levels, and remediation.