Intelligent Key Management

GPT-Load employs intelligent key management mechanisms through polling load balancing, automatic failure detection, and recovery mechanisms to ensure high service availability and stability.

Polling Mechanism

Objective

Achieve fair load balancing for requests among multiple keys within a group.

Implementation

Atomic Counter

Ensures polling consistency and fairness under high concurrency

Status Filtering

Polling scope limited to keys with 'valid' status within the group

Retry & Failure Handling

Trigger Scenarios

Automatically triggered when request fails (HTTP Status ≥ 400 or network errors)

Processing Flow

1
Mark Failure

Increment failure count for the current failed key

2
Get New Key

Immediately obtain the next available key within the group through polling

3
Transparent Retry

Retry request with new key, transparent to client

4
Retry Limit

Configured maximum retry attempts. If all attempts fail, return final error to client

Key Blacklisting & Recovery

Blacklisting Mechanism

Trigger Condition

When a key's cumulative failure count reaches the blacklist threshold

Actions Taken

Key status updated to 'invalid'
Removed from polling queue, no longer receives new requests

Recovery Mechanism

Trigger Timing

Background scheduled task executed at key validation intervals

Recovery Process

1

Health Check

Use blacklisted key to request validation endpoint (e.g., models list)

2

Validation Success

• Key status restored to 'valid'

• Failure count reset to 0

• Re-added to polling queue

3

Validation Failure

Maintain 'invalid' status, wait for next check cycle

Mechanism Advantages

Load Balancing

Atomic counter ensures fair polling under high concurrency

Auto Recovery

Scheduled health checks and automatic recovery of failed keys

Fault Isolation

Quickly identify failed keys to ensure service stability