Gateway request lifecycle

What the lifecycle describes

The gateway request lifecycle gives operators a shared map of what happens to an AI request. An application may submit a request with credentials, model preferences, tools, and a deadline. The gateway authenticates it, checks whether it is allowed, applies routing policy, and sends a provider-specific representation to an eligible endpoint. It may then normalize the response, record usage, and return a result to the caller.

Example

Suppose a coding assistant sends a tool-calling request. Admission checks the tenant and budget, capability checks identify compatible destinations, and routing chooses a model and provider. The provider returns a streamed response, but a timeout interrupts it. The gateway records the interruption, applies the configured recovery rule, and either completes the request through an alternate path or reports a failure. A lifecycle record lets the team separate policy delay, provider delay, retry work, and application processing.

What to measure

Capture stage timestamps, policy versions, selected destinations, provider errors, retry counts, input and output tokens, and the final task outcome. Use correlation identifiers to join gateway events without storing unnecessary prompt content. Stage timing helps reveal whether an apparent model-latency problem is actually queueing or policy evaluation.

Limitations

The lifecycle is an operational model, not proof that each stage is correct. Different gateways combine or name stages differently, and some provider behavior remains outside the gateway's visibility. Validate the map against implementation logs and representative requests before using it for cost or reliability decisions.