Routing Configs
A routing config is a named, reusable rulebook that decides which API key — and therefore which model and provider — serves each request, and what to do when something goes wrong. It lets you load-balance traffic, fail over to a backup when a provider has an outage, and send different requests to different models based on who’s asking — all without changing a line of application code. You build it once, reference it by a short slug, and change the behaviour anytime from the dashboard.
- Reliability. If OpenAI has a bad hour, automatically fail over to Anthropic (or a second key) so customers never see an error.
- Cost & scale. Spread traffic across several keys or providers by weight — blend a cheap model for the bulk and a premium model for a slice, or stay under each vendor’s rate limit.
- Personalised routing. Send your paying customers to a top-tier model and free users to a cheaper one, route EU traffic to an EU region, or pick a model based on the request — automatically.
- Zero redeploys. Swap the underlying key or model, change weights, or add a fallback from the dashboard. The application keeps sending the same one-line header.
Attach a config to any request by sending its slug in the x-config: <slug> header (e.g. my-router-3f9c1b2a) together with x-organization: <org>. It applies to /chat/completions and /v1/prompts/{slug}/completions.
- The config selects a target using its strategy, then uses that target’s virtual key to supply the provider and credentials for the call. You do not send a virtual key or provider header yourself — the config is the credential.
- A target can override request settings (the model, temperature, token limit, and more) so one config can pin a specific model per branch regardless of what the app asked for.
- On a failure, a FALLBACK config moves to the next target; a per-target retry can re-try the same target first. The caller gets the first successful response, or the last error if everything is exhausted.
- The response tells you what happened on its headers: x-virtual-key-slug (which key served it) and x-cache-status.
The four strategies
Every config — and every branch inside it — has one strategy that decides how it picks a target. Start here to choose the right shape.
| Strategy | What it does | Reach for it when… |
|---|---|---|
| SINGLE | Always routes to one target. A stable alias in front of a key. | you want to swap the underlying key/model later without touching the app. |
| LOADBALANCE | Spreads requests across targets at random, in proportion to each target’s weight. | you’re sharing load across keys/providers, staying under rate limits, or blending models. |
| FALLBACK | Tries targets in order; on a configured error it moves to the next one. | you need high availability and a backup provider for outages. |
| CONDITIONAL | Picks a target by matching rules against the request (who’s asking, what they asked for). | different users or request types should go to different models. |
The simplest config: every request goes to the one key you point it at. Its value is indirection — your application references a stable config slug, and you can change which key or model sits behind it from the dashboard at any time. It’s also the natural place to apply a permanent override (for example, force a specific model for everyone using this config).
Each request is sent to one target, chosen at random but biased by weight. Weights are relative proportions, not percentages — they don’t need to add up to anything. A target with weight 3 next to one with weight 1 receives roughly three-quarters of the traffic; 2 and 2 is an even split. Because the split is per-request, a steady stream of traffic settles close to those ratios.
{
"strategy_mode": "LOADBALANCE",
"targets": [
{ "name": "primary", "virtual_key_slug": "vk-1a2b3c4d5e6f7a8b", "weight": 3 },
{ "name": "secondary", "virtual_key_slug": "vk-c7d4e1f2a3b95a8b", "weight": 1 }
]
}Targets are tried top to bottom. The first target handles the request; only if it fails with an error you’ve marked as “try the next one” does the config move on to the second, then the third, and so on. The first success is returned to the caller. If every target fails, the last error is returned. This is your insurance policy against a provider outage or a key hitting its limit.
You decide which errors count as “move on” with on_status_codes — a list of HTTP error codes (see the reference below). A successful response, or an error not in that list, stops the chain and is returned as-is.
{
"strategy_mode": "FALLBACK",
"on_status_codes": [429, 500, 502, 503, 504],
"targets": [
{ "name": "openai", "virtual_key_slug": "vk-1a2b3c4d5e6f7a8b" },
{ "name": "anthropic", "virtual_key_slug": "vk-c7d4e1f2a3b95a8b",
"override_params": { "model": "claude-haiku-4-5" } }
]
}A list of rules is checked in order against attributes of the request. The first rule that matches sends the request to its target; if none match, a default target catches everything else. Rules can read two things: metadata you attach to the request (like a customer tier or region) and request parameters (like the model name or token limit). This is how you give premium users a better model, keep one region’s traffic on a regional provider, or send big requests to a higher-capacity model — automatically.
Metadata is supplied per request on the x-metadata header as a flat JSON object, e.g. x-metadata: {"tier":"premium","region":"us"}. The full list of operators and field paths is in the conditional reference.
{
"strategy_mode": "CONDITIONAL",
"conditional_default_target_name": "standard",
"targets": [
{ "name": "premium", "virtual_key_slug": "vk-1a2b3c4d5e6f7a8b",
"override_params": { "model": "gpt-4-turbo" } },
{ "name": "standard", "virtual_key_slug": "vk-c7d4e1f2a3b95a8b",
"override_params": { "model": "gpt-4o-mini" } }
],
"conditions": [
{ "condition_order": 0,
"query": { "metadata.tier": { "$eq": "premium" } },
"then_target_name": "premium" }
]
}Targets — the building blocks
Every config is a tree of targets. A target is one of two things:
- A leaf — it points at a virtual_key_slug. This is where a request actually goes: the key’s provider and credentials make the call.
- A branch — it has its own strategy_mode and a nested list of targets. This lets you compose strategies: a FALLBACK whose first target is itself a LOADBALANCE, for example.
A target is one or the other — it can’t both point at a key and have a nested strategy. Branches can nest up to 3 levels deep, which is plenty for patterns like “load-balance two primary keys, and if the whole group is down, fall back to a third provider.” Every target should have a short name — conditions and default targets refer to targets by name.
{
"strategy_mode": "FALLBACK",
"targets": [
{
"name": "primary-pool",
"strategy_mode": "LOADBALANCE",
"targets": [
{ "name": "key-a", "virtual_key_slug": "vk-1a2b3c4d5e6f7a8b", "weight": 1 },
{ "name": "key-b", "virtual_key_slug": "vk-c7d4e1f2a3b95a8b", "weight": 1 }
]
},
{ "name": "backup", "virtual_key_slug": "vk-b1c2d3e4f5a67890" }
]
}Per-target options
Beyond pointing at a key, each target can carry extra settings. All are optional. Settings marked * are required for that feature.
| Field | What it does | Accepted values | Default |
|---|---|---|---|
| name | A short label for the target. Conditions and the default target refer to targets by this name, so give every target one. | any short text | auto-generated |
| virtual_key_slug | The key this leaf routes to (provider + credentials). A leaf must have this; a branch must not. | a vk-… slug | — (required for a leaf) |
| strategy_mode | Makes this target a branch with its own strategy and nested targets. A branch must not also have a key. | SINGLE / LOADBALANCE / FALLBACK / CONDITIONAL | — (required for a branch) |
| weight | LOADBALANCE only. The target’s relative share of traffic (not a percentage). | a number greater than 0 | 1 |
| request_timeout_ms | Give up on this target if the provider hasn’t responded in time (and fall back / error). Guards against a hung provider. | whole milliseconds | no per-target limit |
Any value you set here replaces what the application asked for when this target is chosen. It’s how one config can route to several models: each target overrides the model to match its key’s provider. Anything you leave unset passes through from the original request untouched.
| Field | What it does | Accepted values | Default |
|---|---|---|---|
| model | Force a specific model for this target, regardless of the request. Essential when targets are different providers. | a model id (e.g. gpt-4-turbo) | use the request’s model |
| temperature | How creative vs. focused the output is. Lower is more deterministic. | 0.0 to 2.0 | use the request’s value |
| max_tokens | A hard cap on the length of the response. | a whole number ≥ 1 | use the request’s value |
| top_p | An alternative to temperature that limits word choice to the most likely options. | 0.0 to 1.0 | use the request’s value |
| frequency_penalty | Discourages repeating the same words. Higher reduces repetition. | -2.0 to 2.0 | use the request’s value |
| presence_penalty | Encourages introducing new topics. Higher pushes for novelty. | -2.0 to 2.0 | use the request’s value |
| stop | Sequences that, when produced, end the response immediately. | a list of short strings | use the request’s value |
| seed | Ask the provider for repeatable output across identical requests (best-effort). | a whole number ≥ 0 | use the request’s value |
| response_format_type | Force the shape of the answer — plain text, a JSON object, or a JSON schema. | text / json_object / json_schema | use the request’s value |
Retry re-attempts the same target on a transient error before any fallback kicks in. A short blip (a momentary 429 or 503) often clears on a second try, so a retry can avoid an unnecessary failover. Retries use automatic back-off (a short, growing pause between attempts). Set retry on a specific target, or on the whole config as a default.
| Field | What it does | Accepted values | Default |
|---|---|---|---|
| attempts * | Total tries for this target, including the first. 3 means one initial call plus up to two retries. | a whole number from 1 to 10 | — (required if retry is set) |
| on_status_codes | Which error codes are worth retrying. Others fail immediately. | a list of HTTP status codes | 429, 500, 502, 503, 504 |
A target can serve repeated questions from a cache instead of calling the provider again — cutting cost and latency. simple reuses an answer only for an identical request; semantic also reuses it for requests that mean the same thing.
| Field | What it does | Accepted values | Default |
|---|---|---|---|
| mode * | Exact-match vs. meaning-based caching. | simple / semantic | — (required if cache is set) |
| max_age_seconds | How long a cached answer stays fresh before it’s recomputed. | a number of seconds | platform default |
Fallback in depth
On a FALLBACK config (or branch), on_status_codes is the list of provider error codes that mean “this target is having trouble — try the next one.” Anything not on the list (a success, or an error like a bad request) stops the chain and is returned to the caller as-is.
| Field | What it does | Accepted values | Default |
|---|---|---|---|
| on_status_codes | The error codes that trigger a move to the next target. | a list of HTTP status codes | 429, 500, 502, 503, 504 |
Conditional routing in depth
A CONDITIONAL config holds an ordered list of rules plus a default target. Each rule has a query (the test), a then_target_name (where matches go), and a condition_order (rules are checked low-to-high; the first match wins). If no rule matches, the conditional_default_target_name handles the request.
A query tests a field path against a value. Paths come in two families:
| Field path | What it reads |
|---|---|
| metadata.<key> | A value you attach to the request on the x-metadata header — e.g. metadata.tier, metadata.region, metadata.user_type. This is how you route by your own business attributes. |
| params.model | The model the request asked for. |
| params.temperature | The request’s temperature. |
| params.max_tokens | The request’s token limit — e.g. send large jobs to a higher-capacity model. |
| params.top_p | The request’s top-p value. |
| params.stream | Whether the request asked for a streamed response (true/false). |
These are the comparisons the dashboard offers (the label is what you see in the builder; the code is what appears in the JSON).
| Operator | Reads as | What it matches |
|---|---|---|
| $eq | equals | the value is exactly this. |
| $ne | not equals | the value is anything but this. |
| $gt | greater than | numbers strictly above this. |
| $gte | greater or equal | numbers at or above this. |
| $lt | less than | numbers strictly below this. |
| $lte | less or equal | numbers at or below this. |
| $in | is one of | the value appears in a list you give. |
| $nin | is not one of | the value is absent from a list you give. |
| $regex | matches regex | the text matches a pattern (case-insensitive). |
| $exists | exists | the field is present (true) or absent (false). |
Wrap several tests in $and (all must be true) or $or (any can be true) to build richer rules — for example, “premium tier and a non-streaming request.”
"query": {
"$and": [
{ "metadata.tier": { "$eq": "premium" } },
{ "params.max_tokens": { "$gt": 2000 } }
]
}What makes a config valid
The dashboard checks a config before it saves (and you can validate any draft on demand). The rules:
- Each target is either a leaf (has a key) or a branch (has a strategy) — never both, never neither.
- A CONDITIONAL branch needs at least one rule, and every rule’s then_target_name and the conditional_default_target_name must point at targets that exist.
- A query may only use the operators and field paths above; paths must start with metadata. or params.
- Load-balance weight must be greater than 0.
- Retry attempts must be between 1 and 10.
- Overrides stay in range: temperature 0–2, top_p 0–1, max_tokens ≥ 1, the penalties −2 to 2, seed ≥ 0.
- Nesting is at most 3 levels deep, and every referenced virtual key must exist in your organization and be usable.
Building & managing configs
The Configs page renders each config as an interactive graph: every target is a node, strategy branches fan out, and you set weights, conditions, overrides, retry and cache from side panels. Prefer JSON? The builder has an escape hatch to edit the raw structure directly. Either way, the dashboard validates as you go and shows clear errors before you save.
- Audit log — every create, update, and delete is recorded with who did it and when.
- Analytics — see how much traffic a config handled over a date range, to confirm it’s routing the way you expect.
- Edit anytime — change targets, weights, conditions, or fallbacks and save; the next request picks up the new behaviour. Your application keeps sending the same header.