VectorAxis
← Blog

One endpoint, every provider: routing, fallbacks, and caching

VectorAxis Team ·

In the first post we argued that shipping AI in production quietly turns you into an AI infrastructure team. This post is about the half of that work that keeps you online and cheap: routing, fallbacks, and caching.

The theme running through all three is the same — you get them without rewriting your app.

Start where every provider agrees: swap the base URL

Every LLM provider has its own SDK and wire format, which is exactly why going multi-provider normally means a refactor. VectorAxis removes that by speaking the OpenAI wire format for everyone — 37+ providers behind one endpoint.

So the integration is a one-line change:

# before
client = OpenAI(api_key="sk-...")

# after — one endpoint, every provider
client = OpenAI(
    base_url="https://api.vectoraxis.ai/v1",
    api_key="vk-1a2b3c4d5e6f7a8b",
)
# everything else stays the same

Same SDK, same method calls, same response shape. From here, everything below is either a header or a config — never a code rewrite.

Routing: send each request to the right model

Not every request deserves your most expensive model. A classification call or a "summarize this in one line" doesn't need the same horsepower as a complex reasoning task. But hard-coding that logic into your app is brittle, and changing it means a deploy.

VectorAxis lets you define routing configs — visual graphs that decide where a request goes:

  • Load-balance across multiple providers or keys to spread traffic and rate limits.

  • Weight targets so you can shift, say, 80% to a cheap model and 20% to a premium one.

  • Conditionally route based on the request, so simple calls take the cheap path and hard calls take the strong one.

You build these in the visual graph builder under Configs, give the config a slug, and reference it per request. The routing logic lives in the gateway, so tuning your cost/quality tradeoff is a config change — not a code change and not a redeploy.

Fallbacks and retries: AI that doesn't go down

This is the feature that turns a provider outage from an incident into a non-event.

Every non-streaming request through VectorAxis is wrapped in retry-and-fallback handling:

  • Automatic retries with exponential backoff — 1s, 2s, 4s, 8s… — on the status codes that mean "try again": 429, 500, 502, 503, 504, 529. If the provider sends a Retry-After header, VectorAxis honors it instead of guessing.

  • Fallback chains you declare per request. If your primary provider keeps failing, the request fails over to the next provider in the chain — automatically, mid-flight.

You wire a fallback chain with headers:

x-fallback-1-provider: ANTHROPIC
x-fallback-1-api-key:  sk-ant-...
x-fallback-1-model:    claude-sonnet-5     # optional model override

Now picture the scenario from post one. OpenAI starts returning 429s. VectorAxis retries with backoff; if it's still failing, it fails over to Anthropic. Your user gets an answer. Your on-call phone stays quiet. And you didn't write a single line of failover logic — you added a header.

(One honest note: streaming responses can't be retried mid-stream, because you can't un-send bytes already flushed to the client. Retries and fallbacks apply to non-streaming calls, where buffering is safe.)

Caching: stop paying for the same answer twice

Routing and fallbacks keep you online. Caching keeps you cheap. A huge share of real-world AI traffic is repetitive — the same FAQ, the same document summarized again, the same prompt fired twice by an impatient user. Paying full price and full latency for an answer you already computed is pure waste.

VectorAxis gives you two cache layers, both opt-in per request.

Exact-match cache (simple)

For identical requests. VectorAxis builds a SHA-256 key over the full request — provider, model, system prompt, messages, temperature, top-p, max tokens, stop — and returns the stored answer on a hit. You opt in with x-cache-mode: simple and control freshness with a per-request TTL. Same question, instant answer, zero provider cost.

Semantic cache

Exact-match only catches identical requests. But "What's your refund policy?" and "How do I get a refund?" want the same answer. The semantic cache catches those.

It embeds the request and searches previously cached entries by cosine similarity using pgvector with an HNSW index, pre-filtered by provider and model so you never cross-contaminate. On a hit, you serve a known-good answer for the cost of an embedding instead of a full completion. On a miss, VectorAxis stores the new answer in the background so the next similar question is a hit.

This is where support bots, FAQ assistants, and any "long tail of slightly different phrasings" workload get dramatically cheaper.

You can see it working

Every response carries a cache status header so this isn't a black box — HIT, SEMANTIC_HIT, MISS, BYPASS, DISABLED, REFRESH. You can watch your hit rate climb and put a real number on what caching saved you.

Putting it together

These three features tell one story:

  • Routing sends each request to the right-priced model.

  • Fallbacks and retries mean a provider having a bad day doesn't take you down with it.

  • Caching means you compute each answer once and reuse it — exactly or semantically.

Uptime from the first two, cost from the third — and all of it behind the single base URL you already pointed your SDK at.

What's next

The next post covers the other half of the infrastructure you inherited in production: governance and safety — encrypted virtual keys with spend caps, guardrails on inputs and outputs, and full request-level observability. That's the difference between a demo and something you can actually run in production.

Start free — no credit card, free tier included.