Core Features

Caching

When the same question comes in twice, why pay to answer it twice? Caching remembers a model’s reply and serves it again instantly — no provider call, no token cost, and a response in milliseconds instead of seconds. VectorAxis offers two flavours: exact-match for identical requests, and semantic for ones that simply mean the same thing.

Why it matters

Lower cost. A cache hit doesn’t call the provider, so it costs nothing in tokens.
Faster responses. Hits return immediately instead of waiting on the model.
Steadier load. Repeated traffic (FAQs, popular prompts, retries) is absorbed before it ever reaches a provider’s rate limit.

Caching is opt-in. Nothing is cached unless you ask for it on the request. Send the x-cache-mode header to turn it on and pick a flavour; leave it off and every request goes straight to the provider (reported as DISABLED).

The two modes

Mode	Reuses an answer when…	Best for
simple exact	a new request is byte-for-byte identical to one seen before.	repeated identical calls, idempotent retries, deterministic prompts.
semantic meaning	a new request means the same thing as one seen before, even if worded differently.	FAQ-style traffic and chat where users ask the same thing many ways.

Exact vs. semantic, in plain terms. Exact-match is like recognising the same sentence word-for-word. Semantic understands meaning: “What’s the capital of France?” and “Tell me France’s capital city” are treated as the same question and share one cached answer.

Turning it on — request headers

cache headers

Header	What it does	Accepted values	Default
x-cache-mode	Enables caching and chooses the flavour. Without it, caching is off.	simple / semantic	off (DISABLED)
x-cache-ttl	How long a stored answer stays fresh before it’s recomputed (its “time to live”).	a number of seconds	platform default
x-cache-force-refresh	Skip any stored answer for this call, fetch a fresh one, and overwrite the stored entry. Use after content changes.	true	false

What “TTL” means. Time-To-Live is how long a cached answer is considered current. A short TTL keeps answers fresh but caches less; a long TTL saves more but may serve slightly stale replies. Pick it to match how often the underlying answer actually changes.

Force-refresh doesn’t switch caching on. x-cache-force-refresh only refreshes a cache that’s already enabled with x-cache-mode. On a request with no cache mode, it has no effect — the request simply isn’t cached.

Reading the result — the x-cache-status header

Every (non-streaming) response carries an x-cache-status header so you always know whether the answer came from cache or the provider:

Status	Meaning
HIT	Served from the exact-match cache — no provider call.
SEMANTIC_HIT	Served from the semantic cache (a meaning-match) — no provider call.
MISS	Caching was on, but nothing matched; the provider answered and the reply was stored for next time.
REFRESH	Force-refresh: the provider was called and the stored entry overwritten.
DISABLED	No cache mode was sent — caching didn’t run.
BYPASS	Caching was skipped for this request (always the case for streaming).

Under the hood

For the engineers wiring this up — exactly how each mode decides a match.

Exact-match simple

VectorAxis builds a fingerprint (a SHA-256 hash) over the parts of the request that change the answer — provider, model, system + messages, temperature, top_p, max_tokens, and stop. Two requests with the same fingerprint share an answer; change any of those and it’s a new entry. The store is an in-process cache with a per-entry TTL, so hits are returned with effectively zero added latency.

Because the fingerprint includes generation settings, the same prompt at temperature 0 and at temperature 0.7 are cached separately — they genuinely produce different answers.

Semantic semantic

The request is turned into an embedding — a numeric “meaning vector” — and compared against previously stored requests by similarity (cosine distance, via the pgvector extension with an HNSW index for fast nearest-neighbour search). If the closest match scores above the similarity threshold (high by default, around 0.90), its cached answer is returned. A pre-filter on provider and model ensures a match only counts within the same model — so a question answered by one model is never served from another.

On a miss, the embedding and the fresh answer are written after the response is sent (fire-and-forget), so storing never adds to the caller’s latency.

Freshness & eviction. A stored answer is served only while it’s still within its TTL (the x-cache-ttl you set, or the platform default of about an hour) — and the window is sliding: every time an entry is served, its clock resets, so answers that keep getting used stay warm while ones that fall out of use quietly go stale. Separately, to keep storage bounded, any entry that hasn’t been served for 7 days is physically deleted from the cache. So the TTL governs whether a given request gets a cached reply; the 7-day idle window governs how long the stored row itself lives before it’s reclaimed.

Semantic caching is a plan feature — on plans that don’t include it, a semantic request safely falls back to a normal provider call (BYPASS) rather than erroring.

Caching & streaming

When a request asks for a streamed reply (stream: true), caching is skipped in both directions: no lookup and no store, whatever cache headers you send. This is structural — a stream is a series of incremental chunks, not a single finished answer, so there’s nothing whole to read from or write to the cache mid-stream.

On a streamed request, x-cache-mode, x-cache-ttl, and x-cache-force-refresh are accepted but have no effect, and the x-cache-status response header isn’t returned at all (response headers are committed before the status — always BYPASS for streams — is known). If you need caching, send the request without stream.

Tip: you can also attach caching to a specific branch of a routing config so a whole class of traffic is cached without setting headers on every call.