Skip to main content
Wardin has two caching layers. They are distinct mechanisms — don’t conflate them.

​
L1: Exact-match cache

A Redis-backed cache keyed on a deterministic hash of the request body (model + messages + params). On a hit, the cached response is returned immediately at $0 cost — the request never reaches the provider. Eligibility — a request is cache-eligible when all of these are true:
  • temperature is explicitly set to 0 (omitted temperature uses the provider’s non-deterministic default, so it is not cacheable)
  • No streaming ("stream": false or omitted)
  • No tools / function calling (an empty tools array is fine)
Cache hits are marked with the X-Wardin-Cache: HIT response header. Tenant scoping — cache keys are always prefixed with tenant_id. A cross-tenant cache hit is impossible by construction. Dashboard — the ROUTE stage shows the cache split (exact vs. semantic hits) with savings calculated at the model’s full input price; exact and semantic are never blended into one number.

​
L2: Semantic cache (pgvector)

A vector similarity cache backed by pgvector. When an incoming prompt has no exact match, its embedding is compared against stored embeddings. If similarity ≥ the configured threshold (default 0.92), the cached response is returned. Semantic hits are approximate — the response was generated for a similar but not identical prompt. The dashboard separates these from exact hits because they carry approximation risk. Semantic hits are marked with the X-Wardin-Cache: SEMANTIC response header. Threshold — configurable per tenant in Console → Cache. Raise it for more conservative matching; lower it for a higher hit rate with more approximation. Threshold changes are recorded to the EVIDENCE audit trail. Bypass — set "stream": true, include tools, or use a non-zero (or omitted) temperature to bypass both cache layers for that request.

​
Choosing an embedder (no OpenAI account required)

The semantic cache needs an embedder to turn prompts into vectors. It defaults to OpenAI’s text-embedding-3-small (1536 dimensions), but it is not tied to OpenAI — the embedder speaks the standard /v1/embeddings protocol, so any OpenAI-compatible endpoint works. Anthropic has no embeddings API, so an Anthropic-only deployment uses a local or third-party embedder here. To run L2 against a local/self-hosted model (Ollama, LM Studio, vLLM, Hugging Face TEI): WARDIN_EMBEDDING_DIM must match the model’s output length. It sets the pgvector column dimension when database migrations run — so set it before your first migration. The gateway then probes the embedder once at boot and, if its output length doesn’t match the actual column dimension, disables L2 with a clear error rather than ever storing mismatched vectors. Changing the dimension on an already-migrated database means re-running the reconcile migration (migration:revert then migration:run), which rebuilds the column — safe because the L2 store holds only ephemeral, TTL’d rows. Set WARDIN_EMBEDDER=off to disable L2 embedding entirely.

​
Provider-native prompt caching (Anthropic)

This is not Wardin’s cache — it’s Anthropic’s server-side cache_control feature. Wardin passes cache_control blocks through unmodified and parses the resulting token types from the response separately: If these are blended, cost dashboards for agentic traffic (Claude Code, Cursor) will be significantly wrong. Wardin never blends them.

​
Cache metrics

The dashboard shows these three numbers separately so you know which savings carry zero risk (exact) vs approximation risk (semantic) vs provider-level savings (prompt cache).