Smartflow Cache Ecosystem — Beyond Local Cache

Cache Phases

65%

Avg Compression

1.25M+

Tokens Saved

~5ms

Cache Hit Latency

0.90

Semantic Threshold

Why single-layer caching misses almost everything

Identical prompts from identical users are rare. System messages evolve. User questions vary in phrasing. Conversation context grows with each turn. A naive exact-match cache misses almost everything — and the tokens it does not catch are forwarded to the provider in full, with no compression, no provider-side caching signal, and no visibility into what was saved or why.

Layer 1 — Four-Phase Local Cache (MetaCache)

Every request is checked against four local lookup phases before any token is sent to a provider. Phases execute in order; the first hit wins and the remaining phases are skipped.

Exact Key Lookup — L1 In-Memory

The request key (SHA-256 over the normalised prompt, the target model, and a hash of the system prompt) is checked against the in-process L1 cache first. Including model and system prompt in the key means the same question asked of two different models, or under two different system personas, never shares — or serves — the wrong cached answer. A hit returns the stored response in sub-millisecond time with no Redis round-trip. Covers identical repeat requests — the most common and cheapest case.

Exact Key Lookup — L2 Redis

On an L1 miss the same SHA-256 key is checked against the shared Redis store (L2). This catches exact repeats served by other proxy replicas or after an L1 eviction, and works for multimodal and structured requests where intent fingerprinting is less applicable — image inputs, JSON-heavy payloads, code prompts.

Intent Signature Match — default semantic tier

When neither exact key hits, the request is normalised and its semantic intent is fingerprinted (intent class + core topic). If the same intent signature has been seen before, the stored response is returned immediately — sub-millisecond, O(1), with no embedding inference and no external API call. This is the default semantic tier: it catches reformulations that share the same underlying question regardless of phrasing.

VectorLite BERT Semantic Search — optional embedding tier

For deployments that enable it, an additional embedding tier is available: request text is embedded locally with sentence-transformers/all-MiniLM-L6-v2 (384-dim vectors, local inference — no external API call) and a K-nearest-neighbours cosine search runs across stored embeddings in the Redis Stack vector index (default threshold 0.90). It catches paraphrases that differ entirely in wording but carry the same meaning. This tier is not on the default hot path — intent-signature matching handles the common cases at O(1) cost — but is available when maximum paraphrase recall is required.

A cache hit at any phase = zero provider tokens consumed

When the optional embedding tier is enabled, vectors are generated locally using native BERT inference — no external vector database (Qdrant, Weaviate, Pinecone) required, no network hop, and no per-embedding API cost.

Persistent Archive

Full conversation and response history is written to MongoDB in parallel with every response. Supports long-range cost attribution, compliance retrieval, cache warming on restart, and audit replay. Not a lookup phase — runs asynchronously and does not add latency.

Layer 2 — Prompt Compression (In-Flight Token Reduction)

On every cache miss, Smartflow compresses the outgoing request body before forwarding it to the provider. Compression runs transparently — clients send and receive standard API payloads.

Verbose Phrase Reduction

39 common verbose patterns are replaced with concise equivalents automatically. “In order to” → “To”. “Due to the fact that” → “Because”. Average 20–30% token reduction on typical prose-heavy system prompts.

Semantic Deduplication

Repeated concepts across the message history are detected via embedding similarity and replaced with compact references. On decompression, references are resolved before the response reaches the client.

Model-Specific Optimisation

Compression aggressiveness is tuned per provider — Anthropic models tolerate tighter compression than GPT-3.5-class models. Token savings are tracked per request in VAS logs.

65%

Avg Compression Ratio

1.25M+

Tokens Saved to Date

Verbose Patterns

Layer 3 — Transparent LLM-Side Prompt Caching

Providers like Anthropic (Claude 3+) and OpenAI offer server-side prompt caching — repeated input prefixes stored on their infrastructure and billed at a fraction of the full input rate. Anthropic cached token rate ~$3/MTok vs ~$15/MTok. OpenAI caches automatically for prompts over 1,024 tokens.

The problem: clients must opt in by attaching cache markers. Most do not. Smartflow’s prompt cache injector does this transparently for every proxied request.

Large System Messages (≥ 4,000 chars)

cache_control: {type: ephemeral} injected on every request, immediately priming the provider cache. Every subsequent call within the 5-minute window hits cached tokens.

Repetitive Medium Messages (≥ 2,000 chars, ≥ 3×)

Cache markers injected automatically once the repetition threshold is reached — covering applications with shorter but heavily reused system prompts.

SHA-256 Hash Tracking

System message hashes stored in Redis with a 15-minute TTL. The injector knows whether a system message is new (priming) or warm (cache likely live).

Response Parsing

Cache hit tokens read from every provider response (cache_read_input_tokens for Anthropic, prompt_tokens_details.cached_tokens for OpenAI) and logged with estimated dollar savings.

Observability — Cache Data in VAS Logs

Every request produces a VAS audit log entry at GET /api/vas/logs. Cache-related fields:

Field	Cache Hit	Cache Miss
`metacache.hit`	true	false
`metacache.query`	The matched query string	null
`metacache.tokens_saved`	≈ response length ÷ 4	0
`latency_ms`	5 (no provider call)	Actual RTT
`routing_strategy`	"cache"	"direct"
`routing_reason`	"cache_hit:tier=L1"	"provider:openai"
`model`	From request body	From provider response

Response headers on every cache hit:

X-Cache-Hit: true
x-cache-similarity: 1.0
x-tokens-saved: 209

Hit-Type Breakdown — /api/metacache/stats

Aggregate counters are exposed at GET /api/metacache/stats. Each lookup phase increments a distinct counter, so operators can see not just how often the cache hit, but why it hit:

Counter	Incremented by	Meaning
`exact_matches`	Phase 1 & 2	Identical request served from L1 or L2 by exact SHA-256 key.
`semantic_matches`	Phase 3	Same intent signature, different phrasing — the default O(1) semantic tier.
`vector_matches`	Phase 4 (optional)	BERT KNN paraphrase hit. Stays `0` unless the embedding tier is enabled.
`cache_hits` / `cache_misses`	All phases	Total hits (the sum of the three above) and misses forwarded to a provider.
`tokens_saved`	Every hit	Cumulative provider tokens avoided across all phases.

Combined Savings by Request Type

Request Type	Layer 1 (Local)	Layer 2 (Compression)	Layer 3 (Provider)	Est. Saving
Repeated query (exact)	Phase 1 hit (L1)	—	—	~100% cost
Structured / multimodal exact	Phase 2 hit (L2)	—	—	~100% cost
Same intent, different phrasing	Phase 3 hit (intent)	—	—	~100% cost
Paraphrased (≥ 0.90 similarity)	Phase 4 (optional)	—	—	~100% cost
New query, large system prompt	Miss	20–30% saved	60–90% on prefix	70–95% cost
New query, novel content	Miss	20–30% saved	—	20–30% cost

What Other Solutions Do

Basic AI proxies offer Phase 1 exact-match caching only. LiteLLM adds a Redis semantic cache backed by an external vector store (Qdrant). Neither applies in-flight compression. Neither injects provider-side cache markers. Neither tracks what was saved or why across all surfaces.

Smartflow catches the common cases with O(1) intent-signature matching — no embedding inference required — and, when enabled, its optional VectorLite tier runs entirely inside the proxy process using a native BERT model: no external vector database dependency, no network hop, no per-embedding API cost.

The CacheEcosystem

The Cache
Ecosystem