Zero-Cost Semantic Caching: Slashing LLM Bills Without Stale Answers
By AI Infrastructure Engineering
The Inefficiency of Repetitive LLM Generations
In customer support for aesthetic clinics and boutique hotels, inquiries follow a Pareto distribution:
"Where can I park my car?"
- "What time is checkout?"
"Can I exercise after Botox injections?"
Dispatching every identical or closely paraphrased question to an external LLM costs unnecessary money and introduces 2 to 4 seconds of generation latency.
Two-Tier Semantic & Exact Caching
Aesthetic AI introduces a dual-layer caching pipeline:
1. **Exact Canonical Query Hash**: Normalizes user input (lowercasing, punctuation stripping, language canonicalization) and checks Redis in <2ms.
2. **Cosine Vector Semantic Cache**: Generates a high-speed embedding of the query. If the cosine similarity to a recently answered question exceeds 0.94 within the same tenant context, the verified answer is returned immediately.
Zero Credit Charge Policy
Because cache hits incur negligible compute overhead on Cloudflare Workers, **tenants are charged 0 credits for cache hits**. This directly rewards clinics and hotels for having well-defined, frequently accessed FAQs.
Instant Versioned Invalidation
When a tenant updates their clinic pricing or hotel check-in policy, how do we prevent stale cached responses?
Every cache entry is tagged with the tenant's current `knowledge_version`. As soon as a clinic coordinator publishes an updated price list, the `knowledge_version` increments atomically, invalidating all older cached answers instantly.