Introduction

In a previous article, I described building an agentic search framework in Go. While that architecture handled the functional requirements well, operating it at scale revealed significant cost and latency challenges. At millions of queries per month, LLM API costs, and P95 latency approached 5 seconds.

This article presents the semantic caching architecture we implemented to address these issues. The system reduced LLM costs by 45-50% and improved P95 latency to under 2 seconds, while maintaining response freshness guarantees.

The key insight: caching at multiple granularities within the agentic pipeline provides better results than end-to-end response caching alone. Specifically, caching the agent's planning decisions—which are deterministic and independent of result freshness—achieved a 50% hit rate even with conservative similarity thresholds.

Problem Analysis

Cost Structure

A single query in an agentic search system involves multiple LLM calls:

1. Planning/Tool Selection (~8,500 input tokens): Agent reads tool definitions and decides which tools to invoke

2. Tool Execution (minimal cost): External API calls

3. Summarization (~24,000 tokens): LLM formats tool outputs into natural language

4. And couple of smaller models for rewriting the query, and selecting the tools based on the query, and tool responses.

Why Traditional Caching Fails

String-based caching provides minimal hit rates for natural language queries:

"weather in san francisco"           → cache key: hash_1
"what's the weather in sf"           → cache key: hash_2  (miss)
"san francisco weather"              → cache key: hash_3  (miss)
"tell me about san francisco weather"→ cache key: hash_4  (miss)

These queries are semantically identical but produce different cache keys. Our initial implementation with exact string matching achieved only ~15% hit rate.

The Freshness Challenge

Semantic similarity matching introduces a new problem: distinguishing between queries that should produce identical responses versus those requiring fresh data.

Consider:

• "who invented the telephone" (answer never changes)
• "what's the weather today" (answer changes daily)

Both queries might have similar embeddings to cached entries, but only the first should retrieve cached results. This requires query classification and validation beyond pure similarity matching.

Architecture Design

Multi-Layer Caching Strategy

Rather than caching only the final output, we implemented caching at three distinct layers in the processing pipeline:

Query Input
    ↓
┌─────────────────────────────┐
│ Agent Planning & Tool       │ ← Layer 1: Planner Cache (50% hit rate)
│ Selection                   │   Cache tool calls, not tool results
└─────────────────────────────┘
    ↓
┌─────────────────────────────┐
│ Tool Execution              │   No caching (requires fresh data)
└─────────────────────────────┘
    ↓
┌─────────────────────────────┐
│ Response Generation/        │ ← Layer 2: Summarization Cache (35% hit rate)
│ Summarization               │   Cache only when tools used static data
└─────────────────────────────┘
    ↓
Final Response                  ← Layer 3: End-to-End Cache (3% hit rate)

Each layer targets different characteristics:

• Layer 1 (Planning): High hit rate, deterministic outputs, no freshness concerns
• Layer 2 (Summarization): Medium hit rate, requires freshness validation
• Layer 3 (End-to-End): Low hit rate but zero computation when hits occur

Core Components

The semantic cache system consists of:

1. Embedding Generator: Converts queries to dense vector representations (768-dimensional)
2. Vector Database (ANN): Stores embeddings with approximate nearest neighbor search
3. Key-Value Store: Stores response payloads separately from embeddings
4. Query Classifier: Categorizes queries as evergreen vs time-sensitive

Implementation

Layer 1: Agent Planning Cache

The agent planning step is particularly well-suited for caching because:

• Tool selection depends only on query intent, not result freshness
• Planning outputs are deterministic for semantically similar queries
• This is the most expensive operation (6,500+ tokens per call)

type PlannerCache struct {
    embeddingClient EmbeddingClient
    vectorDB        VectorDB
    kvStore         KVStore
}

type CacheContext struct {
    ModelVersion     string
    ToolVersions     []string  // sorted
    Locale          string
    EmbeddingVersion string
    Temperature     float32
}

func (pc *PlannerCache) Get(query string, ctx CacheContext) (*ToolCalls, bool) {
    // Generate embedding
    embedding := pc.embeddingClient.Embed(query)

    // Create cache key from context
    contextHash := hashContext(ctx)

    // Search with conservative threshold
    candidates := pc.vectorDB.Search(VectorSearchRequest{
        Embedding: embedding,
        Tags:      contextHash,
        Threshold: 0.98,  // Near-exact matching
        Limit:     10,
    })

    // Verify context match and retrieve from KV store
    for _, candidate := range candidates {
        if candidate.ContextHash == contextHash {
            if toolCalls := pc.kvStore.Get(candidate.Key); toolCalls != nil {
                return toolCalls, true
            }
        }
    }

    return nil, false
}

func (pc *PlannerCache) Set(query string, ctx CacheContext, toolCalls *ToolCalls, ttl time.Duration) {
    embedding := pc.embeddingClient.Embed(query)
    contextHash := hashContext(ctx)
    key := generateKey(embedding, contextHash)

    // Store in both vector DB (for similarity search) and KV store (for retrieval)
    pc.vectorDB.Insert(embedding, key, contextHash)
    pc.kvStore.Set(key, toolCalls, ttl)
}

Cache Context Importance: Including model version, tool versions, and other configuration parameters in the cache key prevents serving stale plans after system updates. When tool definitions change, the context hash changes, effectively invalidating old cache entries.

Layer 2: Summarization Cache with Freshness Validation

The summarization cache requires additional logic to prevent serving stale responses:

func shouldCacheSummarization(query Query, toolResults []ToolResult) bool {
    // Multi-turn queries have context dependencies
    if query.IsFollowUp {
        return false
    }

    // Check tool result freshness
    for _, result := range toolResults {
        switch result.Source {
        case "web_fresh", "web_daily":
            // Results from frequently-updated sources
            return false
        case "static_index":
            // Results from weekly-updated index are cacheable
            continue
        }
    }

    // Additional constraints
    return query.IsSimpleQuery &&
           query.IsSingleStep &&
           len(toolResults) == 1
}

This conservative approach accepts a lower hit rate (35%) to ensure response freshness. Only queries that exclusively use static data sources are cached.

Layer 3: End-to-End Cache

The end-to-end cache serves as a catch-all for repeated identical queries:

func (sc *SemanticCache) GetEndToEnd(query string, ctx CacheContext) (*Response, bool) {
    // First try exact match
    exactKey := md5Hash(normalize(query))
    if resp := sc.kvStore.Get(exactKey); resp != nil {
        return resp, true
    }

    // Fall back to semantic search with high threshold
    embedding := sc.embeddingClient.Embed(query)
    candidates := sc.vectorDB.Search(VectorSearchRequest{
        Embedding: embedding,
        Tags:      hashContext(ctx),
        Threshold: 0.98,
        Limit:     5,
    })

    for _, candidate := range candidates {
        if resp := sc.kvStore.Get(candidate.Key); resp != nil {
            return resp, true
        }
    }

    return nil, false
}

Freshness Control: Two-Stage Gating

To prevent serving stale responses, we implement a two-stage freshness check:

Stage 1: Query Classification

A BERT-based classifier categorizes queries as evergreen (static answers) or time-sensitive (dynamic answers):

type QueryClassifier struct {
    model BERTClassifier
}

func (qc *QueryClassifier) Predict(query string) (isEvergreen bool, confidence float64) {
    features := qc.extractFeatures(query)
    logits := qc.model.Forward(features)

    isEvergreen = logits[0] > logits[1]  // [evergreen_logit, time_sensitive_logit]
    confidence = softmax(logits)[0] if isEvergreen else softmax(logits)[1]

    return isEvergreen, confidence
}

The classifier is trained with 95%+ precision at the cost of recall. This conservative tuning ensures we rarely cache time-sensitive queries incorrectly.

Stage 2: Post-Execution Validation

Even if the classifier isn't confident, we can still cache if we verify that only static data sources were used:

func (sc *SemanticCache) GetOrCompute(
    query string,
    ctx CacheContext,
    compute func() (Response, error),
) (Response, error) {

    // Stage 1: Query classification
    isEvergreen, confidence := sc.classifier.Predict(query)

    if isEvergreen && confidence > 0.95 {
        // High confidence evergreen - try cache
        if cached, found := sc.Get(query, ctx); found {
            return cached, nil
        }
    }

    // Execute computation
    response, err := compute()
    if err != nil {
        return nil, err
    }

    // Stage 2: Validate based on execution results
    shouldCache := false
    if isEvergreen && confidence > 0.95 {
        shouldCache = true
    } else if response.OnlyUsedStaticSources() {
        shouldCache = true
    }

    if shouldCache && response.IsCacheable() {
        sc.Set(query, ctx, response, 7*24*time.Hour)
    }

    return response, nil
}

Embedding Generation

We use a 768-dimensional embedding model (similar to BERT base) for semantic similarity:

type EmbeddingClient struct {
    endpoint string
    model    string
}

func (ec *EmbeddingClient) Embed(text string) []float32 {
    // Normalize text
    normalized := strings.ToLower(strings.TrimSpace(text))

    // Call embedding service
    resp := ec.callEmbeddingAPI(EmbeddingRequest{
        Text:  normalized,
        Model: ec.model,
    })

    // L2 normalize for cosine similarity
    return l2Normalize(resp.Embedding)
}

func l2Normalize(vec []float32) []float32 {
    var norm float32
    for _, v := range vec {
        norm += v * v
    }
    norm = sqrt(norm)

    normalized := make([]float32, len(vec))
    for i, v := range vec {
        normalized[i] = v / norm
    }
    return normalized
}

Latency: ~15-20ms per embedding generation call.

Vector Search with ANN

For efficient similarity search at scale, we use an approximate nearest neighbor (ANN) index:

type VectorDB struct {
    index    HNSWIndex  // Hierarchical Navigable Small World graph
    metadata map[string]Metadata
}

func NewVectorDB(dimension int) *VectorDB {
    return &VectorDB{
        index: NewHNSWIndex(HNSWConfig{
            Dimension:      dimension,
            M:              16,   // connections per node
            EfConstruction: 200,  // search quality during construction
            Metric:         "cosine",
        }),
        metadata: make(map[string]Metadata),
    }
}

func (vdb *VectorDB) Search(req VectorSearchRequest) []Candidate {
    // ANN search returns approximate nearest neighbors
    neighbors := vdb.index.Search(req.Embedding, req.Limit*2)

    var candidates []Candidate
    for _, neighbor := range neighbors {
        // Filter by tags and threshold
        meta := vdb.metadata[neighbor.ID]
        if meta.Tags == req.Tags && neighbor.Similarity >= req.Threshold {
            candidates = append(candidates, Candidate{
                Key:         neighbor.ID,
                Similarity:  neighbor.Similarity,
                ContextHash: meta.Tags,
            })
        }

        if len(candidates) >= req.Limit {
            break
        }
    }

    return candidates
}

Latency: ~10-15ms for search across millions of vectors.

Production Results

After deploying the multi-layer semantic cache to production (serving 10M+ queries/month), we observed:

Hit Rates by Layer

Aggregate Impact

• Cost Reduction: 48% reduction
• Latency Improvement: P95 latency from 3.2s to 1.9s (41% reduction)
• Freshness: Zero incidents of stale responses since implementing two-stage gating

Layer Contribution to Savings

The planner cache (Layer 1) contributes disproportionately to total savings:

• Planner cache: ~56% of cost savings
• Summarization cache: ~30% of cost savings
• End-to-end cache: ~10% of cost savings

This validates the strategy of caching deterministic intermediate computations rather than focusing solely on final outputs.

Design Considerations and Trade-offs

Similarity Threshold Selection

We experimented with thresholds from 0.85 to 0.99:

• Threshold < 0.90: Unacceptable false positive rate. Example: "weather in Seattle" matched "weather in San Francisco"
• Threshold 0.90-0.95: Better hit rate but occasional semantic mismatches
• Threshold ≥ 0.98: Near-exact matching, very low false positive rate

We chose 0.98 as the default. This conservative approach sacrifices some hit rate for quality guarantees.

TTL Strategy

Cache TTLs vary by layer:

• Planner cache: 7 days (tool definitions change infrequently)
• Summarization cache: 2 days (more conservative due to potential freshness issues)
• End-to-end cache: 1 days

Shorter TTLs provide additional freshness guarantees at the cost of reduced hit rates.

Cache Context Granularity

Including too many parameters in cache context reduces hit rate. Including too few causes incorrect cache hits after configuration changes.

Our context includes:

• Model name and version
• Tool definitions and versions (sorted for consistency)
• User locale
• Embedding model version
• Temperature and top-p parameters

We exclude:

• Request timestamp
• User ID
• Session ID

Embedding Model Selection

We evaluated several embedding models:

• BERT-base (768-dim): Baseline performance
• Sentence-BERT (768-dim): Similar performance to BERT-base
• MiniLM (384-dim): Faster but slightly lower quality
• Larger models (1024-dim+): Marginal improvement, significant latency cost

We selected a 768-dimensional model as the best performance/latency trade-off.

Operational Considerations

Monitoring

Critical metrics tracked in production:

type CacheMetrics struct {
    HitRate          float64  // by layer
    MissRate         float64  // by layer
    LatencySaved     time.Duration
    CostSaved        float64
    FalsePositiveRate float64  // classifier accuracy
    CacheSize        int64
    EvictionRate     float64
}

We alert on:

• Hit rate drops >10% week-over-week
• False positive rate >5% for evergreen classifier
• P99 cache lookup latency >100ms

Cache Invalidation

Beyond TTL-based expiry, we implement targeted invalidation:

func (sc *SemanticCache) InvalidateByContext(ctx CacheContext) {
    contextHash := hashContext(ctx)

    // Find all cache entries with this context
    keys := sc.vectorDB.GetKeysByTag(contextHash)

    // Delete from both vector DB and KV store
    for _, key := range keys {
        sc.vectorDB.Delete(key)
        sc.kvStore.Delete(key)
    }
}

This allows immediate invalidation when tool definitions or models are updated, rather than waiting for TTL expiry.

Storage Requirements

Approximate storage per 1M cached queries:

• Vector DB: ~3GB (768-dim floats + metadata)
• KV Store: ~5GB (compressed response payloads)
• Total: ~8GB per 1M cache entries

With 7-day TTLs and 10M queries/month, steady-state storage is approximately 50-60GB.

Conclusion

Multi-layer semantic caching is a critical component for production LLM systems operating at scale. By caching at multiple granularities—particularly at the agent planning layer—we achieved significant cost reduction (48%) and latency improvement (41%) while maintaining response quality and freshness.

The key architectural insight is that deterministic intermediate computations (agent planning, tool selection) are more cacheable than final outputs, which depend on fresh data. This inverts the typical caching strategy and provides better hit rates where they matter most.

For teams building agentic systems, the planner cache should be the first caching layer implemented. It provides the highest return on investment and requires no freshness validation.

References

• Previous article: Building a Production-Ready Agentic Search Framework