Office Hours — How should you combine global knowledge, internet search, and user RAG in a single system?

How should you combine global knowledge, internet search, and user RAG in a single system?

The real problem isn’t picking one over the others. It’s that they serve different functions and operate at different latencies, costs, and reliability levels. Most systems fail because they try to use a single retrieval strategy for everything.

Three retrieval layers, not one

Think of this as a cascade. Start with what’s cheap and fast, escalate to what’s expensive only when necessary.

Layer 1: User RAG (your docs, cached). This is your first move. It’s instant, deterministic, and contains no hallucinations about your own stuff because the source material is right there. Pre-embed and cache your knowledge base. Claude Opus 5 supports prompt caching at ~90% discount for cached tokens, so store your domain-specific context, documentation, and user history once and reuse it across requests. This layer should handle 70-80% of queries if your RAG is tuned.

Layer 2: Global knowledge (model weights). Your frontier model already contains vast training data. For questions about public facts, historical events, or widely-known technical concepts, the model’s weights are sufficient and faster than searching. The catch: context window is finite, so you’re not querying a full knowledge base here, just what the model learned during training. This is free (no extra API calls) but can hallucinate on recent or obscure information. Use it for questions where the model’s training data is likely to be sufficient.

Layer 3: Internet search (expensive, slow). Only reach here when the previous two layers indicate they’re uncertain or missing information. This is the circuit breaker. If your RAG returned nothing and the model isn’t confident on global knowledge, then pay for a web search. Real-time data, current events, breaking information—this is where search lives. But it’s 5-10x slower and costs real money per call.

Implementation pattern: Decide before you search

The key is making the decision upfront, not trying all three and hoping one works.

async def retrieve_context(query: str, user_id: str) -> dict:
    # Layer 1: Check user RAG first (cached, instant)
    rag_result = await query_cached_user_docs(user_id, query)
    
    if rag_result.confidence > 0.8:
        return {
            "source": "user_rag",
            "content": rag_result.content,
            "cost": 0  # Already cached
        }
    
    # Layer 2: Ask the model if it knows this from training
    model_confidence = await estimate_model_confidence(query)
    
    if model_confidence > 0.7 and not requires_recency(query):
        return {
            "source": "model_weights",
            "content": None,  # Use model's own knowledge
            "cost": 0
        }
    
    # Layer 3: Only search if we're uncertain and need current info
    if requires_recency(query) or model_confidence < 0.6:
        search_result = await web_search(query)
        return {
            "source": "web_search",
            "content": search_result,
            "cost": 0.01  # Actual search cost
        }
    
    # Fallback: admit uncertainty
    return {"source": "none", "content": None, "cost": 0}

The requires_recency() check is critical—it filters out queries about stock prices, news, weather, and breaking events that your training data can’t answer. For a legal system, this might flag queries about recent court decisions. For a medical system, it might flag new clinical guidelines.

Real tradeoff: routing complexity vs. token savings

Every extra decision point costs tokens. If you call the model twice (once to estimate confidence, once for the actual answer), you’ve doubled your costs. Databricks benchmarked this and found that GLM-5.2 outperformed Claude Opus 4.8 on their codebase by a 1.5x margin purely because it required fewer routing decisions before answering. You’re trading orchestration complexity for inference cost.

The winning pattern from production teams: precompute routing signals offline. Build a classifier on historical queries that learns “does this need web search?” or “is this in the user’s docs?” Then use that classifier to route new queries without calling the model first. This shifts the routing cost to training time, not inference time.

Caching and invalidation: the hidden gotcha

If you cache user RAG, you need a cache invalidation strategy. Claude Sonnet 5’s new tokenizer emits ~30% more tokens than the previous version for the same text, which means cached cost calculations can drift. If you’re relying on prompt caching to make RAG economical, monitor your actual cached hit rates—they often drift in production because user docs change and cache keys become stale.

One production system at a large insurance company cached employee handbooks in prompts and discovered after three months that 40% of cached results were outdated because policies had changed but the cache key hadn’t. They switched to a TTL-based invalidation (regenerate cache every 24 hours) at the cost of losing cache hits, but gained correctness.

Combining them without cascading costs

The worst mistake is implementing this as a waterfall where each layer calls the next. Layer 1 fails → Layer 2 → Layer 3, and now you’ve made three API calls on a single user query. Instead, make one upfront routing decision and commit to it.

A better pattern: fetch RAG and estimate model confidence in parallel. If RAG fails and model confidence is low, then search. But don’t search after RAG fails if the model is confident. The model’s confidence on global knowledge is free information you already have.

Bottom line: Implement a three-layer cascade (user RAG → model weights → web search) with upfront routing decisions that avoid repeated API calls. Use prompt caching for user docs to make RAG economical, route to web search only when you need real-time information, and avoid cascading failures by making routing decisions once, not at every retrieval step.

Question via Hacker News