Office Hours — How do I integrate knowledge graphs or deterministic search into my LLM application to reduce hallucinations and improve accuracy?

How do I integrate knowledge graphs or deterministic search into my LLM application to reduce hallucinations and improve accuracy?

This is the right instinct, but the execution matters more than the architecture choice. Both knowledge graphs and deterministic search reduce hallucinations, but they work in fundamentally different ways, and picking the wrong one wastes time.

The actual problem

LLMs hallucinate because they’re predicting the next token based on patterns in training data, not retrieving facts from a database. When you ask a frontier model about a specific company’s API, it’s generating plausible-sounding documentation from correlations it learned, not reading from a source of truth. No amount of fine-tuning or clever prompting fixes this when the model has never seen your data or when information has changed since training cutoff.

Knowledge graphs and deterministic search both solve this by replacing generation with retrieval. The difference is subtle but critical: deterministic search (keyword matching, semantic search via embeddings, BM25) answers “what information is most relevant to this query?” Knowledge graphs answer “what are the logical relationships between entities?” For most production use cases, you want deterministic search first.

Start with deterministic search, not graphs

Deterministic search is faster to implement and sufficient for 80% of use cases. A basic retrieval-augmented generation (RAG) pipeline with a vector database (Pinecone, Weaviate, Qdrant) or even a simple BM25 keyword search gives you an immediate win: the LLM can now cite sources instead of hallucinating.

Here’s a minimal example using Claude Opus 5 with retrieval:

from anthropic import Anthropic

def rag_query(query: str, documents: list[dict]) -> str:
    # Naive semantic search: find most similar documents
    # In production, use a vector DB like Weaviate or embed with a proper model
    from difflib import SequenceMatcher
    
    ranked = sorted(
        documents, 
        key=lambda d: SequenceMatcher(None, query, d['text']).ratio(),
        reverse=True
    )[:3]  # Top 3 results
    
    context = "\n".join([f"Source: {d['source']}\n{d['text']}" for d in ranked])
    
    client = Anthropic()
    response = client.messages.create(
        model="claude-opus-5",
        max_tokens=1024,
        system=f"""You are answering questions based on the provided documents.
If the documents don't contain the answer, say so explicitly.
Always cite your sources by including [Source: document_name].

Documents:
{context}""",
        messages=[
            {"role": "user", "content": query}
        ]
    )
    
    return response.content[0].text

# Usage
docs = [
    {"source": "api_docs.md", "text": "GET /users returns a JSON array of user objects..."},
    {"source": "pricing.md", "text": "The Pro plan costs $99/month..."},
]

answer = rag_query("What does the GET /users endpoint return?", docs)
print(answer)

The key insight: adding source context forces the model to either cite from the documents or explicitly admit it doesn’t know. This alone cuts hallucinations dramatically. Deterministic search is fast, cacheable, and verifiable.

When knowledge graphs actually help

Knowledge graphs are valuable when you have:

• Highly structured, relational data (org hierarchies, medical drug interactions, component dependencies)
• Complex multi-hop queries (“What are all the cloud services this customer uses and which SLAs apply to them?”)
• Reasoning that requires understanding relationships, not just retrieving isolated facts

If your problem is “I need to retrieve the right document and have the LLM summarize it,” use deterministic search. If your problem is “I need to traverse relationships between entities to answer a question,” a graph helps.

A simplified example using Neo4j and Claude Opus 5:

from neo4j import GraphDatabase
from anthropic import Anthropic

def graph_rag_query(query: str, graph_driver) -> str:
    # First, let Claude figure out what entities to query
    client = Anthropic()
    
    # Step 1: Extract entities/relationships from the query
    entity_response = client.messages.create(
        model="claude-opus-5",
        max_tokens=256,
        messages=[
            {"role": "user", "content": f"""Extract entities and relationships from this query.
Respond in JSON format: {{"entities": [...], "relationships": [...]}}
Query: {query}"""}
        ]
    )
    
    # Step 2: Use that to query your graph
    # (This is pseudocode; actual Neo4j queries are more complex)
    cypher = """
    MATCH (a)-[r]-(b) WHERE a.name IN $entities RETURN a, r, b
    """
    
    with graph_driver.session() as session:
        results = session.run(cypher, entities=["User", "Service"])
        context = str(results.data())
    
    # Step 3: Answer with graph context
    response = client.messages.create(
        model="claude-opus-5",
        max_tokens=1024,
        system=f"""Use this graph data to answer the question. Cite relationships explicitly.

Graph data:
{context}""",
        messages=[
            {"role": "user", "content": query}
        ]
    )
    
    return response.content[0].text

The bottleneck here isn’t the graph; it’s getting the LLM to generate correct Cypher or SPARQL queries. Claude Opus 5 is strong at this, but it’s still not 100% reliable. Most teams end up with a hybrid: deterministic search for simple lookups, hand-written graph queries for complex business logic.

The actual tradeoff

Deterministic search (vector or keyword) wins on:

• Speed (milliseconds vs. multiple round trips)
• Simplicity (one LLM call plus a database query)
• Debuggability (if the answer is wrong, you can inspect the retrieved documents)
• Cost (cheaper, fewer tokens burned on extra reasoning)

Knowledge graphs win on:

• Representing implicit relationships (what services depend on each other?)
• Multi-step reasoning (traverse three levels of hierarchy)
• Consistency (graph constraints enforce rules like “a user can’t be deleted if they own a service”)

For most production systems, start deterministic. Upgrade to a graph only when you hit a query pattern that genuinely requires relationship traversal.

Practical reality check

The Daily Signal (Oct 14) flagged a key warning: “AI Agent Teams Waste 5x More Tokens for Barely Better Results.” The same applies to architectures. If your knowledge graph adds complexity without proportional accuracy gains, you’re burning tokens on the wrong problem. A solid RAG pipeline with Claude Opus 5 will out-perform a weak graph architecture every time.

Build incrementally: start with a vector database (Weaviate or Pinecone) + Claude Opus 5, measure hallucination rate, and only add graph structure if you find multi-hop reasoning actually improves your metrics. Most teams never need it.

Bottom line: Use deterministic search (vector or keyword RAG) as your first move—it’s faster to implement and solves hallucinations in 80% of cases. Add a knowledge graph only if you’re actually traversing relationships, not just retrieving isolated facts.

Question via Hacker News