Library of the Week — Guidance

Guidance — structured LLM output validation without the schema boilerplate

GitHub · Language: Python · License: Apache 2.0

What it does

Guidance lets you interleave generation and control flow — conditions, loops, function calls — directly inside a prompt template, then constrains the model to follow that structure token-by-token. It targets developers who need reliable structured output from local or API-served models without bolt-on validators or retry loops.

Why it stands out

• Constrained decoding at the token level — rather than generating free text and parsing it afterward, Guidance masks the logits so the model physically cannot emit an invalid token, eliminating parse failures entirely for local backends
• Handles more than JSON — regex patterns, context-free grammars, and hand-written select constraints all work natively, so you can enforce things like "status": "ok" | "error" without writing a full schema
• Unified API across backends — the same template runs against Transformers, llama.cpp, and OpenAI-compatible endpoints; swap the llm= argument and nothing else changes
• Stateful generation — you can branch mid-generation on what the model just produced (e.g. route to a different sub-template if status == "error"), which is hard to replicate cleanly in parse-and-retry patterns

Quick start

from guidance import models, gen, select

lm = models.LlamaCpp("mistral-7b.gguf")

result = lm + (
    "Classify the following review as positive or negative.\n"
    "Review: The battery life is amazing!\n"
    "Sentiment: " + select(["positive", "negative"], name="sentiment") + "\n"
    "Confidence (0-100): " + gen(name="confidence", regex=r"[0-9]{1,3}") + "\n"
)

print(result["sentiment"])    # "positive"
print(result["confidence"])   # e.g. "94"

When to use it

• You need guaranteed valid structured output from a locally-served model and can’t afford retry overhead in a hot path
• Your output constraints are more expressive than a flat JSON schema — enums, regex-validated fields, conditional branches
• You’re building evaluation harnesses or data pipelines where a single malformed generation breaks downstream processing

When to skip it

• You’re exclusively using hosted frontier APIs (GPT-6 Astra, Claude Opus 5.5) where token-level masking isn’t available and Guidance falls back to prompt-based steering — at that point Instructor is simpler and equally reliable
• Your team already owns a working Pydantic validation + retry loop and the failure rate is acceptably low; the migration cost may not justify it

The verdict

Guidance earns its place when output correctness is a hard requirement rather than a best-effort one. The token-masking approach is genuinely different from the “generate, parse, retry” loop that most alternatives use, and for local model deployments it makes structured generation feel mechanical in the best sense. Start here if you run your own inference.