Library of the Week — lm-format-enforcer

lm-format-enforcer — Enforce output formats (JSON, regex, grammars) on any local LLM inference backend

GitHub · Language: Python · License: Apache 2.0

What it does

When you run local models through vLLM, llama-cpp-python, or Hugging Face Transformers, there’s no built-in guarantee the model produces valid JSON or follows a schema. lm-format-enforcer solves this by filtering the token logits at each generation step — only tokens that could lead to a valid output are allowed through. It’s the structured-output layer for teams running their own inference rather than relying on a hosted API.

Why it stands out

• Backend-agnostic by design — ships first-class integrations for Hugging Face pipelines, llama-cpp-python, and Exllamav2, plugging into each backend’s native logits-processor hook rather than wrapping the whole stack
• Grammar support beyond JSON — accepts regular expressions and full GBNF context-free grammars, so you can enforce things like [A-Z]{2}-\d{4} output codes or custom structured formats that JSON Schema can’t express
• Zero-retry overhead — because invalid tokens are masked before sampling, you never burn context or latency on a failed generation that needs reparsing; the model produces a valid result on the first pass
• Pydantic integration — pass a Pydantic model directly and it auto-derives the JSON Schema, keeping your type definitions as the single source of truth

Quick start

from pydantic import BaseModel
from lmformatenforcer import JsonSchemaParser
from lmformatenforcer.integrations.transformers import build_transformers_prefix_allowed_tokens_fn
from transformers import pipeline

class Answer(BaseModel):
    reasoning: str
    score: int

pipe = pipeline("text-generation", model="mistralai/Mistral-7B-v0.1")
parser = JsonSchemaParser(Answer.model_json_schema())
prefix_fn = build_transformers_prefix_allowed_tokens_fn(pipe.tokenizer, parser)

result = pipe(
    "Rate the code quality from 1-10 and explain why.",
    prefix_allowed_tokens_fn=prefix_fn,
    max_new_tokens=200,
)
print(result[0]["generated_text"])

When to use it

• You’re running local inference (Transformers, llama-cpp-python, Exllamav2) and need reliable structured output without adding a hosted-API dependency
• Your output format is more complex than JSON — regex patterns, enums, or domain-specific grammars — and Pydantic-based libraries like Instructor don’t give you enough control
• Latency matters and you can’t afford retry loops on malformed generations

When to skip it

• If you’re calling a hosted API (GPT-6 Astra, Claude Fable 5.1, Gemini 3.8 Flash), just use the provider’s native structured-output mode — lm-format-enforcer adds nothing there and Instructor handles the schema layer cleanly
• Very long or deeply nested schemas can noticeably slow token generation because the allowed-token mask must be recomputed at each step; for those cases, Outlines (already covered) uses a precompiled finite-state machine and may be faster

The verdict

lm-format-enforcer is the pragmatic choice when you control your inference stack and need format guarantees without switching frameworks. It’s narrower in scope than Outlines but lighter to drop into an existing pipeline, and the Pydantic integration keeps boilerplate minimal. If structured output from local models is a recurring pain point in your stack, this belongs in your toolkit.