Library of the Week — Semantic Router

Semantic Router — blazing-fast semantic decision layer for LLM pipelines

GitHub · Language: Python · License: MIT

What it does

Semantic Router lets you classify incoming text into predefined “routes” using embedding similarity rather than an LLM call — routing a user message to the right handler in microseconds instead of seconds. It’s aimed at developers who need to gate, guard, or direct LLM traffic without paying per-token for a classifier model. Think of it as a semantic if/elif chain that actually understands language.

Why it stands out

Encoder flexibility: swap between local encoders (FastEmbed, HuggingFace) and hosted ones (OpenAI, Cohere, Mistral) with a single argument — no rewriting logic when you change your embedding stack
Dynamic routes: beyond static classification, dynamic routes pass matched utterances into a small LLM call for slot-filling, letting you extract structured parameters from the same phrase that triggered the route
Policy/guardrail routes: you can define a “politics” or “jailbreak” route with a handful of examples and block or redirect those queries before they ever touch your expensive frontier model
Local-first option: with FastEmbed as the encoder, the entire router runs fully offline with no API calls — useful for latency-sensitive or air-gapped deployments

Quick start

from semantic_router import Route, RouteLayer
from semantic_router.encoders import FastEmbedEncoder

billing = Route(
    name="billing",
    utterances=[
        "What's on my invoice?",
        "I was charged twice",
        "Can I get a refund?",
    ],
)

support = Route(
    name="support",
    utterances=[
        "My integration is broken",
        "How do I authenticate?",
        "I'm getting a 429 error",
    ],
)

encoder = FastEmbedEncoder()
rl = RouteLayer(encoder=encoder, routes=[billing, support])

result = rl("I keep getting rate limited")
print(result.name)  # → "support"

When to use it

You’re building a multi-intent chatbot or agent and need to dispatch to different tools, prompts, or handlers without burning tokens on a routing LLM call
You want a lightweight guardrail layer in front of an expensive model — e.g., catch off-topic or harmful queries before they reach Claude Fable 5.1 or GPT-6 Astra
You need sub-10ms routing decisions in a high-throughput API where an extra LLM hop is unacceptable

When to skip it

If your routing logic is highly context-dependent or requires multi-turn history, embedding similarity on a single utterance will miss nuance — a dedicated LLM classifier or a fine-tuned model will outperform it
Projects already using a framework with built-in routing (LangGraph conditional edges, Gemini Managed Agents) may find the overlap isn’t worth the added dependency

The verdict

Semantic Router earns its place in any production LLM stack as the cheap, fast first layer before anything hits your real models. The encoder-agnostic design means it stays useful as your embedding stack evolves, and the guardrail use case alone can pay for itself in reduced moderation costs. If you’re hand-rolling if "refund" in message checks anywhere, this is the clean replacement.