Library of the Week — Pipecat
A weekly teardown of one open-source AI/ML library: what it does, why it stands out, and when to use it.
Pipecat — real-time voice and multimodal AI pipeline framework
GitHub · Language: Python · License: BSD-2-Clause
What it does
Pipecat is an open-source framework for building real-time, streaming voice and multimodal AI applications — think voice agents, live translation bots, and video-capable assistants. It handles the hard plumbing: audio transport, turn detection, STT/TTS integration, and LLM chaining, so you can focus on conversation logic rather than WebRTC glue code.
Why it stands out
- Transport-agnostic architecture — ships with built-in support for Daily (WebRTC), WebSockets, and local audio, letting you swap transports without rewriting pipeline logic
- First-class streaming design — frames flow through a processor pipeline rather than request/response cycles, which means latency compounds favorably; voice interruptions and barge-in detection work naturally
- Broad integrations out of the box — connects to Deepgram, ElevenLabs, Cartesia, OpenAI, and Gemini with typed frame classes, so adding a new STT or TTS provider is a single processor swap
- Multimodal beyond audio — supports video frames and image input, making it practical for agents that need to see a screen or a camera feed, not just hear speech
Quick start
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask
from pipecat.services.deepgram import DeepgramSTTService
from pipecat.services.openai import OpenAILLMService
from pipecat.services.elevenlabs import ElevenLabsTTSService
from pipecat.transports.local.audio import LocalAudioTransport
transport = LocalAudioTransport()
stt = DeepgramSTTService(api_key="...")
llm = OpenAILLMService(api_key="...", model="gpt-4.1-nano")
tts = ElevenLabsTTSService(api_key="...", voice_id="...")
pipeline = Pipeline([transport.input(), stt, llm, tts, transport.output()])
runner = PipelineRunner()
task = PipelineTask(pipeline)
await runner.run(task)Pin pipecat-ai>=1.2.0. Versions 0.0.90 through 1.1.x carry a path traversal bug in the development runner (CVE-2026-44716): a runner started with --folder will serve arbitrary file reads to unauthenticated requests, including anything the process can open. Fixed in 1.2.0. Separately, steer clear of the deprecated LivekitFrameSerializer — it runs pickle.loads() on WebSocket input (CVE-2025-62373). It is optional and off by default; use LiveKitTransport instead.
When to use it
- Building a real-time voice agent where latency matters — customer support bots, interview simulators, or live coaching tools where you need sub-second response feel
- You want to mix and match STT, LLM, and TTS vendors without building your own streaming glue, and you need the swap to be surgical rather than a rewrite
- Your agent needs to handle interruptions gracefully — barge-in mid-sentence is a real UX requirement, not an edge case
When to skip it
- If you’re building batch or async pipelines (document Q&A, RAG, agentic coding), Pipecat’s streaming-first model adds complexity with no payoff — reach for a simpler agent framework instead
- WebRTC infrastructure via Daily adds a managed dependency; if your org has strict data-residency requirements and no existing WebRTC stack, the transport layer needs more work
The verdict
Pipecat fills a genuine gap: most LLM frameworks treat voice as an afterthought bolted onto a text pipeline, while Pipecat was designed from the start around real-time audio frames. If you’re shipping a voice product and you’re tired of duct-taping Deepgram webhooks to an async queue to an ElevenLabs HTTP call, this is the framework to reach for. The frame-based pipeline model is clean enough that adding custom processors feels idiomatic rather than hacky.