The Stack — Otter.ai — Stochastic Sandbox

Otter.ai is a real-time meeting transcription and AI notes platform built on top of speech recognition and language models.

What It Is

Otter.ai records, transcribes, and summarizes meetings across Zoom, Google Meet, and Microsoft Teams — automatically joining as a bot participant and delivering structured notes, action items, and searchable transcripts. It’s used by sales teams, journalists, students, and distributed engineering orgs who need an accurate, queryable record of spoken conversation. The product sits at an interesting intersection: it’s doing both hard speech recognition work and downstream language processing on every session.

The Architecture

Otter’s core pipeline has two distinct stages: audio-to-text and text-to-insight. The ASR (automatic speech recognition) layer is the foundation of the product — Otter has built proprietary speaker diarization and transcription models rather than routing everything through a commodity API. This is evidenced by their 2018 acquisition of speech AI talent and years of domain-specific training on meeting audio specifically. They’ve published engineering content referencing in-house models trained on meeting-specific acoustic conditions — crosstalk, variable microphone quality, accented speech — where general ASR models tend to underperform. This is confirmed, not inferred.

For the language layer — summaries, action items, OtterPilot-generated meeting notes — the specific model used is not publicly disclosed. Based on Otter’s pricing tier structure and latency characteristics, they appear to use a cost-efficient frontier-class model (possibly a mid-tier offering from one of the major providers, or a fine-tuned open-weight model) rather than routing every transcript through a flagship model. Meeting summaries are generated post-hoc, not streaming, which gives them flexibility on inference cost that a real-time product wouldn’t have.

Inference likely runs as a hybrid: proprietary ASR on self-managed infrastructure (latency-sensitive, needs tight control), with language model calls going to an external API for summary generation. This is a natural split — the ASR pipeline benefits from hardware optimization and custom model control, while the summarization step is bursty and stateless enough to offload.

At scale, Otter’s biggest infrastructure concern is probably storage and retrieval, not raw inference. Every meeting generates a transcript, audio artifact, and structured metadata. Search across a user’s meeting history — a core feature — requires either a vector index or a well-designed keyword search layer. They appear to use a hybrid retrieval approach: full-text search for exact queries and embedding-based retrieval for semantic search. Neither is confirmed in detail, but the behavior of their search feature is consistent with a dual-retrieval architecture.

Cost control here is unusually important because Otter serves a freemium user base where a significant portion of users are on the free tier generating real transcription cost with no revenue offset. Their business model pressures them toward efficient inference more than, say, a pure enterprise product would.

The Smart Decision

Otter’s most defensible architectural decision is building proprietary ASR rather than riding a third-party transcription API. In the early days, this looked expensive and risky — training speech models requires massive labeled datasets and real expertise. But it created a moat that API-dependent competitors can’t easily replicate.

Because they control the ASR layer, they can do speaker diarization at quality levels tuned specifically for meeting audio, train continuously on error corrections surfaced through user feedback, and avoid dependency on providers like Google, OpenAI (GPT Transcribe), or ElevenLabs for their core value proposition. Notably, independent benchmarks have shown that even current-generation API-based transcription products — including OpenAI’s GPT Live Transcribe — lag behind specialized tools on real meeting audio with crosstalk and accents. Otter’s years of domain-specific training is directly relevant here.

This is a classic “build vs. buy” decision that paid off because the capability was core, not adjacent. If your differentiator is transcription quality, you can’t afford to be at the mercy of another vendor’s error rate.

The Tradeoff

The flip side of owning your ASR stack is that you now have to maintain it as the speech recognition landscape evolves rapidly. Otter is running what is essentially a research-grade speech team inside a product company — and that’s expensive and organizationally demanding in a way that a pure API integration wouldn’t be.

More concretely, this investment may be over-indexed on a capability that’s becoming commoditized fast. GPT Transcribe and GPT Live Transcribe launched via API in 2026, and while current benchmarks still show gaps on meeting-specific audio, the trajectory is clear. Google’s Gemini Omni handles multimodal audio natively. ElevenLabs continues to improve on voice clarity tasks. The bespoke ASR advantage that justified years of investment is under real competitive pressure — and unlike the language model layer, which Otter can swap or upgrade via API, their transcription differentiation is load-bearing in a way that’s hard to pivot away from.

What You Can Steal

Split your pipeline at the latency boundary. Otter runs ASR in near-real-time and summarization post-hoc. If parts of your pipeline don’t need to be synchronous, don’t make them synchronous — it unlocks cheaper, larger model calls.
Own the layer that’s actually your moat. Otter uses APIs for commodity tasks and builds in-house for what differentiates them. Map your own product the same way before defaulting to “build everything” or “buy everything.”
Design for freemium cost pressure from day one. When a meaningful share of your users generate compute cost without revenue, your model selection and caching strategy need to reflect that — not just your enterprise tier pricing.
Dual retrieval beats single retrieval for meeting content. Structured meeting data has both exact-match needs (“find the meeting where we discussed Salesforce”) and semantic needs (“find all discussions about pricing strategy”). A hybrid index handles both without forcing a choice.
Domain-specific fine-tuning on acoustic data compounds over time. If your product lives or dies on a specific input modality (audio, code, legal docs), even modest domain adaptation on a base model — or on your retrieval corpus — pays dividends that generic API callers can’t replicate.