The Stack — Notion AI

Notion AI is a context-aware writing and reasoning layer embedded directly into a team’s existing knowledge base.

What It Is

Notion AI extends the Notion workspace — notes, wikis, databases, project trackers — with generation, summarization, and Q&A capabilities. It’s used primarily by knowledge workers and teams who live in Notion already, making it one of the few AI products where the data layer and the AI layer are genuinely co-located rather than bolted together. That co-location is the whole engineering bet.

The Architecture

Notion AI uses a hybrid model routing architecture. Notion has publicly confirmed partnerships with both OpenAI and Anthropic, and their engineering team has discussed using multiple providers simultaneously. For high-quality long-form generation and complex Q&A — their “AI Q&A” feature that searches across your entire workspace — they almost certainly route to frontier models. For lighter tasks like tone rewrites, quick summaries, or autofill in database fields, they likely route to faster, cheaper models to keep latency acceptable and margins intact. The specific model versions in use today are not publicly disclosed.

The most architecturally interesting piece is how Notion handles retrieval. Their AI Q&A feature, which lets you ask questions across your entire connected workspace, is confirmed to use RAG — they’ve written about this in their product announcements. But RAG over a Notion workspace is genuinely hard: you have heterogeneous content types (databases with structured rows, freeform pages, nested blocks, comments), and the schema is user-defined and inconsistent. Notion appears to have built a custom indexing layer that normalizes this content into retrievable chunks, likely with metadata tagging (workspace, page type, last-modified date) to support relevance filtering before the vector search step.

Inference is almost certainly fully API-based rather than self-hosted. Notion is a product company, not an infrastructure company, and running their own model serving at Notion’s scale would require a team and capital allocation that doesn’t match their public engineering profile. What they do own is the orchestration layer: the routing logic, the retrieval pipeline, the prompt construction, and the context assembly that happens before any token hits a model endpoint.

On latency and cost: Notion has made a deliberate choice to show streaming output for generation tasks, which masks model latency perceptually. For the AI Q&A feature, they’ve accepted slightly longer response times — retrieval plus generation is inherently two-step — and the product UX reflects this by framing it as a “search” interaction rather than an instant autocomplete. This is a smart framing decision that manages user expectation without requiring a faster stack.

The Smart Decision

The genuinely clever architectural call Notion made is building AI on top of the block graph rather than on top of raw text exports.

Most productivity apps that bolt on AI treat their content as a document dump: export everything to plain text, chunk it, embed it, retrieve it. That’s fast to build and works adequately. Notion instead indexed against their native block structure — which preserves hierarchy, parent-child relationships between pages, inline database references, and block types. This means their retrieval system can return not just relevant text but relevant structural context: the page this lives in, the project it belongs to, the properties attached to it.

The payoff is that answers from AI Q&A are linkable and traceable in a way that generic RAG answers aren’t. Notion can surface a citation that links directly back to the source block, not just the document. That citation behavior isn’t just a UX nicety — it’s only possible because the retrieval layer understands the graph, not just the text. It makes the AI feel native to the product rather than like a wrapper.

The Tradeoff

Notion’s core architectural bet — that co-locating data and AI is the moat — also creates their most significant constraint: they can only be as good as what’s in your Notion.

For teams that use Notion as their true system of record, this is a non-issue. But most real organizations are fractured: key decisions live in Slack threads, specs live in Google Docs, customer context lives in a CRM, code context lives in GitHub. Notion AI has no native access to any of that unless a user has manually copied it in. Competitors who approach AI as a connective tissue layer across all those systems — rather than as a feature inside one app — don’t have this limitation.

The tradeoff also shows up in cold-start dynamics. A new Notion workspace with ten pages gets almost no value from AI Q&A. The product gets meaningfully better as the workspace grows, which is a great retention mechanic but a poor activation mechanic. Teams that don’t already have years of Notion content are paying for a feature they can’t yet use, and that’s a real adoption friction that Notion has to navigate with onboarding design rather than engineering.

What You Can Steal

• Route by task complexity, not by default. Don’t send every request to your most capable (and expensive) model. Classify tasks at the edge — rewrite vs. summarize vs. multi-step reasoning — and route accordingly. Notion almost certainly does this; you can implement a simple classifier or even heuristics based on input length and task type.

• RAG over structured data requires metadata, not just embeddings. If your content has structure (owner, date, type, status), preserve it as filter metadata in your vector store. Retrieve with filters first, then rerank semantically. This dramatically improves precision over embedding-only retrieval.

• Frame latency with UX, not just engineering. Two-step pipelines (retrieve + generate) are slow. Framing the interaction as “search” rather than “autocomplete” recalibrates user expectation without changing the stack. Streaming helps too, but expectation framing costs nothing.

• Build on your data model, not on text exports. If your product has a graph, a schema, or a hierarchy, index against it natively. Plain-text chunking throws away structural signal that makes citations, traceability, and relevance dramatically better.

• Accept the cold-start problem explicitly in product design. If your AI feature requires data to be useful, design onboarding to either import existing data or set honest expectations. Trying to hide the cold-start with a generic fallback usually just produces bad answers that erode trust faster than an honest “not enough data yet.”