The Stack — Glean — Stochastic Sandbox

Glean is an AI-powered enterprise search and work assistant built on top of a company’s connected internal tools

What It Is

Glean indexes content across dozens of enterprise SaaS connectors — Slack, Google Drive, Confluence, Jira, Salesforce, GitHub, and more — and surfaces answers, summaries, and documents through a unified search interface. It’s aimed at knowledge workers in mid-to-large enterprises who waste time hunting across fragmented tooling. Its primary competitors are Microsoft Copilot (via 365 integration) and Notion AI, but Glean’s pitch is connector breadth and enterprise-grade permissions enforcement.

The Architecture

Glean’s core infrastructure challenge isn’t generation — it’s indexing. The company has built a real-time crawling and sync pipeline that maintains document-level indexes across dozens of SaaS platforms. Every connector must handle OAuth, incremental sync, webhook or polling-based updates, and deduplication. This is genuinely hard systems work that precedes any LLM call, and it’s where most of Glean’s engineering investment visibly lives based on their published connector roadmap and engineering blog posts.

The retrieval layer uses a hybrid approach: keyword search (BM25-style) combined with dense vector embeddings for semantic retrieval. Glean has confirmed in public talks that they run their own embedding models rather than relying entirely on third-party embedding APIs — a decision that gives them latency control and cost predictability at the scale of millions of documents per enterprise tenant. Results from both retrieval paths are fused and re-ranked before being passed to a generation model.

For generation, Glean has not publicly disclosed which frontier model powers their primary assistant responses. Based on their enterprise positioning, security requirements, and the fact that they offer a cloud-plex deployment option (where models run inside a customer’s VPC via AWS or GCP), the generation layer likely supports a mix of hosted frontier models and, for air-gapped deployments, open-weight alternatives — but specific models are not publicly confirmed. Glean has described their architecture as model-agnostic by design, which is consistent with an abstraction layer sitting between their retrieval pipeline and generation.

Permissions enforcement is baked into retrieval, not post-processing. When a user queries Glean, the system filters the candidate document pool against that user’s access controls — sourced from the originating SaaS platform — before any content reaches the context window. This is architecturally non-trivial: ACLs must be synced and kept fresh across every connector, and stale permissions create either data leaks or frustrating gaps. Glean has written publicly about building a dedicated permissions sync pipeline that runs independently of the document index.

The Smart Decision

Glean’s most important architectural decision is treating permissions as a first-class retrieval concern rather than a generation-layer guardrail. Many naive enterprise RAG implementations retrieve broadly, then attempt to filter outputs or instruct the model not to reveal restricted content. Glean filters the retrieval set before the model ever sees a document.

This matters for two reasons. First, it’s actually safe — model-layer filtering via instructions is not a reliable security primitive, and telling a model “don’t mention this document” doesn’t reliably prevent leakage in complex prompts. Second, it means Glean can make a credible security promise to enterprise buyers: the model never sees content the user isn’t authorized to see. That’s a procurement-unblocking claim. In highly regulated industries — legal, finance, healthcare — this is the difference between a signed contract and a failed security review.

The operational cost of this approach is significant: maintaining real-time ACL sync across 100+ connectors is expensive and failure-prone. But Glean has clearly decided that this is a feature, not overhead, because it’s the architectural foundation of their enterprise trust story.

The Tradeoff

Glean’s connector-first, permissions-enforcing architecture means they cannot offer a useful product without deep integration work on the customer side. Unlike a general-purpose assistant that works immediately, Glean requires IT-administered OAuth grants, connector configuration, indexing time (which can take days for large document corpora), and ongoing maintenance as SaaS APIs change. This creates a long time-to-value that makes it structurally difficult to sell to smaller organizations or close deals quickly.

The tradeoff compounds at the model layer. Because retrieval quality depends on connector freshness, any stale index degrades answer quality in ways that are invisible to the end user. A knowledge worker who asks “what’s the current status of Project X?” and gets a confidently-worded summary based on a Confluence page from three weeks ago has a worse experience than getting no answer. The RAG architecture makes hallucination harder to distinguish from stale retrieval — both look like confident, plausible, wrong answers. Glean’s engineering blog acknowledges freshness as an ongoing challenge but has not disclosed specific SLAs around index lag.

What You Can Steal

Enforce permissions at retrieval, not generation. Filter your document candidate set to what the user can access before building the context window. Never rely on prompt instructions to suppress restricted content.
Run your own embedding models for predictable unit economics. At high document volume, third-party embedding APIs become a significant recurring cost. Self-hosted embedding gives you cost control and removes a latency-adding external call from your hot path.
Treat connector freshness as a product metric, not just an ops concern. Surface index age to users when it’s relevant (“sourced from a document last synced 4 days ago”) rather than silently serving stale retrievals with full confidence.
Hybrid retrieval outperforms either approach alone. BM25 handles exact-match lookups (ticket numbers, names, acronyms) that dense retrieval misses; semantic search handles intent that keyword search misses. Fuse both and re-rank rather than picking one.
Model-agnosticism is an enterprise sales feature. If you’re building for regulated industries, designing an abstraction layer between your retrieval pipeline and the generation model lets you offer on-prem or VPC-isolated deployments without rearchitecting. Enterprise buyers will pay a meaningful premium for this — and some won’t buy without it.