Office Hours — How do you manage context and memory for AI agents that need to operate across multiple clients or projects?
A daily developer question about AI/LLMs, answered with a direct, opinionated take.
How do you manage context and memory for AI agents that need to operate across multiple clients or projects?
This is the operational problem that separates a working prototype from a system that runs for six months without melting down. The issue compounds fast: every client or project adds state that the agent needs to track, and if you’re not deliberate about what you keep and what you discard, token costs spiral and latency creeps up until your agent is slower than a human.
The core tension
An agent working on a single project can dump everything into context: the codebase, the entire conversation history, all the decision logic that got you here. But the moment you have two clients, or twenty projects, you hit a wall. You can’t afford to reload the full context for every task, and you can’t let context from Project A contaminate Agent decisions on Project B. The naive solution—one fresh agent per client—kills your ability to amortize compute and reuse learned patterns across tasks.
The real issue is that context management for multi-tenant agents is a compiler problem in disguise. You’re deciding what information is essential right now versus what can be archived, summarized, or reconstructed on demand. Most teams skip this entirely and just accept higher latency and token waste.
Separate concerns: state, memory, and session scope
Start by defining three clear layers.
Session scope is what the agent needs for the current task. For a code review agent handling Project B’s pull request, that’s the diff, the project’s architectural guidelines, and maybe the last three relevant commits. Not the entire six-month git history. Not every decision made on Project A.
Project memory is the durable record of decisions, patterns, and context specific to one project. Store this outside the agent’s active context. Common approaches: a structured document with key architectural decisions, a vector database of past design choices, or a simple markdown file of “things that matter here.” When you start a new task for Project B, you query this store to pull in only what’s relevant to the current work.
Cross-project state is harder to get right and often the biggest waste. Don’t share it unless you have a reason. If Agent needs to know “we prefer Go over Rust” because it applies everywhere, that belongs in system context, not in per-project memory. If it’s “Project B uses Kubernetes but Project C uses Lambda,” that’s project-specific and should live in the project memory layer.
A practical schema for multi-tenant agents
Here’s a structure that scales better than just concatenating everything into a prompt:
Agent Context at Runtime:
├── System Instructions (fixed, 2-4K tokens)
│ └── Core values, style, safety constraints
├── Client/Project Metadata (fixed, ~500 tokens)
│ ├── Project name, tech stack, team size
│ └── Active constraints (cost limits, languages, frameworks)
├── Current Task Context (variable, 4-16K tokens)
│ ├── What the user just asked
│ ├── The specific file or change in focus
│ └── Recent error messages or test output
├── Retrieved Project Knowledge (dynamic, 2-8K tokens)
│ ├── Top 3-5 relevant past decisions
│ ├── Style guide excerpts matching current file type
│ └── Known gotchas for this tech stack
└── Session History (sliding window, 2-6K tokens)
└── Last 3-5 turns only, not the full conversation
Total: 11-35K tokens per request depending on task complexity
The key: everything below “System Instructions” gets recomputed per request. You query your project memory store using the current task as the search key. If Agent needs to know about a past decision, retrieval finds it. If it doesn’t come up, you didn’t waste tokens on it.
Memory store implementation
Use a vector database or hybrid search for project memory. Index the following:
- Architectural decisions as individual records: “We chose DynamoDB over Postgres for the recommendation engine because query latency matters more than ACID guarantees.”
- Code patterns by file type and domain: “Payment processing handlers use this retry pattern.”
- Failures and resolutions: “We had a race condition in job processing. Fixed by adding a check here.”
- Style guide excerpts keyed by language and feature type.
When Agent starts a task, embed the current request (“Add a new payment method endpoint”) and retrieve the top 5 matching records. Feed those as context.
A real example from Amplitude’s Wave (demonstrated in the November 9 Daily Signal): the agent identified new product features to build, and the coding agent then executed shipping cycles—with built-in cycle-time measurement. That works because each step has clear retrieval boundaries. The feature-identification step doesn’t need to know about past payment failures. The coding agent queries for relevant patterns in that specific subsystem.
Preventing context pollution
The hardest part isn’t storage—it’s preventing Agent from over-generalizing across projects. A few pragmatic rules:
- Never share client code across projects. Even if two clients use the same framework, their codebases should live in separate retrieval indexes. Cross-project leakage is a data governance nightmare.
- Tag every stored decision with its scope. “This architectural choice applies only to Project B” vs. “This is a company-wide standard.” When you retrieve, filter by project ID first.
- Prune aggressively. If you haven’t queried a past decision in thirty days, archive it or delete it. Old context gets stale, and serving stale information is worse than serving nothing.
- Session isolation: If Agent runs for multiple clients in sequence, reset the session context between projects. Don’t carry forward assumptions.
Token budgeting across clients
Multi-tenant agents magnify the cost-control problem. One lazy agent burning 2M tokens on one client is bad. Twenty agents doing that across twenty clients is a budget emergency.
Implement per-client or per-project token budgets at the orchestration layer, not inside the agent. Before Agent starts a task, check: “How many tokens do we have left for this project this month?” If you’re above 70% of budget, downgrade to a cheaper model or tighter context windows. If you’re hitting 90%, raise an alert.
Claude Opus 5 with cache reads at $0.20/M is a practical choice here because you can cache the project-specific retrieval results across multiple tasks for the same client, cutting repeated lookups by 80%.
The visibility problem
You need observability that tells you where tokens are actually going. Log the context composition for every request:
Task: "Add OAuth to payment flow"
Project: acme-payments
Context breakdown:
- System instructions: 2,847 tokens
- Project metadata: 412 tokens
- Retrieved decisions (5 docs): 3,124 tokens
- Current PR diff: 1,956 tokens
- Session history (last 2 turns): 892 tokens
- Total input: 9,231 tokens
- Cache hit on project metadata: yes
Over time this shows you patterns. Maybe you’re retrieving the same decision record on 60% of tasks—that’s a candidate for moving into system instructions. Or you notice a particular client always needs 40% more context than others, which might mean their project needs better documentation.
Bottom line:
Treat context management as a query compilation problem: define clear scopes (system, project, task, session), store project memory in a retrieval system keyed by task relevance, and enforce token budgets per client or project. This keeps multi-tenant agents tractable at scale without sacrificing quality.
Question via Hacker News