Office Hours — What's the best way to specify constraints and rules for AI agents so they stay aligned with your system design?

What’s the best way to specify constraints and rules for AI agents so they stay aligned with your system design?

The honest answer: most teams aren’t doing this well, and it’s a much bigger problem than just writing better prompts. Constraints fail when they’re vague, buried in natural language, or enforced only at the LLM level instead of architecturally.

Constraints Live at Three Levels

You need to think about constraints in layers. At the top, there’s what you tell the model in the system prompt. Below that, there’s what you actually allow it to do through tool definitions and access control. At the bottom, there’s what your infrastructure will accept or reject. Most failures happen because teams only police the top layer and hope it sticks.

System Prompts Are Guardrails, Not Brakes

A constraint in your system prompt—“never delete data without confirmation”—is a suggestion, not a rule. The model can and will ignore it when it seems expedient. What actually works is making the capability itself impossible. If the agent shouldn’t delete without confirmation, don’t give it a delete tool at all. Give it an “archive” tool. Or make the delete tool require a second parameter: a confirmation code that only a human can provide.

The model can’t violate a constraint it doesn’t have the power to break.

Write Tool Definitions Like Contracts

Your tool definitions should be so precise that the model can’t reasonably misuse them. Vague is dangerous.

Bad:

{
  "name": "execute_query",
  "description": "Run a database query",
  "parameters": {
    "query": "string"
  }
}

The agent will execute whatever it wants, including DROP TABLE statements.

Better:

{
  "name": "fetch_user_data",
  "description": "Retrieve user records matching exact filters",
  "parameters": {
    "user_id": "integer, required",
    "fields": "array of enum: ['name', 'email', 'created_at'], default: all"
  },
  "constraints": "Only SELECT operations. Max 1000 rows returned. user_id must be positive integer."
}

You’ve eliminated a whole class of attacks by making the tool incapable of mutation. The enum on fields prevents over-fetching. You’ve bounded the output size.

Separate Autonomous Actions from Decisions

This one shifts alignment upstream. Don’t let an agent decide whether to send an email, refund a transaction, or modify production config. Let it prepare the action and request human approval. The constraint isn’t a prompt-level rule—it’s an architectural gate.

When the agent needs to take real action, route it through a human-in-the-loop step. Log what it wanted to do. Show the human exactly what would happen. Let the human click “approve” or “reject.”

For lower-stakes actions (writing a Slack message, updating a draft document, creating a ticket), you can afford to be looser. For anything with cost or irreversible consequences, keep the human in.

Use Capability Attestation, Not Trust

Grok 4.3 and Gemini 3.5 Flash both perform well on tool calling, but that doesn’t mean they’ll use your tools correctly in production. Before deploying an agent, run it against a set of adversarial scenarios that should trigger your constraints.

Example scenario: Agent is building an SQL query. Feed it: “Get me all users where name = ‘admin’ or ‘1’=‘1’”. Does it escape properly? Does it even notice the obvious injection? If it fails here, it will fail in production.

Set up a few dozen of these. Run your agent against them. Measure: Did it follow constraints? Did it reject invalid requests? Did it ask for clarification instead of guessing?

Only deploy after you’ve seen it fail gracefully on things you deliberately made ambiguous or dangerous.

Cost Asymmetry Is Your Friend

From Daily Signal 2026-07-08, the insight is crisp: stop using confidence thresholds to decide when an agent acts autonomously. Instead, weight the cost of being wrong.

If the agent is writing a comment in your codebase, the cost of being wrong is low. It’s a draft; a human reviews it. Let the agent be aggressive.

If the agent is firing engineers or transferring money, the cost of being wrong is catastrophic. The agent shouldn’t be autonomous at all—or it should only work on a staging environment with a human review gate.

Make this explicit in your tool definitions:

{
  "name": "modify_employee_record",
  "description": "Update employee data",
  "cost_of_error": "high",
  "requires_human_approval": true,
  "approval_threshold": "must be reviewed by HR before execution"
}

Logging and Audit Trails Are Constraints Too

An agent can’t explain why it violated a rule if there’s no record of what it did. Build logging into your tool execution layer, not the model layer. Every call, every parameter, every result should be logged before the agent moves on.

When the agent does something wrong (and it will), you need a replay mechanism. What input did it see? What did it decide? What happened when it executed? This isn’t just for debugging—it’s how you catch drift before it becomes a systemic failure.

Test Against Your Actual Domain

Generic agent benchmarks don’t catch domain-specific failures. If you’re building an agent for financial operations, it needs to know your chart of accounts, your approval workflows, your regulatory constraints. It won’t learn this from the default system prompt.

Build a test harness with real (or anonymized real) scenarios from your domain. Feed them to the agent. Score not just on correctness but on alignment: Did it follow the company’s finance policy? Did it escalate ambiguous cases? Did it refuse tasks outside its scope?

Deploy once you’re passing 95%+ of domain-specific tests. Not industry benchmarks. Your tests.

Constraints Decay

An agent that respects constraints today won’t necessarily respect them in three months. Model capability improves. Users find new ways to trick the system. Your business rules change. Schedule quarterly constraint audits. Run your adversarial scenarios again. Measure drift.

If the agent starts ignoring constraints, you don’t retrain it. You tighten the architectural gates. Add a new approval step. Remove a tool permission. Make the constraint structural, not aspirational.

Bottom line: Specify constraints architecturally through tool design, capability access control, and approval gates—not through prompts alone. Then validate your agent against adversarial scenarios in your actual domain before deploying.

Question via Hacker News