Office Hours — Should I A/B test my LLM prompts in production or is that overkill?

Should I A/B test my LLM prompts in production or is that overkill?

It depends on your stakes and velocity. If you’re running a customer-facing feature where prompt quality directly affects conversion, retention, or support load, then yes, A/B test it. You’ll catch real behavior differences that eval sets miss because production has messier, more diverse inputs than your test cases.

But there’s a cheaper path first. Run a few hundred examples from your actual production logs through both prompt versions offline using Claude Opus 4.7 or GPT-5.5, score them with a rubric or a model-based evaluator, and see if one clearly wins. This takes maybe an hour and costs $5-10. If the results are ambiguous, then go live with A/B testing.

The offline eval workflow

Here’s what that actually looks like. Export 500-1000 recent requests from your logs. Create two prompt variants. Write a scoring rubric or use a model as a judge. Run both versions through the frontier model of your choice and compare outputs side-by-side.

Example rubric for a customer support classifier:

1. Does it correctly identify the issue category? (binary)
2. Does it extract the severity level accurately? (1-5)
3. Does it avoid hallucinating details not in the input? (binary)

Pipe your logs through both prompts, score, aggregate. If variant B scores 87% and variant A scores 84%, that’s meaningful. If they’re 85% vs 86%, you have no signal and need production data.

The cost math works out because you’re paying for tokens, not QPS. Five hundred evals at ~500 tokens each through Claude Opus 4.7 or GPT-5.5 runs under $10 and takes maybe twenty minutes of wall time.

When offline eval isn’t enough

Sometimes your metric isn’t something a rubric can capture. Downstream engagement, user edits, or escalation rates only emerge at scale. A prompt that scores well on your benchmark might subtly change user behavior in ways that hurt retention. That’s when A/B testing moves from optional to necessary.

The actual overhead of A/B testing is smaller than it used to be. You route a percentage of traffic to variant B, log which version was used, and measure downstream metrics (latency, user corrections, escalations, whatever matters). Just make sure your logging is clean from day one, because retrofitting it is painful.

Store the prompt version in your request context, include it in your observability pipeline, and correlate it with outcomes. Most APM tools handle this now. What kills A/B tests is forgetting to log the variant, then trying to match behavior back to which prompt ran it.

The detection problem

Where people screw up: they A/B test without defining success beforehand, or they don’t run the test long enough to detect real differences. With LLMs, you usually need at least a few thousand samples before patterns emerge. If your baseline escalation rate is 5% and variant B drops it to 4.8%, you need enough volume to see that signal. Run power analysis before you launch.

Also watch for interaction effects. A prompt change might help certain user segments while hurting others. Good A/B infrastructure lets you slice results by user cohort, request type, or time of day. If variant B wins overall but tanks for one customer segment, you’ve learned something valuable.

The test duration matters too. Run for at least one full business cycle (usually a week for most products) to smooth out day-of-week variation. LLM outputs don’t have the same variance as traditional feature flags, but user behavior does.

Bottom line: Skip A/B testing for low-stakes features. For anything that impacts user experience or business metrics, do quick offline evaluation first, then A/B test in production if that’s inconclusive. Define your success metric before you launch, allocate enough traffic to detect real effects, and keep your logging clean.

Question via Hacker News