Office Hours — Does your mind drift noticeably while waiting for long AI inference times, and how does that affect your coding workflow?

Does your mind drift noticeably while waiting for long AI inference times, and how does that affect your coding workflow?

Yes, and it’s worse than I expected. Not the drifting itself—that’s fine, even useful for stepping back—but the context switching cost when inference finally returns.

The Problem Isn’t the Wait, It’s the Return

When you hit submit on a long inference call and step away, you’re context-switching. You open Slack. You check the PR queue. You glance at a GitHub issue. Fifteen seconds later the response lands, but you’re now cognitively committed to the thing you switched to. When you turn back to the code you’re waiting for, there’s a tax: reloading the problem state, the specifics of what you asked for, the next three steps you had planned.

This is especially brutal with Claude Code or Cursor Agent. You send a multi-file refactor, expect 10-20 seconds of inference, and instinctively start context-hopping. The agent returns with a 200-line rewrite and you’re scrambling to reorient yourself before you can validate or iterate. That latency tax compounds across a session.

Where Latency Actually Kills Velocity

The real damage isn’t 10-second waits. It’s anything over ~5 seconds per task, because that’s the threshold where “I’ll just wait” stops being a credible strategy. Below that, you stay focused. Above that, your brain starts shopping for something else to do.

For coding agents specifically, this is brutal. Cursor Agent or Claude Code running on a complex task might take 20-40 seconds. You can’t meaningfully context-switch and come back coherent. So what happens is you either:

Stay mentally frozen, staring at the inference UI (which is exhausting and wastes cognitive energy that should be on validation).
Open another editor tab and start a parallel task, losing velocity on the primary task when the agent finishes.
Obsessively review docs or linter configs for the specific thing the agent is working on (this one actually works, but it’s artificial busywork).

The Model Selection Matters More Than You’d Think

Latency differences between frontier models are significant in practice. Claude Opus 5 on a simple code completion is usually 2-4 seconds. GPT-5.6 Sol on the same task often takes 6-12 seconds. Gemini 3.5 Flash is typically the fastest (1-3 seconds) but trades some reasoning depth. Over an 8-hour session, that difference compounds into an hour of lost context-switching tax.

For local models running on a 24GB GPU, you’re often hitting 15-30 seconds per inference, which makes agentic workflows nearly unviable without batching requests or scheduling them asynchronously.

Here’s a concrete example of how this cascades:

# Scenario: You're using Claude Code to refactor a test file
# Expected latency: 8 seconds (Claude Opus 5 + network)
# Actual workflow:

1. Send request to Claude Code at t=0s
2. At t=2s, you've stopped reading the prompt and your brain is idle
3. At t=4s, you instinctively check Slack to "stay productive"
4. At t=8s, Claude returns a 150-line rewrite and three new test cases
5. At t=9s, you're still parsing Slack and miss the first 10 seconds of the diff
6. At t=13s, you're back in the code but now have to re-read the request you sent
   to understand what the agent did and whether it's right
7. Total cognitive friction: ~5 seconds of pure overhead per agent call

Over 20 agent calls in a session, that's ~100 seconds (~1.7 minutes) of pure context-switching waste.

The Hidden Cost: Validation Breaks

The inference latency also corrupts your ability to validate incrementally. Good agentic workflows rely on fast feedback loops: agent makes a change, you review it in 2-3 seconds, you either approve or steer. If the agent takes 20 seconds to respond, you’re not iterating—you’re batching, which means you lose the ability to catch mistakes early and compound them into larger errors.

This is why Cursor Agent with Gemini 3.5 Flash (faster inference, smaller token output) often feels more productive than Claude Code with Opus 5, even though Opus 5 is technically more capable. The latency difference is noticeable enough that your actual iteration speed is higher.

What Actually Helps

Use fast models for scaffolding, frontier models for reasoning. Cursor Agent’s recent redesign (mentioned in Daily Signal 2026-07-26) uses cheaper models for execution once a frontier model has planned the task. This is smart not just for cost but for latency: the expensive reasoning happens once, the fast iteration happens in loops.
Batch asynchronous inference when you can. If you’re using an API client for analysis or document processing, don’t block on each call. Fire them all off, let them resolve in the background, and check results in a separate pass. You stay in flow on your primary task.
Keep your requests small. Long, complex prompts increase model thinking time and token output time. A tightly scoped “refactor this function’s error handling” is faster than “audit this entire service for logging issues.” Do the focused one.
Disable streaming for agent responses. Streaming tokens in real-time creates an illusion of progress but actually interrupts your attention mid-response. Let the full response land, then read it atomically.

Bottom line:

If your inference is consistently over 5 seconds, the context-switching tax is eating ~30-40% of your iteration speed. Either optimize for latency by switching to faster models (Gemini 3.5 Flash, Claude Sonnet 5) and smaller prompts, or structure your workflow to batch agent calls asynchronously and validate in bulk—don’t expect to stay focused while waiting.

Question via Hacker News