Office Hours — How should you evaluate and design coding interviews to fairly assess candidates when they have access to AI tools?

How should you evaluate and design coding interviews to fairly assess candidates when they have access to AI tools?

The honest answer is that the entire premise of “fair assessment” has shifted, and most interview processes haven’t caught up. You’re not evaluating the same skill anymore when candidates can offload syntax, boilerplate, and even architecture sketches to Claude Opus 5 or GPT-5.6 Sol. The question isn’t whether to allow AI—it’s already happening, and candidates who aren’t using it are at a disadvantage. The real question is what you’re actually hiring for when the coding work itself is no longer the bottleneck.

The skill you thought you were testing is gone

Traditional coding interviews tested pattern recognition, syntax recall, ability to implement a known algorithm under pressure, and real-time debugging. An LLM with computer use (Claude Code Auto Mode, Gemini 3.5 Flash, GPT-5.6 Sol) handles most of that in seconds. If your interview is a 45-minute LeetCode problem, a candidate with access to frontier models can generate a working solution, explain it coherently, and even optimize it—while you think they’re demonstrating mastery of data structures.

This doesn’t mean they’re unprepared. It means you’re measuring something different than you think. The candidate who can critique the AI’s solution, catch its hallucinations, and navigate tradeoffs is actually demonstrating more engineering judgment than someone who codes from scratch. But if your rubric is “did they write correct code,” you’ve lost signal.

What to actually test

Shift to skills that AI doesn’t substitute for and that separate strong engineers from mediocre ones:

Problem decomposition and communication. Before code is written, can the candidate articulate the problem? Do they ask clarifying questions? Do they sketch out trade-offs? The work here is linguistic and conceptual, not syntactic. An engineer who can whiteboard a system design or talk through edge cases without jumping to implementation is demonstrating judgment. Pair this with a take-home where they have to explain why their solution works, not just that it runs.

Code reading and critique. Give them a real (or realistic) piece of code—something a junior might write, or something an LLM-assisted codebase might produce—and ask them to identify bugs, performance issues, or maintainability problems. This directly tests the skill of “working in a codebase with AI-generated code.” Can they spot when an LLM hallucinated a library call? Can they see the performance cliff in an algorithm? This is the inverse of code generation and harder to fake.

Debugging under uncertainty. The classic “here’s a failing test, fix it” interview. The twist: the failure is subtle (a race condition, an off-by-one error in a specific input range, or behavior that only emerges under load). AI models are bad at this because they operate in a single forward pass. A strong engineer narrows the problem space systematically. This is where intuition and experience actually shine.

Design and tradeoffs. Ask them to design a component or system and justify their choices. Which database? Which caching strategy? Why not microservices? Where does this approach break? AI can generate plausible-sounding designs. Good engineers can defend or tear down a design. Force them to defend it against reasonable push-back.

Integration with reality. The best test: give them a task that requires iterating with feedback. “Build a feature where the requirements clarify as you go.” This could be a small web app, a data pipeline, or a system integration. Make it open-ended. An AI-assisted engineer will leverage tools effectively and move fast. An engineer who can’t direct that effort will flounder. You’re measuring judgment about when to use AI and how to verify the output, not raw coding speed.

Practical interview structure

Here’s a concrete approach:

Async take-home (90 min, AI allowed). Give a problem that’s intentionally open-ended. “Build a small feature. You can use any tools you want, including LLMs. We’ll review code and ask you to explain trade-offs.” This removes time pressure and lets you see how they actually work. You’ll learn more from a well-structured solution with AI assistance than a half-working thing under a clock.
Synchronous code review (30 min). Walk through their solution together. Ask “why this library?” “What would you do differently with twice as much time?” “Where’s this approach fragile?” Push them to defend decisions. If they can’t articulate why, that’s signal.
Debugging or problem-solving pairing (45 min). Introduce a bug or a new requirement. Let them use AI if they want—most will. Watch how they use it. Do they blindly accept the LLM’s first answer, or do they verify? Can they spot when the AI is confident but wrong? Can they ask targeted follow-up questions?
Design interview (45 min). A systems or architecture question. No code required. “Design a feature for millions of users. Walk me through your thinking.” AI doesn’t excel at this because you’re directly observing reasoning, not just output. You’ll feel the difference between someone regurgitating a template and someone thinking.

What not to do

Don’t pretend AI isn’t in the room. Some companies explicitly ban LLM use during interviews, which is performative and counterproductive. You’re selecting for people who can work without tools, not people who are good at the job (which requires tools). That’s bad hiring.

Don’t test syntax or memorization. That’s already extinct. Anyone who uses an IDE with Copilot or a terminal with command suggestions knows that recall isn’t a differentiator.

Don’t measure raw speed. A candidate who ships a solution in 20 minutes using AI might be demonstrating superior judgment and tool fluency compared to someone who takes 60 minutes to code it solo. Speed isn’t skill; judgment is.

The calibration problem

Your interview will feel easier to candidates—because it is. They’ll have access to Claude Opus 5 or GPT-5.6 Sol during the take-home. Some will produce work that looks great at first glance. Your job is to dig into why they made each choice, and that’s where the signal emerges. A mediocre engineer with AI can produce a plausible-looking system. A strong engineer with AI produces a well-reasoned system and can defend it.

Set your bar on judgment and communication, not implementation polish. The code doesn’t have to be perfect—it has to be justified.

Bottom line: Stop testing coding speed and syntax recall. Test decomposition, critique, debugging intuition, and design reasoning. Evaluate candidates in environments they’ll actually work in (with AI tools available), and measure their ability to verify and direct the output, not their ability to produce it from scratch.

Question via Hacker News