The Prompt Lab — The Prompt Lab — Confidence Calibration Prompting
Learn the the prompt lab — confidence calibration prompting prompting technique with concrete before/after examples.
The Prompt Lab — Confidence Calibration Prompting
The Technique
Confidence Calibration Prompting explicitly instructs the model to flag its own uncertainty at the claim level — distinguishing what it knows solidly from what it’s inferring or guessing. It works because frontier models like GPT-5.6 Sol and Claude Opus 5 have reasonable metacognitive awareness, but they suppress uncertainty signals by default to produce fluent, authoritative-sounding text. Asking them to surface that internal state gives you a more honest, actionable output.
The Naive Prompt
Summarize the key regulatory requirements for deploying an AI hiring tool
in the EU under the AI Act as of mid-2026.Why It Falls Short
AI Act implementation details are genuinely unsettled — enforcement timelines shifted multiple times through early 2026, and national competent authority guidance varies by member state. A model responding to this prompt will produce a confident-sounding summary that blends firm requirements with provisional guidance and educated inference, and you’ll have no idea which is which. In a compliance context, that’s not just unhelpful — it’s actively dangerous.
The Improved Prompt
Summarize the key regulatory requirements for deploying an AI hiring tool
in the EU under the AI Act as of mid-2026.
For each claim, tag it with one of three confidence markers:
- [HIGH] — well-established in the text of the AI Act itself
- [MED] — based on guidance documents or common interpretation,
but could vary or shift
- [LOW] — inferred, uncertain, or an area where you'd recommend
verifying with a primary source
After the summary, list any assumptions you made that a compliance
professional should double-check.Why It Works
By making confidence a structured output requirement rather than an implicit behavior, you force the model to slow down and evaluate each claim rather than smooth them all into the same confident register. The trailing “assumptions” section catches edge cases the confidence tags might miss — things the model didn’t even realize it was guessing about. You end up with a document that’s immediately triageable: read [HIGH] for orientation, flag [LOW] for your lawyer.
When to Use This
- High-stakes research and compliance work — anywhere a confidently wrong answer causes real harm (legal, medical, financial, security policy). Claude Fable 5’s strength on legal/medical benchmarks makes it a good base here, but even strong models benefit from this technique.
- Rapidly changing domains — AI regulation, geopolitics, drug approvals, model capabilities themselves. When training cutoffs and fast-moving reality diverge, explicit uncertainty tagging surfaces the gap rather than papering over it.
- When you’re not an expert in the domain — if you can’t evaluate the output yourself, confidence tags act as a proxy triage layer, telling you where to spend your verification effort before passing the output downstream.