Library of the Week — Mirascope

Mirascope — wait, already covered. Let me pick a fresh one.

Parea AI — nope, let me find one that fits.

---

inspect-ai — a rigorous evaluation framework for LLM safety and capability research

GitHub · Language: Python · License: MIT

What it does

inspect-ai is an open-source evaluation framework built by the UK AI Safety Institute to run structured, reproducible capability and safety evaluations against LLMs. It handles the scaffolding of tasks, solvers, scorers, and logging so teams can focus on writing eval logic rather than plumbing. It’s aimed at researchers and developers who need auditable, dataset-driven evals rather than vibe checks.

Why it stands out

• First-class task abstraction: Evals are composed from Dataset → Solver → Scorer pipelines, making them composable and testable in isolation — a cleaner design than most eval frameworks that conflate these concerns
• Built-in model-graded scoring: Supports LLM-as-judge natively, with configurable critique prompts, making it straightforward to evaluate free-form outputs against rubrics without glue code
• Multi-provider, single interface: Works with GPT-5.6 Sol, Claude Opus 5, Gemini 3.7 Flash, local models via Ollama, and others through a unified get_model() call — swapping providers is a one-liner
• Reproducibility baked in: Every run produces a structured JSON log with full prompt/response traces, making it easy to diff eval runs across model versions or prompt changes

Quick start

from inspect_ai import task, eval
from inspect_ai.dataset import example_dataset
from inspect_ai.scorer import model_graded_fact
from inspect_ai.solver import generate, prompt_template

@task
def my_eval():
    return Task(
        dataset=example_dataset("theory_of_mind"),
        solver=[
            prompt_template("Answer concisely: {question}"),
            generate(),
        ],
        scorer=model_graded_fact(),
    )

eval(my_eval(), model="openai/gpt-5.6-sol")

When to use it

• You’re running structured capability or safety evals and need reproducible, logged results you can hand to stakeholders or regulators
• You want to benchmark multiple providers (e.g. Claude Opus 5 vs. Gemini 3.7 Flash) against the same task suite without rewriting glue code
• Your team is adapting published safety benchmarks or building internal red-teaming pipelines

When to skip it

• If you need lightweight, ad-hoc prompt regression testing — promptfoo or a simple pytest harness is lower overhead
• The framework is Python-only and somewhat research-oriented; if your team just wants pass/fail unit tests on LLM outputs, it’s more abstraction than you need

The verdict

inspect-ai is the most carefully designed open-source eval framework available right now, and its pedigree (UK AISI) means it’s been pressure-tested against real safety research use cases rather than demo scenarios. If you’re doing anything beyond basic smoke testing — multi-step reasoning evals, model comparisons, or anything that needs an audit trail — it’s worth the setup cost over rolling your own.