Every major fine-tuning approach compared against RAG and long-context prompting — with architecture patterns, cost math, latency profiles, and a decision framework for choosing the right one.

Fine-Tuning vs RAG vs Long Context: Decision Framework with Real Tradeoffs

Fine-tuning, retrieval-augmented generation, and long-context prompting solve overlapping but distinct problems. Teams routinely pick the wrong one — burning weeks on fine-tuning when RAG would suffice, or stuffing 200k tokens into a prompt when a fine-tuned model would answer in 50ms at a fraction of the cost. The decision depends on data characteristics, latency budgets, cost constraints, and how often the underlying knowledge changes.

This post covers the mechanics of each approach, when each wins, when each fails, and how to combine them in production architectures.

The Three Approaches at a Glance
Fine-Tuning: What It Actually Does
RAG: Architecture and Retrieval Quality
Long Context: The Brute-Force Option
Latency Profiles
Cost Math
Quality and Accuracy Comparison
The Decision Framework
Hybrid Architectures
Common Failure Modes
Implementation Patterns
Summary
Further Reading

The Three Approaches at a Glance

Dimension	Fine-Tuning	RAG	Long Context
Knowledge freshness	Frozen at training time	Real-time (limited by index lag)	Real-time (whatever fits in the window)
Upfront cost	High (data prep + training runs)	Medium (embedding pipeline + vector store)	Low (just pay per token)
Per-query cost	Low (smaller model, fewer tokens)	Medium (retrieval + generation)	High (all tokens processed every call)
Latency	Lowest (internalized knowledge)	Medium (retrieval + generation)	Highest (proportional to context length)
Max knowledge size	Unlimited (absorbed into weights)	Unlimited (index can scale to billions of chunks)	Limited by context window
Accuracy on trained tasks	Highest	Depends on retrieval quality	Depends on needle-in-haystack ability
Setup complexity	Moderate to high	High (chunking, retrieval, reranking)	Minimal

The three approaches aren’t mutually exclusive — most production systems combine at least two.

Fine-Tuning: What It Actually Does

Fine-tuning updates a model’s weights on task-specific data. The model doesn’t “memorize” documents in the way a database stores rows — it adjusts internal representations so that certain input patterns produce certain output patterns with higher probability. This distinction matters: fine-tuning teaches behavior and style more reliably than it teaches facts.

What Fine-Tuning Is Good For

Output format and style: Matching a specific tone, structure, or schema. A model fine-tuned on 500 examples of your company’s incident reports will produce incident reports that look right without elaborate prompting.
Task-specific reasoning: Classification, extraction, routing. Fine-tuned models on narrow tasks frequently outperform larger general-purpose models prompted to do the same thing.
Latency reduction: A fine-tuned smaller model can replace a larger model with a long system prompt, cutting both latency and cost.
Distillation: Training a smaller model to mimic a larger model’s outputs on specific tasks. A fine-tuned smaller model on GPT-5.5 outputs for a classification task can match 90%+ of the larger model’s accuracy at 10x lower cost.

What Fine-Tuning Is Bad For

Frequently changing knowledge: Retraining is slow and expensive. If product prices change weekly, fine-tuning is the wrong tool.
Factual recall of rare information: Models absorb general patterns, not encyclopedic entries. Fine-tuning on a 10,000-page technical manual doesn’t guarantee the model can cite page 4,327 correctly.
Small datasets: Below ~100 high-quality examples, fine-tuning tends to overfit or produce minimal improvement over good prompting.

Fine-Tuning Methods Compared

Method	Parameters Updated	Typical Cost	Training Time	When to Use
Full fine-tuning	All	$$$$	Hours to days	Maximum quality, large datasets, own infrastructure
LoRA / QLoRA	0.1-1% of params	$$	Minutes to hours	Most common choice; good quality/cost balance
Prefix tuning	Prepended soft tokens	$	Minutes	Lightweight; useful for task switching
API-based (OpenAI, Anthropic)	Provider-managed	$$-$$$	Minutes to hours	Simplest setup; no infrastructure management

LoRA (Low-Rank Adaptation) has become the default fine-tuning method for good reason: it trains adapter weights that are a fraction of the full model size, achieving 90-95% of full fine-tuning quality at a fraction of the compute cost. QLoRA adds 4-bit quantization of the base model during training, making it possible to fine-tune a 70B parameter model on a single GPU with 24GB VRAM.

The fine-tuning pipeline. Data preparation typically takes longer than training itself.

Practical Fine-Tuning Example

Fine-tuning via the OpenAI API for a classification task:

import openai

# Upload training data (JSONL format)
training_file = openai.files.create(
    file=open("training_data.jsonl", "rb"),
    purpose="fine-tune"
)

# Create fine-tuning job
job = openai.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="gpt-5.4",  # base model
    hyperparameters={
        "n_epochs": 3,
        "learning_rate_multiplier": 1.8
    }
)

Training data format matters more than quantity. 500 carefully curated examples with consistent formatting outperform 5,000 noisy ones. Each example should demonstrate exactly the input-output mapping desired, with no ambiguity in the target output.

RAG: Architecture and Retrieval Quality

RAG separates knowledge storage from reasoning. The model generates answers using retrieved context rather than relying on internalized knowledge. This makes knowledge updates instant (re-index the document) rather than requiring retraining.

The RAG Pipeline

Standard RAG pipeline. Retrieval quality is the bottleneck — a perfect generator can’t fix bad retrieval.

Retrieval Quality Is Everything

The most common RAG failure isn’t the LLM — it’s bad retrieval. If the relevant chunk isn’t in the top-k results, the model either hallucinates or says it doesn’t know. Retrieval quality depends on:

Chunking strategy: Chunks that are too small lose context. Chunks that are too large dilute relevance. For most document types, 512-1024 tokens with 10-20% overlap works well. Semantic chunking (splitting at natural topic boundaries) outperforms fixed-size chunking but is harder to implement.

Embedding model choice: The embedding model determines what “similar” means. Current-generation embedding models from providers like Cohere and open-source families like BGE score well on MTEB benchmarks, but benchmark performance doesn’t always predict domain-specific retrieval quality. Always evaluate on your own data.

Hybrid search: Combining dense vector search with sparse keyword search (BM25) catches cases where semantic similarity misses exact term matches. A query for “error code E-4472” needs keyword matching; a query for “database connection failures” needs semantic matching. Production RAG systems should use both.

Reranking: A cross-encoder reranker applied to the top-50 retrieval results before passing top-5 to the LLM consistently improves answer quality. Reranking models like Cohere Rerank 4.0 and cross-encoder models from the sentence-transformers library are common choices. The latency cost is 20-50ms — almost always worth it.

Retrieval Method	Recall@10 (typical)	Latency	Best For
Dense vector only	70-80%	10-30ms	Semantic queries
BM25 only	60-75%	5-15ms	Keyword/exact-match queries
Hybrid (dense + BM25)	80-90%	20-40ms	General-purpose
Hybrid + reranking	85-95%	50-100ms	Quality-critical applications

RAG Failure Modes

Chunk boundary problems: The answer spans two chunks, but only one is retrieved. Overlapping chunks mitigate this but don’t eliminate it.
Multi-hop reasoning: “What was the revenue of our largest customer’s parent company?” requires chaining multiple retrievals. Standard RAG does one retrieval step.
Metadata filtering failures: Retrieval returns semantically similar but wrong-entity results. A query about “Q3 2025 revenue” retrieves a chunk about “Q3 2024 revenue” because the text is nearly identical. Metadata filters (date, entity, document type) are essential.
Index staleness: Documents are updated but the index isn’t re-embedded. Incremental indexing pipelines need monitoring.

RAG sophistication spectrum. Most production systems should be at the “Advanced” stage at minimum.

Long Context: The Brute-Force Option

Long-context prompting puts everything into the prompt window and lets the model figure it out. No chunking, no embedding, no retrieval pipeline. Just concatenate the documents and ask the question.

Modern context windows have made this viable for surprisingly large document sets:

Model	Max Context Window
Gemini 3.5 Pro (limited preview only; not GA)	2M tokens
Open-source models with extended context	Up to 10M tokens
Open-source frontier-class models	1M tokens
Claude Opus 4.8	1M tokens (API) / 200K tokens (default)
GPT-5.6 Sol (preview)	Context not yet specified

When Long Context Works

One-shot analysis: Analyzing a codebase, contract, or dataset that fits in the window. No pipeline to build — just paste and ask.
Prototyping: Before investing in RAG infrastructure, long context lets you validate that the task is feasible at all.
Small, stable document sets: If total knowledge fits in 50-100k tokens and doesn’t change often, long context avoids the complexity of RAG entirely.
Complex cross-references: When the answer requires synthesizing information across many parts of a document, long context avoids the chunk-boundary problem that plagues RAG.

When Long Context Fails

Cost scales linearly (or worse) with context size. Every query processes the entire context. If the document set is 500k tokens and the query rate is 1,000/day, that’s 500M input tokens per day. At frontier model pricing, this becomes expensive fast.

Latency scales with context. Time-to-first-token increases with context length. A 200k-token prompt on Claude Opus 5 has noticeably higher latency than a 5k-token prompt.

Needle-in-haystack degradation. Models don’t attend equally to all parts of long contexts. Information in the middle of very long contexts is retrieved less reliably than information at the beginning or end. This is the “lost in the middle” problem, and while current-generation models handle it better than their predecessors, it hasn’t been fully solved.

No update mechanism. Changing one fact means reprocessing the entire context on the next query. RAG can re-embed a single document.

Document set size as a rough guide for approach selection.

Prompt Caching Changes the Math

Prompt caching (available from OpenAI, Anthropic, and Google) stores the processed KV cache of static prompt prefixes. Subsequent queries with the same prefix skip the prefill computation, reducing both latency and cost.

This directly benefits long-context use cases: if the same 200k-token document set is queried repeatedly, the first query pays full price, but subsequent queries pay only for the cache read (typically 10-25% of the input token price) plus the new query tokens. The cache typically expires after 5-15 minutes of inactivity, depending on the provider.

With prompt caching, long context becomes viable for repeated-query scenarios that would otherwise be cost-prohibitive. The break-even point depends on query frequency: if queries arrive more often than the cache TTL, caching pays for itself.

Prompt caching lifecycle. High query frequency keeps the cache warm and amortizes the initial prefill cost.

Latency Profiles

Latency matters differently depending on the use case. A chatbot needs sub-second time-to-first-token. A batch analysis pipeline can tolerate 30 seconds per request.

Approach	Time to First Token (typical)	End-to-End (typical)	Primary Latency Driver
Fine-tuned model (short prompt)	50-150ms	200-800ms	Model inference only
RAG (hybrid + rerank)	200-500ms	500-2000ms	Retrieval + reranking + generation
Long context (50k tokens)	300-800ms	1-3s	Prefill computation
Long context (200k tokens)	1-3s	3-10s	Prefill computation
Long context (200k, cached)	100-300ms	500-2000ms	Cache read + generation

Fine-tuning wins on latency when the model has internalized the knowledge and doesn’t need extensive context. A fine-tuned GPT-5.4 Nano answering domain-specific classification questions operates at 50-100ms TTFT with minimal prompt overhead.

RAG adds retrieval latency but keeps generation prompts short. The retrieval step (embedding the query, searching the index, reranking) typically takes 50-100ms. The generation step processes only the retrieved chunks (2-5k tokens typically) rather than the entire document set.

Long context pays the prefill cost on every uncached query. Gemini 3.5 Flash partially mitigates this with ~4x faster output than other frontier models, making it the strongest choice when long context is necessary and latency matters.

Cost Math

Real cost comparisons need concrete numbers. The following analysis uses a standardized scenario: answering questions about a 100-page technical document (~75,000 tokens) at 1,000 queries per day, with an average response of 500 tokens.

Fine-Tuning Cost

Upfront: Data preparation (engineering time, 2-5 days), training run ($10-500 depending on model size and provider), evaluation and iteration (2-3 training runs typical).

Per-query: Only the query prompt + response tokens. If the fine-tuned model needs minimal context (system prompt + query ≈ 500 tokens input):

1,000 queries × (500 input + 500 output tokens) = 1M tokens/day
At budget model pricing (GPT-5.4 Nano: $0.2/M input, $1.25/M output): $0.70-1.50/day

RAG Cost

Upfront: Embedding pipeline, vector database, chunking logic, reranking integration. Engineering time: 1-3 weeks.

Per-query: Embedding the query (~$0.00001), vector search (negligible at this scale), reranking ($0.001-0.002 per query), generation with 3-5k tokens of context:

1,000 queries × (4,000 input + 500 output tokens) = 4.5M tokens/day
At mid-tier model pricing: $2-10/day (generation) + $1-2/day (reranking) + vector DB hosting ($20-100/month)

Long Context Cost (No Caching)

Upfront: Minimal. Format the document, write the system prompt.

Per-query: Full document in every prompt:

1,000 queries × (75,000 input + 500 output tokens) = 75.5M tokens/day
At frontier model pricing ($2-15/M input): $150-1,130/day

Long Context Cost (With Caching)

Assuming the cache stays warm (queries every few minutes):

First query: 75,000 input tokens at full price
Subsequent 999 queries: 75,000 tokens at cache read price (10-25% of input price) + query tokens
Effective daily cost: $15-280/day (depending on provider and cache hit rate)

The Break-Even Calculation

Fine-tuning has higher upfront cost but lower marginal cost. The break-even point against RAG:

Fine-tuning setup: ~$2,000-5,000 (engineering time + training)
RAG setup: ~$5,000-15,000 (engineering time + infrastructure)
Fine-tuning daily cost: ~$1
RAG daily cost: ~$7

Fine-tuning breaks even against RAG on marginal cost from day one — but only if the knowledge doesn’t need updating. If knowledge changes monthly, add $500-1,000 per retraining cycle (including data prep and evaluation).

The real question isn’t which is cheapest in isolation. It’s which meets accuracy requirements at acceptable cost and latency, given the knowledge update frequency.

Quality and Accuracy Comparison

Factual Accuracy

RAG with good retrieval produces the most factually grounded answers because the source text is in the prompt. The model can quote directly. Fine-tuned models can confabulate details that feel right but aren’t — they’ve learned patterns, not exact facts.

Long context also provides grounded answers when the information is in the window, but accuracy degrades with context length. At 200k tokens, current models miss information that appears in the middle 10-20% more often than information at the edges.

Consistency of Output Format

Fine-tuning dominates here. A model fine-tuned on structured outputs (JSON, specific report formats, consistent categorization schemas) produces conformant output with near-100% reliability. RAG and long context rely on prompting to enforce format, which works most of the time but has failure modes at the tails.

Complex Reasoning

Long context has an advantage for questions requiring synthesis across many document sections. RAG is limited to the retrieved chunks, which may miss relevant context. Fine-tuning can learn reasoning patterns but not dynamic cross-referencing of specific documents.

For multi-hop reasoning (“Find all contracts expiring in Q4 that reference the indemnification clause from the master agreement”), RAG struggles because it requires multiple coordinated retrievals. Long context handles this naturally if the documents fit. Fine-tuning can’t handle this at all for novel documents.

Quality Dimension	Fine-Tuning	RAG	Long Context
Factual grounding	Medium (pattern-based)	High (source in prompt)	High (source in prompt)
Format consistency	Highest	Medium	Medium
Cross-document synthesis	Low	Low-Medium	High
Handling novel documents	Low (needs retraining)	High	High
Hallucination rate	Medium	Low (with citations)	Low-Medium
Edge-case robustness	High (if trained on edges)	Depends on retrieval	Depends on attention

The Decision Framework

Start Here: Three Questions

1. How often does the knowledge change?

Daily or more → RAG or long context
Weekly to monthly → RAG (preferred) or periodic fine-tuning
Rarely or never → Fine-tuning is viable

2. How large is the knowledge base?

< 50k tokens → Long context (simplest solution)
50k-500k tokens → RAG or long context with caching
> 500k tokens → RAG (required)
> 10M documents → RAG with tiered indexing

3. Is the task about behavior/style or knowledge/facts?

Behavior (classification, formatting, tone) → Fine-tuning
Facts (answering questions from documents) → RAG or long context
Both → Hybrid

The decision flows through three filters. Each narrows the viable options.

Decision Matrix

Scenario	Recommended Approach	Reasoning
Customer support bot, 500 FAQ entries, updated weekly	RAG	Knowledge changes frequently, structured retrieval works well
Code review tool, enforce company style guide	Fine-tuning	Behavioral task, style guide rarely changes
Legal contract analysis, varies per client	Long context (if fits) or RAG	Novel documents each time, cross-reference needed
Medical diagnosis support, large knowledge base	RAG + fine-tuning	Fine-tune for reasoning patterns, RAG for current guidelines
Email classification into 20 categories	Fine-tuning	Pure classification, no external knowledge needed
Internal search over 100k documents	RAG	Too large for context window, needs to scale
Chatbot with personality and tone requirements	Fine-tuning	Behavioral task
Financial report generation from live data	RAG	Data changes constantly
Analyzing a single uploaded document	Long context	One-shot task, document fits in window
Summarize 50 documents on the same topic	Long context (if fits) or RAG + map-reduce	Cross-document synthesis needed

Hybrid Architectures

The best production systems combine approaches. Hybrids aren’t complexity for its own sake — each layer addresses a specific limitation of the others.

Pattern 1: Fine-Tuned Model + RAG

Fine-tune for output format, tone, and reasoning style. Use RAG for factual grounding. This is the most common production pattern.

Pattern 1: The fine-tuned model provides consistent behavior while RAG provides fresh knowledge.

Example: A customer support system fine-tuned on 2,000 examples of ideal support responses (matching company tone, following resolution playbooks) with RAG over the current product documentation and known issues database. The fine-tuning handles how to respond; RAG handles what to say.

# Pseudocode for fine-tuned model + RAG
def answer_support_query(query: str) -> str:
    # RAG retrieval
    chunks = retriever.search(query, top_k=5, rerank=True)
    context = "\n---\n".join([c.text for c in chunks])
    
    # Generate with fine-tuned model
    response = client.chat.completions.create(
        model="ft:gpt-5.4:company:support-v3",  # Replace with your fine-tuned model ID
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Context:\n{context}\n\nQuery: {query}"}
        ],
        temperature=0.3
    )
    
    # Attach citations
    return add_citations(response.choices[0].message.content, chunks)

Pattern 2: RAG with Long-Context Fallback

Use RAG as the primary retrieval mechanism. When retrieval confidence is low (no chunks score above a threshold), fall back to long-context processing of the most relevant documents.

Pattern 2: RAG handles the common case; long context handles the edge cases where retrieval isn’t confident.

This pattern captures the best of both: RAG’s efficiency for straightforward lookups and long context’s ability to handle complex, cross-referencing queries that chunk-based retrieval misses.

Pattern 3: Model Router

Route queries to different backends based on query characteristics. Simple factual lookups go to a fine-tuned small model. Complex analytical questions go to RAG with a frontier model. Document-specific questions go to long context.

Pattern 3: A lightweight classifier routes queries to the most cost-effective backend.

The router itself can be a fine-tuned small model or even a rule-based system. GPT-5.4 Nano fine-tuned on a few hundred labeled routing examples achieves 95%+ routing accuracy and adds <50ms latency. The cost savings from routing simple queries away from frontier models typically 5-10x the router’s operating cost.

Pattern 4: Agentic RAG with Iterative Retrieval

For complex queries, an agent plans retrieval steps, executes them, evaluates results, and iterates. This handles multi-hop reasoning that single-shot RAG misses.

# Simplified agentic RAG loop
def agentic_rag(query: str, max_steps: int = 3) -> str:
    context = []
    remaining_question = query
    
    for step in range(max_steps):
        # Plan retrieval
        search_queries = planner.decompose(remaining_question, context)
        
        # Execute retrieval
        for sq in search_queries:
            results = retriever.search(sq, top_k=3)
            context.extend(results)
        
        # Check if we have enough to answer
        assessment = evaluator.assess(query, context)
        if assessment.sufficient:
            break
        remaining_question = assessment.follow_up_query
    
    return generator.answer(query, context)

This adds latency (each step is a retrieval + LLM call) but handles queries like “Compare our Q3 revenue growth rate to the industry average mentioned in the analyst report” that require finding multiple pieces of information across different sources.

Common Failure Modes

Fine-Tuning Failures

Overfitting to training distribution: The model performs well on examples similar to training data and poorly on anything else. Mitigation: diverse training examples, held-out evaluation sets that test distribution edges.

Catastrophic forgetting: Fine-tuning on a narrow task degrades the model’s general capabilities. A model fine-tuned exclusively on medical text may struggle with basic math. Mitigation: mix in general-purpose examples (5-10% of training data), use LoRA instead of full fine-tuning.

Stale knowledge: The fine-tuned model confidently answers with outdated information. Users trust it because it sounds authoritative. Mitigation: combine with RAG for factual queries, add a “knowledge cutoff” disclaimer, retrain on schedule.

RAG Failures

Retrieval misses: The relevant chunk isn’t retrieved. The model either hallucinates an answer or says “I don’t know” when the answer exists in the knowledge base. Mitigation: hybrid search, reranking, chunk overlap, query expansion.

Context window overflow: Too many retrieved chunks exceed the model’s effective processing capacity, even within the token limit. Retrieving 20 chunks to “be safe” dilutes the relevant information. Mitigation: strict top-k (3-5 chunks), reranking to ensure relevance.

Citation hallucination: The model generates a response and attributes it to the retrieved context, but the attribution is wrong — the cited chunk doesn’t actually support the claim. Mitigation: post-generation verification, extractive citation matching.

Long Context Failures

Cost blowup: A developer prototypes with long context, it works, it ships to production, and the monthly bill arrives. Long context costs are easy to underestimate because they scale with query volume, not just document size. Mitigation: cost monitoring, prompt caching, migration plan to RAG for high-volume queries.

Lost in the middle: The model misses information positioned in the middle of a very long context. Mitigation: place the most important context at the beginning and end, or use structured headers that the model can navigate.

Prompt injection surface: Every token in the context is a potential prompt injection vector. A 200k-token context from user-uploaded documents has a large attack surface. Mitigation: input sanitization, output filtering, boundary markers between context and instructions.

Implementation Patterns

When to Start with Long Context and Migrate

For new projects, long context is often the right first step:

Day 1: Concatenate documents into a prompt. Validate that the task is feasible. Measure quality.
Week 2: If quality is acceptable and query volume is low (<100/day), ship it with prompt caching.
Month 2: If query volume grows, build the RAG pipeline using the long-context system as a quality baseline.
Month 3+: If format/behavior consistency matters, add fine-tuning.

This approach validates the use case before investing in infrastructure. The long-context system serves as a ground truth for evaluating RAG retrieval quality — if RAG produces different answers than long context on the same documents, the retrieval pipeline probably has gaps.

RAG Infrastructure Checklist

A production RAG system needs more than a vector database and an LLM:

# Core components for production RAG
class RAGSystem:
    def __init__(self):
        self.embedder = EmbeddingModel("voyage-3-large")     # Embedding model
        self.vector_db = QdrantClient(url="...")               # Vector store
        self.sparse_index = BM25Index()                        # Keyword search
        self.reranker = CohereReranker(model="rerank-v3.5")   # Reranker
        self.cache = SemanticCache(ttl=3600)                   # Query cache
        self.monitor = LangfuseClient()                        # Observability
    
    def query(self, question: str) -> Response:
        # Check semantic cache first
        cached = self.cache.get(question)
        if cached:
            return cached
        
        # Hybrid retrieval
        dense_results = self.vector_db.search(
            self.embedder.embed(question), limit=50
        )
        sparse_results = self.sparse_index.search(question, limit=50)
        merged = reciprocal_rank_fusion(dense_results, sparse_results)
        
        # Rerank
        reranked = self.reranker.rerank(question, merged[:50], top_n=5)
        
        # Generate
        response = self.generate(question, reranked)
        
        # Log for monitoring
        self.monitor.log(question, reranked, response)
        
        # Cache result
        self.cache.set(question, response)
        
        return response

Fine-Tuning Data Preparation

The quality bar for fine-tuning data is higher than most teams expect. Common mistakes:

Inconsistent formatting: Some examples use markdown, others use plain text, others use HTML. The model learns the inconsistency.
Ambiguous labels: Classification examples where two reasonable people would disagree on the label. These confuse the model during training.
Missing edge cases: Training exclusively on happy-path examples. The model doesn’t learn how to handle malformed input, out-of-scope queries, or adversarial inputs.

A reliable data preparation process:

# Fine-tuning data validation
def validate_training_example(example: dict) -> list[str]:
    issues = []
    
    # Check required fields
    if "messages" not in example:
        issues.append("Missing 'messages' field")
        return issues
    
    messages = example["messages"]
    
    # Check message structure
    roles = [m["role"] for m in messages]
    if roles[0] != "system":
        issues.append("First message should be system prompt")
    if roles[-1] != "assistant":
        issues.append("Last message should be assistant response")
    
    # Check response length (too short = low quality)
    assistant_content = messages[-1]["content"]
    if len(assistant_content) < 50:
        issues.append(f"Assistant response suspiciously short: {len(assistant_content)} chars")
    
    # Check for format consistency
    if not matches_output_schema(assistant_content):
        issues.append("Response doesn't match expected output format")
    
    return issues

# Run validation
examples = load_jsonl("training_data.jsonl")
for i, ex in enumerate(examples):
    issues = validate_training_example(ex)
    if issues:
        print(f"Example {i}: {issues}")

Evaluation: Comparing Approaches on Your Data

Before committing to an architecture, run a comparative evaluation:

# Comparative evaluation framework
import json
from dataclasses import dataclass

@dataclass
class EvalResult:
    approach: str
    query: str
    response: str
    latency_ms: float
    input_tokens: int
    output_tokens: int
    cost_usd: float
    human_score: float | None = None  # 1-5 scale, filled in later

def evaluate_approaches(test_queries: list[str], ground_truth: list[str]):
    results = []
    
    for query, truth in zip(test_queries, ground_truth):
        # Fine-tuned model
        ft_result = run_fine_tuned(query)
        results.append(EvalResult("fine-tuned", query, ft_result.text, 
                                   ft_result.latency, ft_result.input_tokens,
                                   ft_result.output_tokens, ft_result.cost))
        
        # RAG
        rag_result = run_rag(query)
        results.append(EvalResult("rag", query, rag_result.text,
                                   rag_result.latency, rag_result.input_tokens,
                                   rag_result.output_tokens, rag_result.cost))
        
        # Long context
        lc_result = run_long_context(query)
        results.append(EvalResult("long-context", query, lc_result.text,
                                   lc_result.latency, lc_result.input_tokens,
                                   lc_result.output_tokens, lc_result.cost))
    
    # Automated metrics
    for r in results:
        r.automated_score = compute_similarity(r.response, 
                                                ground_truth[test_queries.index(r.query)])
    
    return results

Run this on 50-100 representative queries. The results almost always surprise: one approach dominates on certain query types while failing on others, which is exactly the signal needed to design a hybrid architecture or router.

Summary

Fine-tuning is best for behavioral tasks (style, format, classification) with stable knowledge. It offers the lowest per-query cost and latency but can’t handle dynamic knowledge without retraining. Start with LoRA on a small model and evaluate against prompted baselines — many teams fine-tune when good prompting would suffice.

RAG is best for large, dynamic knowledge bases where factual accuracy matters. The engineering investment is higher (chunking, retrieval, reranking, monitoring), but it scales to millions of documents and updates instantly. Retrieval quality is the bottleneck; invest in hybrid search and reranking before upgrading the generator model.

Long context is best for one-shot analysis, prototyping, and small document sets. Prompt caching has shifted the cost equation meaningfully — high-frequency queries over static context are now affordable. But cost still scales linearly with document size × query volume, making it impractical for large-scale production workloads without a migration path.

The hybrid approach wins in production. Fine-tune for behavior, RAG for knowledge, long context for session-specific documents, and a router to send each query to the cheapest backend that meets quality requirements. Build the simplest version first (usually long context), validate quality, then add complexity only where the numbers demand it.