Where randomness meets reason

Office Hours — How do I decide between building my own AI infrastructure versus using existing platforms and APIs? A daily developer question about AI/LLMs, answered with a direct, opinionated take. 2026-08-31T12:00:00.000Z Office Hours Office Hours office-hoursq-and-apractical-ai

Office Hours — How do I decide between building my own AI infrastructure versus using existing platforms and APIs?

A daily developer question about AI/LLMs, answered with a direct, opinionated take.

office-hoursq-and-apractical-ai
Daily One question from the trenches, one opinionated answer.

How do I decide between building my own AI infrastructure versus using existing platforms and APIs?

This is the decision tree that separates pragmatists from engineers who end up maintaining proprietary MLOps plumbing for five years. The answer hinges on three concrete factors: cost per transaction at your scale, latency requirements, and whether your competitive advantage actually lives in the model or in the moat around data and integration.

When APIs Make Sense (Most of the Time)

If you’re building a customer-facing application and your differentiator is speed to market or domain-specific features, use APIs. Full stop. Frontier models from OpenAI, Anthropic, and Google are already better than anything you’ll train in-house, and the operational overhead of self-hosting is brutal. A startup using Claude Opus 5 or GPT-5.6 Sol via API gets inference quality that would cost millions to replicate internally plus requires hiring ML infrastructure teams you don’t have.

The math is straightforward. Claude Opus 5 costs roughly $3–$15 per million tokens depending on input/output split. Anthropic’s new tokenizer emits about 30% more tokens for the same text, so plan for real costs around 40% higher than the list price suggests. If you’re processing 100M tokens monthly, you’re looking at $600K annually. That’s cheaper than one senior infrastructure engineer, let alone the entire team you’d need to run inference at scale. Add in the compute hardware (GPUs aren’t getting cheaper) and the maintenance burden, and APIs win for almost every non-hyperscale company.

Use APIs when:

  • You’re building B2B SaaS or user-facing products where inference latency under 5 seconds is acceptable
  • Your workload is variable or bursty (APIs scale automatically; GPUs sit idle between traffic spikes)
  • You need the latest model capabilities without waiting for open-source releases and optimization
  • Your team has one or two ML engineers, not a dedicated infrastructure team

When Self-Hosting Makes Sense (Fewer Cases Than You Think)

Self-hosting becomes viable only when one of these is true: you need sub-second latency for millions of daily requests, you’re handling sensitive data that can’t leave your infrastructure, or you’re operating at scale where per-token costs genuinely move the needle on margins.

Example: Databricks benchmarked coding agents on its million-line production codebase and found the open-source GLM-5.2 matched Claude Opus 4.8 performance while cutting costs from $1.94 to $1.28 per task. That’s a 34% savings. At Databricks’ scale (a massive company), that justifies running inference servers in-house. At a 50-person startup, it doesn’t.

If you’re in regulated finance or healthcare and your compliance posture requires models to run locally, self-hosting becomes necessary, not optional. But “we want to avoid vendor lock-in” is not a sufficient reason—it’s a distraction that delays shipping features.

The Hybrid Reality: Most Teams End Up Here

The practical middle ground is using multiple providers’ APIs (Claude, GPT-5.6 Sol, Gemini 3.5 Flash, Qwen models) with a gateway layer that routes requests based on cost and latency. This gives you redundancy if a provider has an outage (like OpenAI did to Cursor recently over the SpaceX acquisition dispute), and it lets you swap models cheaply when pricing changes.

Here’s what that looks like:

# Simple multi-provider routing
import anthropic
import openai

def get_completion(prompt, priority="speed"):
    if priority == "cost":
        # Use Qwen 3.8-Flash-Next for cheap inference
        # ~1/10th the cost of frontier models
        return call_alibaba_qwen(prompt)
    elif priority == "speed":
        # Use Gemini 3.7 Flash for latency-sensitive tasks
        # Introductory pricing through Dec 31, 2026
        return call_google_gemini(prompt)
    else:
        # Default to Claude Opus 5 for reliability
        client = anthropic.Anthropic()
        return client.messages.create(
            model="claude-opus-5",
            max_tokens=1024,
            messages=[{"role": "user", "content": prompt}]
        )

This pattern lets you migrate gradually. Start with Claude or GPT-5.6 Sol (safe defaults), then add cheaper models like Qwen or Gemini for non-critical tasks once you understand your latency and accuracy requirements.

The Data Moat Exception

If your actual competitive advantage is in proprietary training data or fine-tuning, then building infrastructure makes sense. But be honest: most companies claiming this are overestimating. Fine-tuning for knowledge retention (like “remember my codebase”) is better solved with RAG. Fine-tuning for style or behavior can often be handled with better prompts and few-shot examples.

One legitimate case: if you’re building autonomous agents that run for hours with long context windows, you might want to self-host Qwen 3.8-Max or Kimi K3 (both open-weight with 1M+ context) to avoid per-token costs spiraling. Qwen 3.8-Max offers 1M context for closed-weight API access; running a 2.4T parameter model locally requires H100 clusters, which brings you back to the cost problem.

The Real Trap to Avoid

The most expensive mistake is building your own infrastructure stack (model serving, fine-tuning pipelines, evaluation harnesses) before you have product-market fit. You’ll spend 6–12 months on MLOps that could’ve been features. By the time you’ve optimized inference throughput by 12%, your API costs might have dropped 40% due to model improvements, making all that work pointless.

If you’re going to self-host, wait until:

  • You’re processing >10B tokens monthly and have concrete margin pressure
  • You have a dedicated ML infrastructure team
  • You’ve benchmarked your workload against frontier APIs and confirmed self-hosting actually saves money after all overhead

Bottom line: Use APIs from frontier providers (Claude Opus 5, GPT-5.6 Sol, Gemini 3.5 Flash) unless you’re processing tens of billions of tokens annually or have hard compliance requirements. The engineering debt of maintaining your own inference cluster will cost more than the API bill for the next three years.

Question via Hacker News