Office Hours — What skills and mindsets distinguish someone doing prompt engineering from someone building actual software systems with LLMs?

What skills and mindsets distinguish someone doing prompt engineering from someone building actual software systems with LLMs?

The Core Split

Prompt engineering is optimization within a fixed surface. You’re tuning the input to get better output from a black box. Building software systems with LLMs is designing the entire pipeline, including failure modes, cost bounds, latency requirements, and decision points where the LLM is just one piece.

A prompt engineer asks: “How do I make this model produce better results?” A systems builder asks: “What happens when it doesn’t, and how does the business stay functional?”

Thinking in Workflows, Not Prompts

Prompt engineers live in ChatGPT or Claude’s web interface. They iterate on wording, try different temperatures, maybe add few-shot examples. The mindset is tactical: get the next output right.

Systems builders think in diagrams. They’re asking: If this LLM call fails silently, what breaks? If it takes 15 seconds instead of 2, does my customer see a timeout? If it returns something unexpected, do I validate it or just pass it through? They’re designing state machines with LLM nodes, not optimizing individual nodes.

A systems builder working on document summarization isn’t just crafting a better summary prompt. They’re asking: Should I chunk the document? Validate that chunks are semantically coherent? Run summarization in parallel on chunks then re-summarize the summaries? Add a human approval gate? Cache summaries when documents repeat? Route short documents to a cheaper model? Log failures to retrain classifiers? None of these are prompt problems.

Cost Thinking vs. Quality Thinking

Prompt engineers optimize for response quality in isolation. Better output feels like success.

Systems builders live in cost-per-transaction math. One prompt engineer’s “perfect” 2000-token response costs $0.04 per call. If you run that 100,000 times a month across users, that’s $4,000 in LLM spend. A systems builder might route 70% of requests to a cheaper model that’s “good enough,” use caching to avoid redundant calls, or batch requests during off-peak hours. The “worse” output often costs 1/10 as much for 95% of use cases.

Real example: A summarization system running on Claude Opus 4.8 costs $0.02 per 1K input tokens. If your average document is 8K tokens, that’s $0.16 per summary. Route the same document to Gemini 3.1 Flash at 1/20 the price, measure quality degradation, and you might find 90% of summaries are indistinguishable while slashing costs 18x. A prompt engineer doesn’t ask this question. A systems builder can’t afford not to.

Reliability vs. Perfection

Prompt engineers often shoot for the ideal output. A perfect extraction, a flawless response, a response that impresses.

Systems builders accept that LLMs are probabilistic. They ask: What’s the failure distribution? How often does this call return garbage? What’s my SLA? If I need 99.9% reliability and my base model gets there 94% of the time, do I add validation, retry logic, fallback models, or human escalation? They design for the tail, not the mean.

The mindset shifts from “make this better” to “make this reliable.” Reliability usually means adding infrastructure around the LLM, not tweaking the prompt. You validate structured outputs. You add retry loops with exponential backoff. You implement circuit breakers that stop calling a degraded API and fall back to cached answers. You log every decision so you can audit what happened when something broke.

Operational Awareness

Prompt engineers typically don’t think about:

• How many tokens am I actually using in production?
• What’s my rate limit, and do I ever hit it?
• If this call takes 30 seconds, how many user requests are blocked?
• What does a cascading failure look like when multiple agents retry the same thing?

Systems builders obsess over these. They’re instrumenting observability into every LLM call. They’re setting budgets and alarms. They’re running load tests to find breaking points. They’re understanding that a seemingly innocent retry loop, if replicated across thousands of users, becomes a DDoS on the API.

A real case: A team built an agentic workflow where each step called an LLM. It worked in dev. In production, under load, one slow LLM call backed up the queue, which triggered timeouts, which triggered retries, which doubled the queue, which burned through the API budget in hours. A prompt engineer would have tweaked the wording of each step. A systems builder would have added queuing, rate limiting, dead-letter handling, and monitoring from day one.

Embracing Determinism Where It Matters

Prompt engineers want the flexibility to let the model think freely. More creativity, more variance, more magic.

Systems builders understand that some parts of the pipeline need to be deterministic machines. If you need to extract a date, extract it with regex after the LLM suggests the candidate. If you need a true/false decision, don’t let the model hedge with “probably” or “it depends.” Add validation layers that convert fuzzy LLM outputs into crisp inputs for downstream systems.

This doesn’t mean replacing all LLMs with rule engines. It means being precise about where uncertainty is acceptable and where it isn’t. A systems builder might use Claude Opus 4.8 for open-ended research synthesis but route structured extraction through a validation layer that forces yes/no/invalid.

The Hiring Question

If you’re building a product business, you need systems builders more than prompt engineers. Prompt engineers can be hired as needed, often part-time or via contractors. Systems builders are the core team because they’re designing the actual product architecture.

The distinction shows up in how they talk about problems. A prompt engineer says, “The model keeps getting this question wrong, let me try a different approach.” A systems builder says, “This question has high variance in output quality, so we’re adding a confidence score check and routing uncertain cases to human review.” One is iterating on a prompt. The other is designing a system that can operate reliably in production without human babysitting.

Bottom line: Hire systems builders who’ve shipped LLM products in production and understand the full lifecycle (costs, reliability, monitoring, failure modes). Prompt engineering is a temporary skill; product architecture is the constraint that actually matters.

Question via Hacker News