How to Integrate LLMs Into Production Apps: Engineering Guide

Why production LLMs are different

Deploying an LLM demo is straightforward. Deploying one that 50,000 users depend on daily is an entirely different engineering challenge. The gap between "it works in the playground" and "it works reliably in production at scale" is where most AI integrations quietly fail.

The core challenges are latency (typical OpenAI GPT-4 P99 can be 8-12 seconds), cost (a 128K context window prompt can cost $0.04 per request — at scale that becomes unsustainable), and reliability (model APIs have outages, and your system needs graceful degradation strategies). Teams that treat the model as a magical black box discover these issues only after launch traffic arrives.

Choosing the right model and API

Start with the smallest model that clears your quality bar on a representative eval set. Frontier models are excellent for complex reasoning and tool use, but many production workflows — classification, extraction, short grounded answers — succeed on mid-tier models at a fraction of the cost and latency.

Decide early whether you need streaming, structured outputs, vision, function calling, or long context. Those requirements narrow providers quickly. Also plan for dual-vendor fallbacks: when a primary API degrades, routing a subset of traffic to a secondary model keeps the product usable while you investigate.

Building a reliable RAG pipeline

Retrieval-Augmented Generation is now the standard pattern for grounding LLMs in your proprietary data. A solid RAG implementation has three phases: ingestion, retrieval, and generation.

For ingestion, you chunk your documents intelligently (not just by character count — use semantic chunking), embed them with a consistent model (text-embedding-3-large gives excellent recall), and store in a vector DB like Pinecone, pgvector, or Weaviate.

Retrieval quality is determined by your similarity search strategy. Hybrid search — combining dense vector similarity with BM25 keyword scoring — consistently outperforms pure vector search by 15–25% on recall benchmarks.

// Hybrid search with pgvector + BM25
const results = await db.query(`
  SELECT id, content, 
    (1 - (embedding <=> $1)) * 0.7 + ts_rank(tsv, query) * 0.3 AS score
  FROM documents, plainto_tsquery($2) query
  ORDER BY score DESC
  LIMIT 10
`, [embedding, searchQuery]);

Latency and cost optimisation

The two biggest levers for LLM cost and latency are context window management and semantic caching.

For context, compress your system prompt (use GPT-4 to write a shorter version of itself), trim conversation history to the last N relevant turns, and use the smallest model capable of the task — GPT-4o-mini handles 80% of RAG queries at 10x lower cost.

Semantic caching means storing embeddings of previous queries and returning cached responses for semantically similar questions (cosine similarity > 0.94). In our experience this serves 30–40% of production traffic from cache, with sub-50ms response times.

Observability and evals

You cannot improve what you cannot measure. For LLM observability, instrument every call with: input tokens, output tokens, model used, latency (TTFT and total), cost, and a trace ID for debugging.

More importantly, build an evaluation harness. Define golden datasets for your key tasks and run automated evals on every model or prompt change. Tools like LangSmith, Braintrust, and Weights & Biases give you the infrastructure to do this without building it from scratch. Pair offline evals with online feedback signals so regressions surface before customers open tickets.

Fine-tuning vs prompt engineering

Prompt engineering and retrieval should be exhausted before fine-tuning. Most quality gaps we see in production are retrieval misses, weak instructions, or missing tools — not a model that needs more gradient steps.

Fine-tune when you have stable, high-quality labeled examples and a clear format or domain voice that prompts cannot reliably enforce. Budget for ongoing evaluation: a fine-tuned model that drifts from your product language after a few prompt iterations becomes an expensive liability.

Deployment checklist

Before you ship your LLM-powered feature to production, treat the launch like any other critical dependency. Confirm you have kill switches, budget caps, and a degraded mode that still delivers value when the model layer fails.

Run the checklist below with engineering and product together. Shipping without fallback behaviour or cost alerts is how demos become outages.

Production AI needs a product contract that survives imperfect inputs and changing providers. Define the user decision, the approved data sources, the freshness rule, and the action boundary before tuning a prompt. Keep an example set drawn from real requests, including vague questions, missing context, sensitive data, and attempts to reach another tenant. Review failures by layer: source content, retrieval, instructions, tool execution, or presentation. That diagnosis prevents a cosmetic prompt change from hiding an authorization or data-quality defect. Release gradually, keep a non-AI path available, and make feedback attach to a trace rather than an anonymous thumbs-down. Monitor cost, time to first token, grounded citation rate, refusal quality, and escalation rate together. A feature is operationally ready when the team can explain the answer, reproduce the context, cap spend, disable a risky capability, and still give the customer a useful next step.

For an AI roadmap, make the operating model visible to the whole product team. Product should own the user outcome and refusal experience; engineering should own the data boundary, service reliability, and measurable quality gates; support should own the escalation route; and security should approve the information class before it reaches a provider. Review a small sample of traces on a cadence, including successful answers, expensive answers, and failures. Version prompts, retrieval settings, and tool schemas so an incident can be reproduced rather than discussed from memory. When a source changes, assess retrieval and citations before shipping it. When a provider changes, rerun the same golden tasks. This discipline creates a feedback loop in which AI capability becomes more useful with use instead of becoming an opaque feature that is too risky to improve.

Launch with kill switches and evals

Do not enable an LLM feature for all tenants until golden-set evals pass, cost alerts fire correctly, and a non-model fallback exists. Treat the provider like any other critical dependency with an on-call owner.

Production AI integrations succeed when reliability engineering surrounds the model — not when the prompt looks clever in a playground.