Cold Start vs Warm Inference: Latency Engineering for LLMs

The Cold-Start Tax

The first request to a model that has not been used in a while pays a tax: model loading, kernel JIT, cache warming. After the first call, latency drops to steady-state. The user who hits the cold path has a noticeably worse experience.

By 2026 cold-start latency is a major optimization target for LLM serving. This piece walks through the patterns.

What Cold Start Looks Like

flowchart LR
    Req1[First request: 5-30s] --> Load[Model load + warmup]
    Req2[Second request: 200-500ms] --> Steady[Steady state]
    Req3[Third request: 200-500ms] --> Same

The first request takes seconds; subsequent are sub-second. Cold paths happen for:

• Brand-new model deployment
• First user after auto-scale-down
• After a long idle period
• After a server restart

Why It Happens

• Model weights load from storage to GPU
• JIT compilation of kernels
• KV cache initialization
• Connection setup with model storage

Each adds time. Total varies from 5 seconds to several minutes depending on model size.

Mitigations

flowchart TB
    M[Mitigations] --> M1[Warm pool: keep N replicas hot]
    M --> M2[Pre-warm on schedule]
    M --> M3[Predictive scaling]
    M --> M4[Faster cold-start architecture]
    M --> M5[Synthetic traffic to keep warm]

Warm Pool

Keep a baseline number of replicas always running. New requests hit warm replicas. The cost: paying for idle capacity.

Pre-Warm on Schedule

Anticipate traffic patterns; pre-warm before peaks. Especially useful for predictable patterns (business-hours traffic).

Predictive Scaling

ML-driven scaling that anticipates demand. More efficient than reactive scaling.

Faster Cold-Start Architecture

• Quantized weights (smaller, faster to load)
• Storage closer to compute (in-memory or SSD-backed)
• Kernel pre-compilation
• Connection pre-warming

Synthetic Traffic

For workloads with idle gaps, send synthetic requests to keep replicas warm. Costs more but eliminates cold paths.

Provider-Hosted vs Self-Hosted

For provider-hosted models (OpenAI, Anthropic, Google):

• The provider handles cold-start; you generally don't see it
• Some providers expose "burst" capacity that has cold-start
• Reserved capacity typically eliminates cold-start

For self-hosted:

• You own the cold-start problem
• Auto-scale-down is tempting for cost; cold-starts hurt UX
• The trade-off is workload-specific

A Production Pattern

flowchart LR
    Pool[Warm pool: 2 replicas] --> Reactive[Auto-scale to N on demand]
    Reactive --> Predict[Predictive scaling for known peaks]
    Pool --> Synthetic[Synthetic traffic during quiet hours]

Layered: always-warm pool + reactive auto-scale + predictive scale + synthetic traffic. Eliminates cold-start for all but exotic spike scenarios.

Cost vs Latency

For a large self-hosted model:

• 0 warm replicas: cold-start on every idle gap; cheapest
• 1 warm replica: rare cold-start
• 2+ warm replicas: essentially never cold-start; expensive

Pick based on UX requirement. For consumer apps, 0-1 warm. For enterprise customer service, 2+ minimum.

What CallSphere Does

For voice agents:

• 2 warm replicas baseline (zero cold-start UX is non-negotiable for voice)
• Synthetic heartbeat traffic during quiet hours
• Auto-scale up on traffic patterns
• Reserved capacity for predictable peaks

Cost: roughly 2x what we'd pay with full auto-scale-down. Worth it for the UX.

Cold Start in Edge Inference

For edge / on-device:

• Models load on app start
• Subsequent app launches benefit from page cache
• "Lazy load" patterns delay model load until needed (trade off first-use latency)

What Doesn't Help

• Ignoring cold-start (pretending it doesn't matter)
• Optimizing average latency without checking p99
• Auto-scale settings that swing too aggressively (constant cold-starts)

Sources

• "Cold-start optimization for LLMs" — https://thenewstack.io
• "Inference serving patterns" Modal — https://modal.com
• "AWS Lambda cold start" — https://aws.amazon.com
• "Predictive auto-scaling" — https://kubernetes.io
• vLLM serving patterns — https://docs.vllm.ai

Cold Start vs Warm Inference: Latency Engineering for LLMs: production view

Cold Start vs Warm Inference: Latency Engineering for LLMs sits on top of a regional VPC and a cold-start problem you only see at 3am. If your voice stack lives in us-east-1 but your customer is calling from a Sydney mobile network, the round-trip time alone wrecks turn-taking. Multi-region routing, GPU residency, and warm pools become the difference between "natural" and "robotic" — and it's all infra, not the model.

Broader technology framing

The protocol layer determines what's possible: WebRTC for browser-side widgets, SIP trunks (Twilio, Telnyx) for PSTN voice, WebSockets for the Realtime API streaming session. Each has its own jitter buffer, its own ICE/STUN dance, and its own failure modes when a customer's corporate firewall is hostile.

Front-end is Next.js 15 + React 19 for the marketing surface and the in-app dashboards, with server components used heavily for the SEO-critical pages. Backend splits across FastAPI for the AI worker, NestJS + Prisma for the customer-facing API, and a thin Go gateway that does auth, rate limiting, and routing — letting each service scale on its own characteristics.

Datastores: Postgres as the source of truth (per-vertical schemas like healthcare_voice, realestate_voice), ChromaDB for RAG over support docs, Redis for ephemeral session state. Postgres RLS enforces tenant isolation at the row level so a misconfigured query can't leak across customers.

FAQ

Is this realistic for a small business, or is it enterprise-only? The IT Helpdesk product is built on ChromaDB for RAG over runbooks, Supabase for auth and storage, and 40+ data models covering tickets, assets, MSP clients, and escalation chains. For a topic like "Cold Start vs Warm Inference: Latency Engineering for LLMs", that means you're not starting from scratch — you're configuring an agent template that's already been hardened across thousands of conversations.

Which integrations have to be in place before launch? Day one is integration mapping (scheduler, CRM, messaging) and prompt tuning against your top 20 real call transcripts. Day two through five is shadow-mode running, where the agent transcribes and recommends but a human still answers, so you can compare side-by-side. Go-live is the moment your eval pass-rate clears your internal bar.

How do we measure whether it's actually working? The honest answer: it scales until your tool catalog gets stale. The agent is only as good as the integrations it can actually call, so the operational discipline is keeping schemas, webhooks, and fallback paths green. The platform handles the rest — observability, retries, multi-region routing — without your team owning the GPU layer.

Talk to us

Explore a live demo and compare current plans to find the right fit for your business.