Netflix's vLLM and Triton Stack: 7 Production Lessons

TL;DR

Netflix described an internal LLM serving pathway built around a unified serving layer, Model Scoring Service, NVIDIA Triton and vLLM. The team selected vLLM over its earlier TensorRT-LLM path for faster support of custom architectures, easier debugging, extensibility and researcher familiarity. Production then exposed problems that simple benchmarks miss: Python logits processors became CPU-bound under batching, API fields could be accepted and silently dropped, model downloads delayed startup, incompatible Triton and vLLM versions failed at load time, and separate metrics hid the real state of the service. The transferable lesson is to own the contract, tests, rollout and observability before owning every inference component. Hosted APIs still fit low or bursty demand; an in-house stack becomes credible when custom decoding, steady volume, privacy or model control justify a dedicated platform team.

The viral version of this story is simple: Netflix stopped paying an AI provider and built everything itself. The useful version is more precise. Netflix's AI Platform teams described one internal LLM serving pathway built on NVIDIA Triton and vLLM. They explained why it fit their workloads, where it broke under production concurrency, and what they had to add around it.

That distinction matters if you are making an inference architecture decision. Copying the component list gives you software. Copying the decision boundaries gives you an operating model. This article extracts those boundaries and keeps them separate from the cost and compliance question already owned by our guide to when self-hosting an LLM in the EU pays off.

Does Netflix use OpenAI?

It is not accurate to conclude that Netflix never uses OpenAI. In a separate Netflix system, MediaFM, the team says it used OpenAI's text-embedding-3-large for timed text and title metadata. The in-house serving article proves that Netflix runs an internal vLLM and Triton path for selected workloads. It does not prove an organization-wide ban on hosted models.

The buyer lesson is stronger than the absolutist claim: a mature AI estate can use different inference rails for different jobs. Hosted APIs can cover fast experiments or capabilities that are expensive to reproduce. Internal serving can cover custom models, proprietary decoding rules, predictable load, or tighter control. Architecture should follow workload boundaries, not a company-wide slogan.

What does the Netflix LLM serving architecture look like?

The public design has four layers. Existing applications call a unified JVM serving system over gRPC. That system owns routing, A/B allocation, feature fetching, pre-processing and post-processing. Large models are delegated to Model Scoring Service, Netflix's shared inference backend. NVIDIA Triton manages model loading and GPU execution below it, while vLLM is the paved-path engine for LLM workloads.

Consumer contract: existing services use the same scoring abstraction for classical ML and LLMs.
Serving workflow: routing, experiments and data preparation remain outside the inference engine.
Control plane: deployment, health, autoscaling, model versions and multi-region rollout remain platform responsibilities.
Inference plane: Triton hosts the serving backends, while vLLM handles LLM scheduling, batching and decoding.

Newer LLM applications can also use an OpenAI-compatible HTTP frontend. That is an interface choice, not evidence that an OpenAI-hosted model serves the request. Keeping protocol and provider separate lets callers retain a familiar client while the platform changes the engine behind it.

Why did Netflix choose vLLM over TensorRT-LLM?

Netflix originally used TensorRT-LLM. By summer 2025, its team judged that open-source engines had narrowed the performance gap for its workload mix. The decision moved to operational fit. vLLM loaded custom architectures without the earlier multi-step compilation path, exposed hooks for custom decoding, made intermediate state easier to inspect, and was already familiar to ML researchers.

Decision factor	Why it mattered	Question for your team
Architecture change rate	Custom models reached serving without a separate compiled artifact workflow	How often do your model shapes or custom layers change?
Debuggability	Engine state and failures were easier for practitioners to inspect	Who diagnoses a failed load at 02:00?
Decoding extensibility	Custom logits processing was central to Netflix constraints	Do you need rules that a standard structured-output API cannot express?
Team familiarity	Researchers already used vLLM, reducing handoff friction	Can model authors reproduce production behavior before handoff?
Peak benchmark	Important, but no longer decisive in isolation	Does the benchmark match your models, batch sizes and output constraints?

This is not a universal verdict on vLLM versus TensorRT-LLM. Both projects continue to change. It is a repeatable selection method: benchmark your real workload, then price the engineering path between research, deployment, debugging and rollback. A few more tokens per second do not compensate for a platform your team cannot safely change.

Why did constrained decoding hit the Python GIL?

Constrained decoding prevents invalid tokens during generation instead of validating malformed output afterwards. Netflix modeled each constraint as a state machine that generated a token-eligibility mask at every step. In vLLM V0, the custom Python processor ran per request. The GPU produced batched logits, then CPU-side constraint logic processed the requests sequentially. As batch size grew, CPU time grew with it because the Python Global Interpreter Lock prevented that hot path from running in parallel.

vLLM V1 provided a batch-level processor model. Netflix rewrote the processor around batch data structures and moved the hot path to multi-threaded C++. Current vLLM documentation likewise defines a batch-level logits processor interface whose apply method receives the batch logits tensor.

The hard part did not disappear. Dynamic batches require explicit state updates. Chunked prefills can span several engine steps. Under memory pressure, preemption can remove a request's KV cache and later reschedule it with a shorter token history. Netflix added tracking for partial prefills and reset its state machine when history shrank. Flat processing time was an outcome of a new state model, not a free switch from V0 to V1.

Seven production lessons worth copying

1. Choose for operational fit, then benchmark

Use representative models, concurrency, prompt lengths, output lengths and constraint logic. Measure time to first token, inter-token latency, throughput and tail latency. Add the time required to package, diagnose and upgrade a model. An engine is part of a release system, not a benchmark binary.

2. Keep the consumer contract above the engine

Netflix did not make every caller understand vLLM. Its existing serving layer continued to own routing, experiments and workflow logic. A stable internal contract lets you change an engine without rewriting every product integration. The same principle appears in our stateful LLM platform architecture: durable product behavior should not depend on one model runtime.

3. Pin the full compatibility set

Triton's vLLM backend depends on a specific vLLM API surface. Netflix reported a backend that failed to load when those versions drifted. NVIDIA now publishes a Triton and vLLM compatibility matrix for its container releases. Treat the Triton image, vLLM version, CUDA version, driver and plugins as one tested unit. Do not let a model package override one part independently.

4. Test API semantics, not only status codes

Netflix found that Triton's OpenAI-compatible frontend accepted response_format but dropped it before the request reached vLLM. The call succeeded while the promised output constraint vanished. Netflix patched the translation layer. NVIDIA's current OpenAI-compatible Triton frontend documentation shows how the frontend and vLLM backend are composed, but your contract tests still need to verify every field your product relies on.

5. Match rollout strategy to interface risk

Netflix uses Red-Black deployment when the model interface stays stable. The new version starts beside the old one, passes health checks and receives traffic in phases. Breaking tensor shapes create a coordination problem because old consumers and new models overlap. For those changes, versioned deployments keep both interfaces alive until consumers migrate. The extra GPU cost buys a safe compatibility window.

6. Put model weights on the startup path you can afford

Large downloads from object storage made cold starts too slow, so Netflix materialized announced models on Amazon FSx. AWS documents that an S3-linked FSx for Lustre file incurs latency on first access unless teams preload the required file contents. The broader rule is provider-neutral: make weight placement part of deployment, verify it before readiness, and measure cold start with an empty node rather than a warm developer machine.

7. Merge engine and server observability

Triton's bridge exposed only a subset of the vLLM metrics Netflix needed. The team combined Triton server metrics with vLLM's Prometheus multiprocess files behind one endpoint. Your minimum dashboard should connect request rate, queue time, time to first token, inter-token latency, token throughput, KV cache use, prefix-cache hits, preemption, model-load state, errors and GPU saturation. A green GPU graph can coexist with a broken CPU logits path.

Should your company copy Netflix or use a hosted API?

Signal	Hosted or managed inference	Internal vLLM and Triton platform
Demand	Low, uncertain or bursty	Steady enough to plan GPU capacity
Model needs	Standard models and API features	Custom architectures, plugins or decoding rules
Change ownership	Product team wants provider-managed upgrades	Platform team can pin, test and roll back the stack
Data boundary	Contracted hosted processing is acceptable	Inference must remain inside controlled infrastructure
Observability	Provider metrics meet the service objective	You need token, cache and scheduler-level evidence
Failure budget	Provider failover is preferable	You can operate multi-version GPU rollouts and on-call response

Choose an internal platform only when at least one differentiating requirement survives a serious managed-service evaluation. Custom constrained decoding can qualify. A stable high-volume workload can qualify. A strict data boundary can qualify. Disliking per-token invoices, by itself, is not an architecture.

A practical 90-day validation plan

Weeks 1 to 2, define the contract: freeze representative prompts, response schemas, latency objectives, quality evals and failure behavior. Record the hosted baseline.
Weeks 3 to 5, benchmark the real path: test candidate engines with production-like concurrency, constraints and model variants. Include CPU work and cold starts.
Weeks 6 to 8, prove operations: pin the image set, preload weights, expose unified metrics, rehearse a failed model load and test rollback.
Weeks 9 to 10, run shadow traffic: compare quality, latency and schema compliance without serving user-visible responses.
Weeks 11 to 13, make the commercial decision: compare the full platform cost with the hosted bill, then include engineer time, redundancy, support and upgrade work.

If latency is the main constraint, keep serving architecture separate from model ownership. Our analysis of a shared KV cache that cut time to first token shows how one infrastructure change can matter without replacing the entire inference rail.

Netflix vLLM and Triton FAQ

Does Netflix run all LLM inference in-house?

Netflix has documented an internal vLLM and Triton serving pathway, but its public engineering work also describes OpenAI embeddings in a separate MediaFM system. The evidence supports a workload-specific architecture, not the claim that every Netflix AI workload is self-hosted.

Why did Netflix prefer vLLM to TensorRT-LLM?

For the workload mix it re-evaluated in 2025, Netflix prioritized support for custom architectures, decoding extensibility, debuggability and researcher familiarity after open-source engine performance had become competitive. That is a workload-specific decision, not a permanent universal ranking.

What caused the constrained-decoding bottleneck?

The original vLLM V0 design ran custom Python logits processing per request after the GPU produced batched logits. CPU work grew with batch size and the Python GIL prevented parallel execution. Netflix moved to vLLM V1 batch-level processing and a multi-threaded C++ hot path.

Do you need NVIDIA Triton to run vLLM?

No. vLLM can expose its own OpenAI-compatible server. Triton becomes relevant when a team wants a shared model repository, multiple backends, common control-plane integration or established server operations. It also adds a compatibility surface that must be pinned and tested.

When should a smaller company self-host LLM inference?

Consider it when custom decoding, custom models, strict data boundaries or steady utilization create a measurable advantage and a team can own upgrades, observability, rollback and on-call response. For low, bursty or standard demand, managed inference is usually the simpler starting point.

The decision to own is the operating boundary

Netflix's strongest lesson is not a brand-name stack. It is the sequence of ownership. Keep a stable consumer contract. Select the engine against real workloads. Pin the runtime as a unit. Test whether API promises reach the decoder. Match rollout to compatibility risk. Put weights on the deployment path. Observe CPU, cache, scheduler and GPU behavior together.

Wavect's AI enablement and inference architecture work covers that evaluation from workload benchmark to production handover. The Twinsoft AI case study shows how we turn AI decisions into a tested product workflow. Use our prototype-to-production decision guide to scope the wider platform work, or request an inference architecture review with your current traffic, data and model constraints.