--- title: "Netflix vLLM and Triton: Production Lessons" canonical: https://wavect.io/blog/netflix-vllm-triton-inference-stack/ language: en description: "Netflix's vLLM and Triton stack explained: constrained decoding, GIL bottlenecks, deployment, caching, metrics and a practical build-vs-buy test." image: "https://wavect.io/img/blog/headers/header_netflix-vllm-triton-inference-stack.png" --- [**Back**](/blog/overview/) [![Kevin Riedl](/img/team/kevin.webp)](/team/kevin-riedl/) [Kevin Riedl](/team/kevin-riedl/) https://linkedin.com/in/wsdt 11 min read · 16 Aug 2026 Last reviewed August 16, 2026 [**Next**](/blog/self-hosting-llms-eu-cost/) # Netflix's vLLM and Triton Stack: 7 Production Lessons TL;DR Netflix described an internal LLM serving pathway built around a unified serving layer, Model Scoring Service, NVIDIA Triton and vLLM. The team selected vLLM over its earlier TensorRT-LLM path for faster support of custom architectures, easier debugging, extensibility and researcher familiarity. Production then exposed problems that simple benchmarks miss: Python logits processors became CPU-bound under batching, API fields could be accepted and silently dropped, model downloads delayed startup, incompatible Triton and vLLM versions failed at load time, and separate metrics hid the real state of the service. The transferable lesson is to own the contract, tests, rollout and observability before owning every inference component. Hosted APIs still fit low or bursty demand; an in-house stack becomes credible when custom decoding, steady volume, privacy or model control justify a dedicated platform team. The viral version of this story is simple: Netflix stopped paying an AI provider and built everything itself. The useful version is more precise. Netflix's AI Platform teams described [one internal LLM serving pathway built on NVIDIA Triton and vLLM](https://netflixtechblog.com/in-house-llm-serving-at-netflix-a5a8e799ea2c). They explained why it fit their workloads, where it broke under production concurrency, and what they had to add around it. That distinction matters if you are making an inference architecture decision. Copying the component list gives you software. Copying the decision boundaries gives you an operating model. This article extracts those boundaries and keeps them separate from the cost and compliance question already owned by our guide to [when self-hosting an LLM in the EU pays off](/blog/self-hosting-llms-eu-cost/). ## Does Netflix use OpenAI? It is not accurate to conclude that Netflix never uses OpenAI. In a separate Netflix system, MediaFM, the team says it used [OpenAI's text-embedding-3-large for timed text and title metadata](https://medium.com/netflix-techblog/mediafm-the-multimodal-ai-foundation-for-media-understanding-at-netflix-e8c28df82e2d). The in-house serving article proves that Netflix runs an internal vLLM and Triton path for selected workloads. It does not prove an organization-wide ban on hosted models. The buyer lesson is stronger than the absolutist claim: a mature AI estate can use different inference rails for different jobs. Hosted APIs can cover fast experiments or capabilities that are expensive to reproduce. Internal serving can cover custom models, proprietary decoding rules, predictable load, or tighter control. Architecture should follow workload boundaries, not a company-wide slogan. ## What does the Netflix LLM serving architecture look like? The public design has four layers. Existing applications call a unified JVM serving system over gRPC. That system owns routing, A/B allocation, feature fetching, pre-processing and post-processing. Large models are delegated to Model Scoring Service, Netflix's shared inference backend. NVIDIA Triton manages model loading and GPU execution below it, while vLLM is the paved-path engine for LLM workloads. 1. **Consumer contract:** existing services use the same scoring abstraction for classical ML and LLMs. 2. **Serving workflow:** routing, experiments and data preparation remain outside the inference engine. 3. **Control plane:** deployment, health, autoscaling, model versions and multi-region rollout remain platform responsibilities. 4. **Inference plane:** Triton hosts the serving backends, while vLLM handles LLM scheduling, batching and decoding. Newer LLM applications can also use an OpenAI-compatible HTTP frontend. That is an interface choice, not evidence that an OpenAI-hosted model serves the request. Keeping protocol and provider separate lets callers retain a familiar client while the platform changes the engine behind it. ## Why did Netflix choose vLLM over TensorRT-LLM? Netflix originally used TensorRT-LLM. By summer 2025, its team judged that open-source engines had narrowed the performance gap for its workload mix. The decision moved to operational fit. vLLM loaded custom architectures without the earlier multi-step compilation path, exposed hooks for custom decoding, made intermediate state easier to inspect, and was already familiar to ML researchers. | Decision factor | Why it mattered | Question for your team | | --- | --- | --- | | Architecture change rate | Custom models reached serving without a separate compiled artifact workflow | How often do your model shapes or custom layers change? | | Debuggability | Engine state and failures were easier for practitioners to inspect | Who diagnoses a failed load at 02:00? | | Decoding extensibility | Custom logits processing was central to Netflix constraints | Do you need rules that a standard structured-output API cannot express? | | Team familiarity | Researchers already used vLLM, reducing handoff friction | Can model authors reproduce production behavior before handoff? | | Peak benchmark | Important, but no longer decisive in isolation | Does the benchmark match your models, batch sizes and output constraints? | This is not a universal verdict on vLLM versus TensorRT-LLM. Both projects continue to change. It is a repeatable selection method: benchmark your real workload, then price the engineering path between research, deployment, debugging and rollback. A few more tokens per second do not compensate for a platform your team cannot safely change. ## Why did constrained decoding hit the Python GIL? Constrained decoding prevents invalid tokens during generation instead of validating malformed output afterwards. Netflix modeled each constraint as a state machine that generated a token-eligibility mask at every step. In vLLM V0, the custom Python processor ran per request. The GPU produced batched logits, then CPU-side constraint logic processed the requests sequentially. As batch size grew, CPU time grew with it because the Python Global Interpreter Lock prevented that hot path from running in parallel. vLLM V1 provided a batch-level processor model. Netflix rewrote the processor around batch data structures and moved the hot path to multi-threaded C++. Current vLLM documentation likewise defines a [batch-level logits processor interface](https://docs.vllm.ai/en/latest/api/vllm/v1/sample/logits_processor/index.html) whose `apply` method receives the batch logits tensor. The hard part did not disappear. Dynamic batches require explicit state updates. Chunked prefills can span several engine steps. Under memory pressure, preemption can remove a request's KV cache and later reschedule it with a shorter token history. Netflix added tracking for partial prefills and reset its state machine when history shrank. Flat processing time was an outcome of a new state model, not a free switch from V0 to V1. ## Seven production lessons worth copying ### 1. Choose for operational fit, then benchmark Use representative models, concurrency, prompt lengths, output lengths and constraint logic. Measure time to first token, inter-token latency, throughput and tail latency. Add the time required to package, diagnose and upgrade a model. An engine is part of a release system, not a benchmark binary. ### 2. Keep the consumer contract above the engine Netflix did not make every caller understand vLLM. Its existing serving layer continued to own routing, experiments and workflow logic. A stable internal contract lets you change an engine without rewriting every product integration. The same principle appears in our [stateful LLM platform architecture](/blog/stateful-llm-platform-production-architecture/): durable product behavior should not depend on one model runtime. ### 3. Pin the full compatibility set Triton's vLLM backend depends on a specific vLLM API surface. Netflix reported a backend that failed to load when those versions drifted. NVIDIA now publishes a [Triton and vLLM compatibility matrix](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/introduction/compatibility.html) for its container releases. Treat the Triton image, vLLM version, CUDA version, driver and plugins as one tested unit. Do not let a model package override one part independently. ### 4. Test API semantics, not only status codes Netflix found that Triton's OpenAI-compatible frontend accepted `response_format` but dropped it before the request reached vLLM. The call succeeded while the promised output constraint vanished. Netflix patched the translation layer. NVIDIA's current [OpenAI-compatible Triton frontend documentation](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/openai_readme.html) shows how the frontend and vLLM backend are composed, but your contract tests still need to verify every field your product relies on. ### 5. Match rollout strategy to interface risk Netflix uses Red-Black deployment when the model interface stays stable. The new version starts beside the old one, passes health checks and receives traffic in phases. Breaking tensor shapes create a coordination problem because old consumers and new models overlap. For those changes, versioned deployments keep both interfaces alive until consumers migrate. The extra GPU cost buys a safe compatibility window. ### 6. Put model weights on the startup path you can afford Large downloads from object storage made cold starts too slow, so Netflix materialized announced models on Amazon FSx. AWS documents that an S3-linked FSx for Lustre file incurs latency on first access unless teams [preload the required file contents](https://docs.aws.amazon.com/fsx/latest/LustreGuide/preload-file-contents-hsm-dra.html). The broader rule is provider-neutral: make weight placement part of deployment, verify it before readiness, and measure cold start with an empty node rather than a warm developer machine. ### 7. Merge engine and server observability Triton's bridge exposed only a subset of the vLLM metrics Netflix needed. The team combined Triton server metrics with vLLM's Prometheus multiprocess files behind one endpoint. Your minimum dashboard should connect request rate, queue time, time to first token, inter-token latency, token throughput, KV cache use, prefix-cache hits, preemption, model-load state, errors and GPU saturation. A green GPU graph can coexist with a broken CPU logits path. ## Should your company copy Netflix or use a hosted API? | Signal | Hosted or managed inference | Internal vLLM and Triton platform | | --- | --- | --- | | Demand | Low, uncertain or bursty | Steady enough to plan GPU capacity | | Model needs | Standard models and API features | Custom architectures, plugins or decoding rules | | Change ownership | Product team wants provider-managed upgrades | Platform team can pin, test and roll back the stack | | Data boundary | Contracted hosted processing is acceptable | Inference must remain inside controlled infrastructure | | Observability | Provider metrics meet the service objective | You need token, cache and scheduler-level evidence | | Failure budget | Provider failover is preferable | You can operate multi-version GPU rollouts and on-call response | Choose an internal platform only when at least one differentiating requirement survives a serious managed-service evaluation. Custom constrained decoding can qualify. A stable high-volume workload can qualify. A strict data boundary can qualify. Disliking per-token invoices, by itself, is not an architecture. ## A practical 90-day validation plan 1. **Weeks 1 to 2, define the contract:** freeze representative prompts, response schemas, latency objectives, quality evals and failure behavior. Record the hosted baseline. 2. **Weeks 3 to 5, benchmark the real path:** test candidate engines with production-like concurrency, constraints and model variants. Include CPU work and cold starts. 3. **Weeks 6 to 8, prove operations:** pin the image set, preload weights, expose unified metrics, rehearse a failed model load and test rollback. 4. **Weeks 9 to 10, run shadow traffic:** compare quality, latency and schema compliance without serving user-visible responses. 5. **Weeks 11 to 13, make the commercial decision:** compare the full platform cost with the hosted bill, then include engineer time, redundancy, support and upgrade work. If latency is the main constraint, keep serving architecture separate from model ownership. Our analysis of a [shared KV cache that cut time to first token](/blog/shared-kv-cache-llm-inference-latency/) shows how one infrastructure change can matter without replacing the entire inference rail. ## Netflix vLLM and Triton FAQ ### Does Netflix run all LLM inference in-house? Netflix has documented an internal vLLM and Triton serving pathway, but its public engineering work also describes OpenAI embeddings in a separate MediaFM system. The evidence supports a workload-specific architecture, not the claim that every Netflix AI workload is self-hosted. ### Why did Netflix prefer vLLM to TensorRT-LLM? For the workload mix it re-evaluated in 2025, Netflix prioritized support for custom architectures, decoding extensibility, debuggability and researcher familiarity after open-source engine performance had become competitive. That is a workload-specific decision, not a permanent universal ranking. ### What caused the constrained-decoding bottleneck? The original vLLM V0 design ran custom Python logits processing per request after the GPU produced batched logits. CPU work grew with batch size and the Python GIL prevented parallel execution. Netflix moved to vLLM V1 batch-level processing and a multi-threaded C++ hot path. ### Do you need NVIDIA Triton to run vLLM? No. vLLM can expose its own OpenAI-compatible server. Triton becomes relevant when a team wants a shared model repository, multiple backends, common control-plane integration or established server operations. It also adds a compatibility surface that must be pinned and tested. ### When should a smaller company self-host LLM inference? Consider it when custom decoding, custom models, strict data boundaries or steady utilization create a measurable advantage and a team can own upgrades, observability, rollback and on-call response. For low, bursty or standard demand, managed inference is usually the simpler starting point. ## The decision to own is the operating boundary Netflix's strongest lesson is not a brand-name stack. It is the sequence of ownership. Keep a stable consumer contract. Select the engine against real workloads. Pin the runtime as a unit. Test whether API promises reach the decoder. Match rollout to compatibility risk. Put weights on the deployment path. Observe CPU, cache, scheduler and GPU behavior together. Wavect's [AI enablement and inference architecture work](/services/ai-enablement/) covers that evaluation from workload benchmark to production handover. The [Twinsoft AI case study](/case-studies/twinsoft-ai/) shows how we turn AI decisions into a tested product workflow. Use our [prototype-to-production decision guide](/software-development-guide/vibe-coded-prototype-to-production/) to scope the wider platform work, or [request an inference architecture review](/contact/) with your current traffic, data and model constraints. Models and infrastructure ## Continue through this cluster Model selection, inference economics, local deployment, compression and serving architecture. [Start with the cornerstone**Self-Hosting LLMs in the EU: When Open Weights Actually Pay Off**](/blog/self-hosting-llms-eu-cost/) - [Transformers.js Browser AI: When Local Inference Belongs in Your Product](/blog/transformers-js-browser-ai-guide/) - [How to Self-Host LiteLLM in Production: 2026 Guide](/blog/self-host-litellm-production-2026/) - [AI-Ready Company Wiki: Architecture and Build Guide](/blog/ai-ready-company-wiki/) - [Does Claude Watermark Text? The 2026 API Answer](/blog/claude-text-watermark-api-2026/) - [OpenKB Review: Knowledge Compiler vs RAG](/blog/openkb-review-vs-rag/) Inbox, without the noise ## Follow the work that matters to you Get a short email when we publish something new. Follow the whole blog or only the problems you care about. [**Back**](/blog/overview/) [![Kevin Riedl](/img/team/kevin.webp)](/team/kevin-riedl/) [Kevin Riedl](/team/kevin-riedl/) https://linkedin.com/in/wsdt 11 min read · 16 Aug 2026 Last reviewed August 16, 2026 [**Next**](/blog/self-hosting-llms-eu-cost/) New posts by email × × Get new posts by email A short email when we publish. Free, no tracking. ## Structured Data ```json { "@context": "https://schema.org", "@graph": [ { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ], "employee": [ { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } } ], "founder": [ { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } } ], "legalRepresentative": [ { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } } ], "name": "Wavect GmbH", "subjectOf": { "@id": "https://wavect.io/verified-claims.json#dataset", "@type": "Dataset", "creator": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] }, "description": "A machine-readable registry of quantitative and qualitative claims published by Wavect, with review dates, localized page appearances and public third-party citations where available.", "inLanguage": "en", "isAccessibleForFree": true, "license": "https://creativecommons.org/licenses/by/4.0/", "name": "Wavect verified publication claims", "url": "https://wavect.io/verified-claims.json" }, "url": "https://wavect.io/" }, { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "sameAs": [ "https://www.wikidata.org/wiki/Q139796365", "https://www.linkedin.com/in/wsdt", "https://github.com/wsdt" ], "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "sameAs": [ "https://www.wikidata.org/wiki/Q139796367", "https://www.linkedin.com/in/jocr77/", "https://github.com/jo-chris" ], "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/#website", "@type": "WebSite", "inLanguage": [ "en", "de", "es", "zh" ], "name": "Wavect", "potentialAction": { "@type": "SearchAction", "query-input": "required name=search_term_string", "target": { "@type": "EntryPoint", "urlTemplate": "https://wavect.io/search/?q={search_term_string}" } }, "publisher": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] }, "url": "https://wavect.io/" }, { "@id": "https://wavect.io/blog/netflix-vllm-triton-inference-stack/#webpage", "@type": "WebPage", "dateModified": "2026-08-16", "inLanguage": "en", "isPartOf": { "@id": "https://wavect.io/#website", "@type": "WebSite" }, "lastReviewed": "2026-08-16", "url": "https://wavect.io/blog/netflix-vllm-triton-inference-stack/" } ] } ``` ```json { "@context": "https://schema.org", "@type": "BlogPosting", "abstract": "Netflix described an internal LLM serving pathway built around a unified serving layer, Model Scoring Service, NVIDIA Triton and vLLM. The team selected vLLM over its earlier TensorRT-LLM path for faster support of custom architectures, easier debugging, extensibility and researcher familiarity. Production then exposed problems that simple benchmarks miss: Python logits processors became CPU-bound under batching, API fields could be accepted and silently dropped, model downloads delayed startup, incompatible Triton and vLLM versions failed at load time, and separate metrics hid the real state of the service. The transferable lesson is to own the contract, tests, rollout and observability before owning every inference component. Hosted APIs still fit low or bursty demand; an in-house stack becomes credible when custom decoding, steady volume, privacy or model control justify a dedicated platform team.", "articleBody": " Blog overview/AI and agents/Models and infrastructure Netflix's vLLM and Triton Stack: 7 Production Lessons TL;DR Netflix described an internal LLM serving pathway built around a unified serving layer, Model Scoring Service, NVIDIA Triton and vLLM. The team selected vLLM over its earlier TensorRT-LLM path for faster support of custom architectures, easier debugging, extensibility and researcher familiarity. Production then exposed problems that simple benchmarks miss: Python logits processors became CPU-bound under batching, API fields could be accepted and silently dropped, model downloads delayed startup, incompatible Triton and vLLM versions failed at load time, and separate metrics hid the real state of the service. The transferable lesson is to own the contract, tests, rollout and observability before owning every inference component. Hosted APIs still fit low or bursty demand; an in-house stack becomes credible when custom decoding, steady volume, privacy or model control justify a dedicated platform team. The viral version of this story is simple: Netflix stopped paying an AI provider and built everything itself. The useful version is more precise. Netflix's AI Platform teams described one internal LLM serving pathway built on NVIDIA Triton and vLLM. They explained why it fit their workloads, where it broke under production concurrency, and what they had to add around it. That distinction matters if you are making an inference architecture decision. Copying the component list gives you software. Copying the decision boundaries gives you an operating model. This article extracts those boundaries and keeps them separate from the cost and compliance question already owned by our guide to when self-hosting an LLM in the EU pays off. Does Netflix use OpenAI? It is not accurate to conclude that Netflix never uses OpenAI. In a separate Netflix system, MediaFM, the team says it used OpenAI's text-embedding-3-large for timed text and title metadata. The in-house serving article proves that Netflix runs an internal vLLM and Triton path for selected workloads. It does not prove an organization-wide ban on hosted models. The buyer lesson is stronger than the absolutist claim: a mature AI estate can use different inference rails for different jobs. Hosted APIs can cover fast experiments or capabilities that are expensive to reproduce. Internal serving can cover custom models, proprietary decoding rules, predictable load, or tighter control. Architecture should follow workload boundaries, not a company-wide slogan. What does the Netflix LLM serving architecture look like? The public design has four layers. Existing applications call a unified JVM serving system over gRPC. That system owns routing, A/B allocation, feature fetching, pre-processing and post-processing. Large models are delegated to Model Scoring Service, Netflix's shared inference backend. NVIDIA Triton manages model loading and GPU execution below it, while vLLM is the paved-path engine for LLM workloads. Consumer contract: existing services use the same scoring abstraction for classical ML and LLMs. Serving workflow: routing, experiments and data preparation remain outside the inference engine. Control plane: deployment, health, autoscaling, model versions and multi-region rollout remain platform responsibilities. Inference plane: Triton hosts the serving backends, while vLLM handles LLM scheduling, batching and decoding. Newer LLM applications can also use an OpenAI-compatible HTTP frontend. That is an interface choice, not evidence that an OpenAI-hosted model serves the request. Keeping protocol and provider separate lets callers retain a familiar client while the platform changes the engine behind it. Why did Netflix choose vLLM over TensorRT-LLM? Netflix originally used TensorRT-LLM. By summer 2025, its team judged that open-source engines had narrowed the performance gap for its workload mix. The decision moved to operational fit. vLLM loaded custom architectures without the earlier multi-step compilation path, exposed hooks for custom decoding, made intermediate state easier to inspect, and was already familiar to ML researchers. Decision factorWhy it matteredQuestion for your team Architecture change rateCustom models reached serving without a separate compiled artifact workflowHow often do your model shapes or custom layers change? DebuggabilityEngine state and failures were easier for practitioners to inspectWho diagnoses a failed load at 02:00? Decoding extensibilityCustom logits processing was central to Netflix constraintsDo you need rules that a standard structured-output API cannot express? Team familiarityResearchers already used vLLM, reducing handoff frictionCan model authors reproduce production behavior before handoff? Peak benchmarkImportant, but no longer decisive in isolationDoes the benchmark match your models, batch sizes and output constraints? This is not a universal verdict on vLLM versus TensorRT-LLM. Both projects", "articleSection": "Engineering", "author": { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "name": "Kevin Riedl", "sameAs": [ "https://www.wikidata.org/wiki/Q139796365", "https://www.linkedin.com/in/wsdt", "https://github.com/wsdt" ], "url": "https://wavect.io/team/kevin-riedl/" }, "citation": [ { "@type": "WebPage", "name": "one internal LLM serving pathway built on NVIDIA Triton and vLLM", "url": "https://netflixtechblog.com/in-house-llm-serving-at-netflix-a5a8e799ea2c" }, { "@type": "WebPage", "name": "OpenAI's text-embedding-3-large for timed text and title metadata", "url": "https://medium.com/netflix-techblog/mediafm-the-multimodal-ai-foundation-for-media-understanding-at-netflix-e8c28df82e2d" }, { "@type": "WebPage", "name": "batch-level logits processor interface", "url": "https://docs.vllm.ai/en/latest/api/vllm/v1/sample/logits_processor/index.html" }, { "@type": "WebPage", "name": "Triton and vLLM compatibility matrix", "url": "https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/introduction/compatibility.html" }, { "@type": "WebPage", "name": "OpenAI-compatible Triton frontend documentation", "url": "https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/openai_readme.html" }, { "@type": "WebPage", "name": "preload the required file contents", "url": "https://docs.aws.amazon.com/fsx/latest/LustreGuide/preload-file-contents-hsm-dra.html" } ], "dateModified": "2026-08-16", "datePublished": "2026-08-16", "description": "Netflix described an internal LLM serving pathway built around a unified serving layer, Model Scoring Service, NVIDIA Triton and vLLM. The team selected vLLM over its earlier TensorRT-LLM path for faster support of custom architectures, easier debugging, extensibility and researcher familiarity. Production then exposed problems that simple benchmarks miss: Python logits processors became CPU-bound under batching, API fields could be accepted and silently dropped, model downloads delayed startup, incompatible Triton and vLLM versions failed at load time, and separate metrics hid the real state of the service. The transferable lesson is to own the contract, tests, rollout and observability before owning every inference component. Hosted APIs still fit low or bursty demand; an in-house stack becomes credible when custom decoding, steady volume, privacy or model control justify a dedicated platform team.", "headline": "Netflix's vLLM and Triton Stack: 7 Production Lessons", "image": "https://wavect.io/img/blog/headers/header_netflix-vllm-triton-inference-stack.svg", "inLanguage": "en", "keywords": "AI Infrastructure, LLM Inference", "mainEntityOfPage": { "@id": "https://wavect.io/blog/netflix-vllm-triton-inference-stack/", "@type": "WebPage" }, "publisher": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] }, "url": "https://wavect.io/blog/netflix-vllm-triton-inference-stack/", "wordCount": 2293 } ``` ```json { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "item": "https://wavect.io/", "name": "Home", "position": 1 }, { "@type": "ListItem", "item": "https://wavect.io/blog/overview/", "name": "Blog overview", "position": 2 }, { "@type": "ListItem", "item": "https://wavect.io/blog/topics/ai-agents/", "name": "AI and agents", "position": 3 }, { "@type": "ListItem", "item": "https://wavect.io/blog/clusters/models-infrastructure/", "name": "Models and infrastructure", "position": 4 }, { "@type": "ListItem", "item": "https://wavect.io/blog/netflix-vllm-triton-inference-stack/", "name": "Netflix vLLM and Triton: Production Lessons | ", "position": 5 } ] } ``` ```json { "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "Netflix has documented an internal vLLM and Triton serving pathway, but its public engineering work also describes OpenAI embeddings in a separate MediaFM system. The evidence supports a workload-specific architecture, not the claim that every Netflix AI workload is self-hosted." }, "name": "Does Netflix run all LLM inference in-house?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "For the workload mix it re-evaluated in 2025, Netflix prioritized support for custom architectures, decoding extensibility, debuggability and researcher familiarity after open-source engine performance had become competitive. That is a workload-specific decision, not a permanent universal ranking." }, "name": "Why did Netflix prefer vLLM to TensorRT-LLM?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "The original vLLM V0 design ran custom Python logits processing per request after the GPU produced batched logits. CPU work grew with batch size and the Python GIL prevented parallel execution. Netflix moved to vLLM V1 batch-level processing and a multi-threaded C++ hot path." }, "name": "What caused the constrained-decoding bottleneck?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "No. vLLM can expose its own OpenAI-compatible server. Triton becomes relevant when a team wants a shared model repository, multiple backends, common control-plane integration or established server operations. It also adds a compatibility surface that must be pinned and tested." }, "name": "Do you need NVIDIA Triton to run vLLM?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "Consider it when custom decoding, custom models, strict data boundaries or steady utilization create a measurable advantage and a team can own upgrades, observability, rollback and on-call response. For low, bursty or standard demand, managed inference is usually the simpler starting point." }, "name": "When should a smaller company self-host LLM inference?" } ] } ```