Self-Hosting LLMs in the EU: When Open Weights Actually Pay Off

On a spreadsheet, self-hosting an open-weight LLM looks like an easy win. Rent a GPU for a few dollars an hour, run a free model, stop paying per token. The bill goes flat and you own the stack. That is the pitch, and it is half true. The half it leaves out is the part that decides whether you save money or quietly lose it: the GPU is the cheap part. The expensive part is the engineer who keeps the inference server alive, the eval harness that proves a quantised model still answers correctly, and the upgrade cadence that does not stop when a better open model ships two months later.

This is an engineering and process perspective, not a vendor pitch. The numbers below are directional, drawn from public 2026 trends, and you should re-check them against live quotes before you commit, because GPU rates and token prices both move month to month. We set up internal AI on client infrastructure under our AI Enablement work, so the trade-offs here are the ones we actually weigh with clients, not a theoretical model. This post extends our token-cost playbook, which only touches self-hosting, by going deep on the one question that decides it: when does bringing inference in-house beat a hosted API?

What does self-hosting an LLM actually cost, beyond the GPU?

The GPU line item is the one everyone quotes, but products and configurations are not interchangeable. As reviewed on 2 September 2026, Scaleway listed one H100 PCIe instance from €2.86 per GPU-hour, while OVHcloud listed its one-H100 instance at €3.10 per hour, both before tax. At 730 hours, that is about €2,088 or €2,263 per month before storage, traffic, redundancy, support, and operations. Compare GPU type, CPU, RAM, storage, region, SLA, commitment, and availability before comparing price.

That number is real, and it is also the smallest part of the total. The costs that decide the math are the ones the spreadsheet skips:

• Engineer time. Someone has to stand up the inference server, tune batch sizes, manage GPU drivers and CUDA versions, handle model loading and rollback, and keep it running. This is not a one-time setup. It is ongoing operations, and in the EU a competent ML-ops engineer costs far more per month than the GPU does.
• Redundancy. One GPU is a single point of failure. Production usually means at least two, plus a failover plan, which roughly doubles the hardware line and adds load-balancing work.
• Eval and quality assurance. A self-hosted model is your responsibility when it regresses. You need an eval harness that proves the model holds quality after every quantisation choice and every version bump. Without it you are flying blind.
• Upgrade cadence. Open weights move fast. A model that is competitive today is mid-tier in a quarter. Staying current is recurring engineering work, not a fire-and-forget install.

Add it up and the honest framing is the one from our token-cost work: the break-even is governed by engineer time, not GPU rack rate. The model is cheap to run. The discipline around it is not.

Where is the break-even against a hosted API?

There is no single break-even number, because it depends entirely on which API you are replacing. The spread is wide enough that getting this wrong is the most common self-hosting mistake.

• Against any hosted API, calculate rather than cite a threshold. Divide the full monthly self-hosted cost by the API's weighted cost per million equivalent input and output tokens, then divide by operating days. Include cache, batch, tool, and reasoning charges where they apply.
• Then validate equivalent quality and service. A small quantised open model is not a cost-equivalent substitute for a frontier model unless it meets the same task-quality, latency, context, safety, and availability target. Tokens per day alone cannot establish break-even.

The practical read: bursty traffic can make dedicated self-hosting less attractive because deployed capacity bills while idle, whereas usage-billed APIs charge for requests, though minimum commitments, storage, tools, and support may still apply. Calculate utilization, queueing, redundancy, traffic mix, and engineer cost with equivalent quality and SLA targets. We cover the broader cost curve in what cheap tokens change in your AI architecture.

"Most teams self-host too early. They price the GPU, not the engineer who has to keep it alive, and a few months in the hosted API would have been cheaper and less work."

When does data residency force self-hosting regardless of cost?

Cost is only one axis. The other is governance, and for some EU workloads it overrides the math entirely. If you process personal or regulated data and you cannot send it to a non-EU endpoint, then the cost comparison is moot. The question stops being "is self-hosting cheaper" and becomes "what is the cheapest compliant option."

GDPR does not impose blanket EU-only processing. Chapter V permits third-country transfers through adequacy decisions or appropriate safeguards, alongside the rest of the processing assessment. A data-processing agreement, purpose limitation, security controls, records of processing, a transfer mechanism where needed, and an accurate data-flow map are relevant building blocks, not automatic compliance. Self-hosting can reduce model-provider exposure, but a cloud, datacenter, support vendor, telemetry service, or administrator may still be a recipient or processor. It also moves serving, logging, access control, patching, and rollback onto your team. We go deeper in EU data residency for AI apps.

There is a regulatory layer on top. The EU AI Act is phasing in. General-purpose AI model obligations already apply, Article 50 transparency applies from 2 August 2026, stand-alone high-risk duties from 2 December 2027, and product-embedded high-risk duties from 2 August 2028. The practical takeaway for an EU team is not only a date. Controlling where your model runs is becoming a governance asset, not just a cost line, and self-hosting is one way to keep that control in your own hands. Confirm the classification and role against official sources before you build a compliance claim on them.

What is the production stack if you do self-host?

If volume or compliance has decided it for you, the 2026 production stack is well established, and it is not the same thing you used to prototype on your laptop. A local single-stream runner is fine for experiments and wrong for production, because it leaves most of the GPU idle.

This section owns the broad stack choice. For the narrower operating question, our Netflix vLLM and Triton production analysis covers constrained decoding, version pinning, rollout safety, model caching and unified metrics.

• Inference engine. vLLM supports continuous batching, PagedAttention, and multiple parallelism strategies, but no universal throughput multiplier follows. Benchmark vLLM, SGLang, TensorRT-LLM, or another supported engine on the exact model, precision, input/output mix, concurrency, and latency target. A shared KV-cache result such as the reported 14x time-to-first-token improvement is specific to that model, topology, and cache-hit setup, not a general serving promise.
• Precision and quantisation. FP8, INT8, and 4-bit formats can reduce weight memory and sometimes raise throughput, but memory savings, kernel support, latency, and quality loss depend on model, hardware, quantizer, calibration, and task. Do not rank AWQ, GPTQ, or FP8 from a generic percentage. Benchmark the candidate artifact and gate it on your own evals.
• The model itself. Llama, Qwen, DeepSeek, and Mistral-class open weights are the usual candidates. Picking among them is its own decision, and we work through it in our open-weight model comparison. If phone or laptop memory is the hard constraint, our Bonsai 27B review separates the 3.9 GB weight headline from real runtime memory and task quality. Model weights are only part of the memory bill: if you run retrieval, the embedding index has its own footprint, and our guide to cutting RAG vector memory 16x shows how data-oblivious quantization shrinks it.

How do you know the cheaper path actually held quality?

This is the part teams skip and then regret. Every choice in a self-hosted stack, the model, the quantisation, the batch settings, can quietly drop quality, and a cheaper path that silently answers worse is the most expensive outcome of all. The only honest defence is an eval harness that scores your actual tasks, not a public benchmark.

We are deliberately humble about this. Building and maintaining good evals is harder than standing up the inference server, and most of the ongoing engineering cost of self-hosting lives here rather than in the GPU. You need a representative task set, a scoring method you trust, and a gate that runs before every model or quantisation change. Without it you cannot tell whether your FP8 swap cost you two points of accuracy on the cases that matter. This is the same discipline we apply on production AI engagements such as Twinsoft AI: the model choice is only safe on top of an eval that proves it held the bar.

So should you self-host? A decision checklist.

Run these questions in order. If the honest answer to the first two is no, default to a hosted API and revisit later.

1. Is data residency forcing your hand? If you cannot legally send the data to a non-EU endpoint and no compliant EU-hosted API fits, self-hosting may be the cheapest compliant option regardless of token math. This alone can decide it.
2. Is your volume high and steady? Compare full monthly cost with the weighted hosted bill at equivalent quality and SLA. Bursty or low utilization often favors usage billing, but there is no universal token threshold.
3. Do you have the ops capacity? Self-hosting is recurring engineering work: drivers, redundancy, upgrades, monitoring. If your team cannot own that without dropping product work, the GPU savings evaporate.
4. Can you prove quality with evals? If you cannot measure whether a quantised open model holds your quality bar, you are not ready to depend on one in production.
5. Have you priced the full picture? GPU plus redundancy plus engineer time plus eval upkeep, against the all-in API cost. Compare totals, not the GPU line against the token line.

For most early and mid-stage products, the answer is still: stay on a hosted API, and pick an EU-resident one if residency matters. Self-host when volume or compliance forces the question, not before. If you want help running that comparison on your own numbers and infrastructure, that is exactly what our AI Enablement work is for.