Self-Hosted LLMs in 2026: Private AI for EU Firms

For years, the honest advice about self-hosted LLMs was simple: don't, unless compliance forces you to — open models were a full capability class behind the APIs, and running them well was a specialist ops burden few teams could justify. In 2026 that advice expired. The capability gap closed, the licenses opened, the tooling matured into ordinary infrastructure — and for a growing set of European companies, self-hosting flipped from compliance workaround to the technically and economically better answer.

We serve private models for clients on EU infrastructure — vLLM and Ollama are standing parts of the stack we ship. Here is what changed, when self-hosting actually wins, and the architecture we recommend.

The models caught up

Look at the last twelve months of open-weight releases:

• GLM-5.2 (June 2026, MIT license) — currently the strongest text-only open-weights model by independent rankings, with benchmark results competitive with frontier APIs on coding and terminal tasks.
• DeepSeek V4 (April 2026, MIT license) — a 1.6T-parameter mixture-of-experts with a 1M-token context and open weights, plus a smaller Flash variant for cheap serving.
• Qwen 3.5 (February–March 2026, Apache 2.0) — natively multimodal flagship plus a family of small dense models, with strong multilingual coverage that matters for German-language workloads.
• Gemma 4 (April 2026) — Google's open family moved to Apache 2.0, with sizes from on-device to a 31B dense model near the top of open leaderboards.
• And the 2025 foundations still doing daily work: OpenAI's gpt-oss (Apache 2.0, o4-mini-class reasoning on a single 80GB GPU), Meta's Llama 4, Moonshot's Kimi K2 line.

The pattern: MIT and Apache 2.0 licenses, mixture-of-experts efficiency, and small variants that run on hardware you already own. Sparse MoE architectures mean a nominally huge model activates only ~2–5% of parameters per token — frontier-adjacent quality at mid-range serving costs.

Note what we are not claiming: that open models beat the best APIs at everything. For the hardest reasoning, frontier APIs still lead. The claim is narrower and more useful — for the bounded tasks that dominate business AI (extraction, summarisation, retrieval-augmented answering, agent steps), current open models are simply good enough that where they run becomes the deciding factor.

The serving layer became boring — in the best way

Two projects carry most of the self-hosted world, and both had a strong first half of 2026:

• vLLM — the production serving standard: continuous batching, quantization support, speculative decoding, day-one support for new open releases (DeepSeek V4 and the newest open models landed within weeks). This is what we deploy for client workloads with real throughput requirements.
• Ollama — from laptop tool to platform, with major speedups on Apple Silicon and consumer GPUs this year. This is what we use for prototypes, edge cases, and "run it on the office server" deployments.

Between them, the ops burden argument from 2024 has mostly evaporated: serving an open model today is closer to running Postgres than to research infrastructure.

Why EU companies specifically should care

Sovereignty stopped being a niche concern. In a March 2026 Red Hat survey of European IT leaders, 77% said public policy should mandate open-source principles — transparency, auditability — to achieve AI sovereignty. The European Commission's own research describes open-source AI as a lever for European innovation and sovereignty. And EU enterprise AI adoption keeps climbing — 20% of enterprises in 2025, per Eurostat, up from 13.5% a year earlier — which means ever more European customer data flowing through AI systems somewhere.

Somewhere is the issue. With a self-hosted model, the answer is: your infrastructure, full stop. Prompts, documents and outputs never transit a third-party API, no cross-border transfer analysis, no dependency on a US provider's retention promises. That is why we host everything in the EU — and why our AI reference architecture keeps inference inside it.

When to self-host — our actual decision rule

Across 25+ delivered projects, we apply a simple three-question test:

1. Is the data sensitive enough that "trust the API's terms" is not an answer? Health, finance, HR, legal, anything under strict GDPR interpretation → self-host, or at minimum zero-retention API tiers.
2. Is the task bounded? Extraction, classification, RAG answering, agent steps with defined tools → open models excel. Open-ended frontier reasoning → the API may still earn its keep.
3. Is there sustained volume? Self-hosting has a fixed floor; APIs price per token. At steady production volume the economics cross over surprisingly early — and MoE small variants keep pushing that point down.

The honest architecture for most clients is hybrid: a self-hosted workhorse on vLLM inside EU-hosted AWS for everything touching sensitive data, an API escape hatch (zero-retention tier) for the rare tasks that need it, and routing logic that makes the choice per request — not per company religion. Because we build on standard interfaces, swapping models stays an afternoon's work, not a migration; the Model Context Protocol does the same for tools.

What it actually costs (rules of thumb, not vendor math)

Budget conversations about self-hosted LLMs go wrong in both directions — teams either assume research-lab costs or believe a vendor's "runs on a laptop" slide. Our rules of thumb from operating these systems:

• Small workhorses are genuinely cheap. The 4B–30B class (Qwen small dense models, Gemma 4, gpt-oss-20b) runs well on a single mid-range cloud GPU. For extraction, classification and RAG answering at company scale, one EU-region GPU instance — often a few hundred euros a month, less with quantization — covers a surprising amount of daily work.
• The big MoE flagships need planning, not miracles. Models in the GLM-5.2 / DeepSeek V4 class want multi-GPU nodes with high memory — real money, justified when workloads are heavy, sustained and sensitive. Quantized variants (native INT4/FP4 releases are becoming standard) roughly halve the footprint at minor quality cost.
• Throughput economics beat latency economics. vLLM's continuous batching means cost per token collapses with utilisation. A GPU serving one user is expensive; the same GPU serving your whole back office is not. Sizing is a utilisation question — measure your token volume for two weeks before buying anything.
• The hidden line item is evaluation, not hardware. The teams that overspend are the ones re-benchmarking by anecdote every time a new model drops. A fixed golden-question set turns "should we switch to the new release?" into a one-hour automated answer.

And one accounting note that CFOs appreciate: API costs are an operating expense that scales with success. Self-hosted inference is closer to infrastructure — its cost curve flattens exactly when your usage grows, which is when the API bill would have exploded.

Questions clients keep asking us

"Do we need our own GPUs?" No — EU-region cloud GPUs on AWS work fine and stay GDPR-clean; data residency comes from where inference runs, not from owning hardware. On-prem is for when policy demands it.

"Which model should we standardise on?" Wrong question — the family churns every quarter (see the release list above). Standardise on the serving layer and evaluation set; then model upgrades become a benchmark run, not a project.

"How do we handle model updates and security patches?" The serving layer updates like any infrastructure component — pinned versions, staged rollouts, a canary workload. Model weights are immutable artifacts: you version them, benchmark the candidate against your golden set, and promote deliberately. This is tangibly calmer than the API world, where the model behind your product can change behavior on a provider's schedule, not yours — silent model drift is a real operational risk that self-hosting simply deletes.

"What about multilingual quality — our business runs in German?" Test it explicitly; do not trust English benchmarks. The strongest current open families are genuinely multilingual, and for the bounded tasks that dominate business automation, German performance is production-ready — but your evaluation set should contain your German documents, your dialect of bureaucracy included. Ours do.

"Is a self-hosted model less safe?" It is differently safe: you gain data control, you own guardrails and monitoring. We ship those as part of the system — logging, scoped tools and human checkpoints, the same discipline as our production agent work.

The short version

2026's open-weight wave — GLM-5.2, DeepSeek V4, Qwen 3.5, Gemma 4 — plus boring, excellent serving via vLLM and Ollama means private AI is no longer a capability sacrifice. The three-question test (data sensitivity, task boundedness, sustained volume) decides the architecture; the hybrid pattern with an EU-hosted workhorse and a zero-retention escape hatch covers nearly every client case we see. For EU companies with sensitive data, self-hosted LLMs on EU infrastructure are now frequently the strongest option, not the compromise. The differentiator is no longer access to models; it is the engineering around them.

This post is part of our practical guide to AI automation for businesses. If you want private AI running on infrastructure where your data already lives — book a call with Nexuri. We respond within 24 hours.