Local AI models compared: the guide

“Local AI” is on the agenda of every other strategy meeting by now. The reason is concrete: GLM-5.2, Kimi and DeepSeek put several frontier models under open licences at once, and Moonshot’s Kimi K3 is the largest open model there has ever been. Only nobody has time to work through 49 models and seven quantisation levels.

Here is when running your own pays off, and when the cloud stays the calmer choice.

What local AI models are

Local AI models are openly licensed language models you download without a licence fee and run on your own hardware. Tools like Ollama, LM Studio or llama.cpp load the model, inference runs on your GPU. “Open weight” means the weights are freely available, the training process is not necessarily.

This is where the first misconception starts. “Local” sounds like a data centre, like a GPU farm of your own, like an ops team nobody has. In reality a usable 14B model runs on a single consumer GPU. “Local” means exactly one thing: the request does not go through a third-party cloud API.

Open weight is like a car you bought instead of one you rented. Once acquired it drives without per-kilometre billing, but maintenance and parking are on you.

Data sovereignty is an axis, not a switch

This is the mental model that carries the whole decision. There is no “cloud or safe”, there are three levels:

• Level 1: cloud API with EU data processing. Data stays in the EU, the backend is somebody else’s.
• Level 2: open source, EU-hosted. The same open models, on EU servers, with a contract, without your own ops team.
• Level 3: truly local or air-gapped. Your own hardware, inference never leaves the building.

On LinkedIn it sounds like this instead: “US cloud means insecure, so we need our own GPU farm.” That is the wrong question, because it presses an axis with three levels into two boxes. Start with that binary logic and you block the simplest answer: the same sovereignty is available EU-hosted, without a server room.

For completeness, European models exist too: Teuken 7B from Fraunhofer IAIS, built in the OpenGPT-X project and trained on all 24 EU languages, and EuroLLM 9B from the UTTER project. Both are Apache 2.0 and strong in German. Both also cap out at 4K of context, and that is the hard limit today: it is not enough for a longer document. As proof that European training works they matter, as an everyday workhorse they do not yet.

And something else gets overlooked along the way. Open weights mean independence from every vendor roadmap. No vendor changes the terms overnight, no model gets discontinued and forces you into a migration. Gemma today, Qwen3 tomorrow, the next release the day after: the tooling stack stays, only the weights file changes. That is not a data protection argument but a strategic one, and it holds even when inference runs EU-hosted.

Seven criteria instead of a question of belief

Hold this table against your use case, line by line.

Criterion	Local	Cloud API	Note
Sensitive data (personal, health, client)	Yes	Partly	Local eliminates the Schrems II risk entirely, cloud only with an EU contract
High request volume	Yes	Partly	From stable continuous load onwards, running your own beats pay-per-token
Air gap or offline setup	Yes	No	Only possible locally, for instance in public agencies or shop floors without internet
Fast iteration, changing models	Partly	Yes	The cloud delivers new models without a hardware commitment
Frontier performance in reasoning and coding	Partly	Yes	Top benchmark scores are not yet reachable 1:1 locally
Single user instead of a team	Partly	Yes	Local pays off only at team utilisation
Low latency	Partly	Partly	Local is strong with small models, at 70B it tends to be slower

So if you process client data you do not need a GPU farm, you first need a clear sovereignty level and then the right model. Which models exist and what they need is in the filterable model database, updated daily.

The honest cost picture

Cloud costs grow linearly with usage. Local inference has fixed hardware costs plus electricity. As an order of magnitude, the maths tips towards running your own at stable, high volume, roughly from 50 million tokens per month.

That is a rule of thumb, not a threshold. The real break-even depends on model size, utilisation and the price of electricity. What matters is the logic, not the second decimal place: low volume argues for the cloud, high continuous volume for your own hardware.

What hardware you need

“Local” does not mean “data centre”. Three classes cover almost every need. The budgets are our own research (German prices incl. VAT, as of May 2026), an order of magnitude, not a list price guarantee.

Class	Budget	GPU	Runs
Consumer	€2,000–4,500	RTX 5070 (12 GB) to RTX 5090 (32 GB)	Models up to roughly 24B at Q4, MoE up to roughly 30B
Workstation	€8,000–18,000	RTX 6000 Ada (48 GB) or 2× RTX 5090	Models up to roughly 70B at Q4, 5 to 15 concurrent users
Server	from €40,000	NVIDIA B200 (192 GB), H100 (80 GB) or H200 (141 GB)	Frontier MoEs locally

For the vast majority of companies the workstation class is enough. Apple Silicon (M4 Max with 64 to 128 GB of unified memory) is the lowest-latency single-box alternative without a heavy electricity bill. The rule of thumb for memory: roughly 0.55 GB per billion parameters at Q4 quantisation. A 14B model therefore runs on a 12 GB GPU, a 32B model on 24 GB.

There is one exception, and missing it costs money. Some models are trained quantisation-aware from the outset instead of being scaled down afterwards. For gpt-oss and Kimi K3 the rule of thumb badly understates the real footprint: it works out 11.5 GB for gpt-oss 20B, while the vendor states 16 GB. Buy your card on that basis and you buy one size too small. This is why the model database does not apply the formula to these models. It carries the vendor’s figure, and where the vendor never published one, as with Kimi K3, the weight of the files themselves.

What you run them with

Tool	Role	Good for	Not for
Ollama	The beginner’s standard	A quick start on laptop and desktop	Production with high throughput
LM Studio	GUI without a terminal	Local experiments, comparing models	Server deployment
llama.cpp	The engine beneath it all	Maximum control, custom builds	Anyone who wants to start fast
vLLM	The production server	Multi-user, OpenAI-compatible API	A single user on a laptop

For the first steps Ollama or LM Studio is enough. As soon as several people access it at the same time, the path leads to vLLM with an OpenAI-compatible API.

Is local AI automatically GDPR-compliant?

No, and this is the point where I see the most misconceptions.

Running locally removes the third-country transfer and with it the Schrems II question: as long as inference runs on your own hardware, inputs never leave your own infrastructure. That drops the transfer under Art. 44 GDPR and the articles following it. The technical and organisational measures under Art. 32 are often easier to satisfy on your own hardware.

Two points remain nonetheless. Purpose limitation, a deletion concept and documentation continue to apply unchanged. And as soon as the hardware sits with an external host, via colocation for instance, you need a data processing agreement under Art. 28 with that operator. The inference itself is not processing on behalf of a controller: the contract attaches to the hardware, not to the model. With a true air gap that falls away too.

How to verify a vendor’s place of processing in concrete terms is in EU hosting and GDPR.

Both routes in one platform

The binary question dissolves once both routes sit side by side. That is exactly what RelationFlow does.

The first route is Bring Your Own Model: you connect your own model endpoint with your own key, OpenAI-compatible like vLLM or llama.cpp, or a secured Ollama endpoint. The model runs on your hardware or with the host of your choice, billed through your contract.

The second route is the same open models, EU-hosted: no ops team of your own, data in the EU regardless, with a contract and without a US transfer.

The decisive point is not one route or the other. It is that you switch per use case without swapping the stack. Why that is more than convenience is in Why one AI model is never enough.

Three steps

First check in the hardware calculator which models fit your existing hardware at all. Then compare ELO, memory footprint and licence side by side in the model database. And finally assign every use case to one of the three sovereignty levels, before you talk about a specific model.

In that order you hold a defensible answer to the local-or-cloud question before the first audit asks it. The other way round you spend three weeks discussing model names and still have no decision at the end.