An on-premises LLM (also written on-premise or on-prem LLM) is a large language model whose weights and serving stack run on hardware you control: in your data centre, in a private cloud, or in a dedicated VPC in your country. Text is not sent to an outside provider and returned with an answer. This guide explains the practical difference, when the option is worth it, and when it is not.

The difference from a cloud API, in four points

  • The data path. With a cloud API, every question and every attached document leaves your network for the model provider. On-premises, the text, the indexes and the logs stay on your hardware, and the system can be cut off from the internet entirely.
  • The cost model. A cloud API charges per call, so the bill grows with use. On-premises, you buy capacity you size in advance, so the cost is predictable, and whether it pays off depends on your volume.
  • Control over change. A cloud provider can update or retire a model at any time. On-premises, the model changes only when you decide, after the new version passes your tests.
  • Dependence on the network. A cloud API stops when the connection drops. An on-premises model works on isolated sites and in environments where no outside connection is allowed.

What it is actually made of

The model alone is not enough. A serious on-premises deployment has five layers, and every one of them has to run in the same place, or the data path leaks back out through a side door:

  • The model and its serving stack on one or more GPU nodes, depending on the number of users and the size of the model.
  • Retrieval over your documents, respecting access rights, so nobody sees in an answer what they are not allowed to see in the source.
  • A safety layer: a classifier and policy rules that can be reviewed.
  • Logs: every question, answer and action recorded inside your environment.
  • The team that runs it: someone to watch quality, refresh the indexes and test every new version before it goes live.

Each layer and its security control are covered in our article on the five layers, and what air-gapped really means in this one.

What hardware it needs

The main constraint is GPU memory. A model’s weights need roughly its parameter count times the bytes per parameter: about two bytes at 16-bit precision, one byte at 8-bit, and half a byte at 4-bit. So a 14-billion-parameter model needs about 28 GB for its weights at 16-bit, about 14 GB at 8-bit, and about 7 GB at 4-bit.

On top of that, every active conversation needs working memory (the KV cache), which grows with the number of simultaneous users and the length of the documents in play. That is why sizing starts from your usage, not from the model: how many people at once, how long their documents are, and how fast answers must come back. Lower precision saves memory but can cost quality, especially in Arabic, so test the quantised version you will actually run, not the full one.

When it is the right choice

  • Your data is regulated: personal, financial, health or government data, and your country's law restricts moving it abroad.
  • Use is high and steady, so fixed capacity becomes cheaper than paying per call.
  • You need behaviour that stays stable and evaluated, not behaviour that shifts with a provider's updates.
  • You operate on isolated sites or on networks where outside connections are not allowed.

And when it is not

  • A small experiment on non-sensitive data: a cloud API is faster to start with.
  • Nobody will own running the system after launch, neither on your side nor the vendor's.
  • Use is light and occasional, so the capacity you buy sits idle.

And Arabic in particular

Many models available to run on-premises were trained mostly on English, and read formal Arabic better than dialects. Before any decision, test the model on your own documents and your customers' real messages, in the formal Arabic and the dialects they actually write, and agree the pass criteria before you buy. That is what we do in the Discover phase with Orchestra, our Arabic model that runs on-premises.

Five questions for any vendor

  1. Where exactly does each of the five layers run?
  2. Does any part of the system call an outside service, even for licensing or telemetry?
  3. How do updates arrive, and who decides to apply them?
  4. How is the system evaluated before launch, and on what data?
  5. Who runs it after launch, and what happens when quality drops?

Frequently asked questions

Can an on-premises LLM run without an internet connection?

Yes. It can be fully air-gapped, with updates arriving on media you control and check before applying them. Details in what air-gapped really means.

Is an on-premises LLM as capable as ChatGPT?

The largest cloud models are stronger at open-ended general tasks. But for a defined task on your own documents, a smaller model with good retrieval can meet the pass criteria you agree, and the only way to know is to test it on your data. See how to evaluate an Arabic LLM.

Is it cheaper than a cloud API?

It depends on volume. With heavy, steady use, fixed capacity often works out cheaper; with light use, a cloud API is cheaper. Work it out with your own numbers in our on-premises vs cloud comparison.

Does on-premises have to mean my own data centre?

No. What matters is that you control the hardware and the data path: your data centre, a private cloud, or a dedicated VPC in your country. The test is that text never leaves your environment for the model provider.

If you decide the model should run inside your organisation, this is what it actually takes: an Arabic language model inside your organisation.

Related pages