Architecture · 8 min read · Aug 12, 2026

Not every task needs the big model.

Self-hosting a small model is either an obvious win or an expensive detour, and which one it is depends on facts most teams do not check before starting. Here is the honest accounting on both sides.

The three real reasons to self-host

There are good reasons and there are reasons that sound good. The good ones are specific:

  • The data cannot leave. A regulatory or contractual boundary that no provider agreement satisfies. This is the strongest case and it is not really a cost decision at all.
  • Volume is enormous and the task is narrow. Millions of near-identical classifications where per-token pricing compounds into a serious number.
  • Latency must be local. A network round trip is genuinely too slow, or the system must keep working without connectivity.

The reason that sounds good and usually is not: cost. Self-hosting frequently ends up more expensive than API pricing at anything below high, sustained volume, for reasons covered below.

Capability is not the constraint people expect

Small open models have become genuinely capable at bounded tasks — classification, extraction, routing, summarising a short document, tagging against a fixed taxonomy. For that class of work, the gap to a frontier model is often narrow enough that it does not affect the outcome.

Small local models handle bounded transformation: classification, extraction, routing and tagging, and redaction before data leaves the boundary. Frontier models handle open-ended judgement: multi-step reasoning, complex instructions, long-context work and broad world knowledge.
Small models do a thing to a piece of text. Frontier models decide what to do.

Where the gap remains wide is multi-step reasoning, instruction-following under complex constraints, long-context work, and anything needing broad world knowledge. A useful way to think about it: small models are good at doing one thing to a piece of text. Frontier models are good at deciding what to do.

Most production AI work is bounded transformation, not open-ended reasoning. Which is why routing by task rather than defaulting to one model is usually the cheapest correct answer.

The costs of self-hosting that get missed

The comparison is usually GPU rental against API spend. That is one line of several.

CostWhy it gets missed
Idle capacityYou pay for the GPU continuously; APIs bill per call
Engineering timeServing, batching, quantisation and scaling are real work
Model upgradesNobody upgrades your model for you
On-callYour inference is now a service you operate at 3am
EvaluationYou own proving the model still works after every change

Idle capacity is the one that decides most cases. API pricing is consumption-based, so an application with peaky or modest traffic pays nothing between requests. A rented GPU is billed whether or not anything is being asked of it. Below high sustained utilisation, that difference alone usually settles the comparison — and it argues for self-hosting on continuous batch workloads rather than on interactive ones.

The hybrid is usually the answer

This mirrors the model-tier routing argument: match the task to the cheapest thing that does it correctly. A small local model handles the high-volume bounded work — classification, extraction, redaction before data leaves your boundary. A frontier API handles the reasoning, the exceptions and anything open-ended.

One pattern is worth calling out because it solves a governance problem rather than a cost one: running a small local model as a pre-processing step that strips or tokenises identifiers before anything reaches an external provider. You get frontier capability on the hard part while the sensitive fields never cross your boundary. That is often what makes an otherwise-blocked deployment approvable.

How to decide without a pilot

Three checks, in order:

  • Is there a hard boundary? If data genuinely cannot leave, self-host and stop comparing costs — that decision is made.
  • Is the task bounded and high-volume? Bounded plus sustained volume favours local. Varied or spiky favours the API.
  • Do you have someone to operate it? If nobody owns inference as a service, self-hosting adds an operational obligation you have not staffed.
Three checks in order: is there a hard data boundary, is the task bounded and high-volume, and is anyone resourced to operate inference as a service. The third check is highlighted as the one that most often settles the decision.
Three checks, in order. The third ends more of these discussions than the other two.

That third question ends more of these discussions honestly than the other two. Self-hosting is not a purchase, it is a commitment to run something — and the same test applies as with any build: are you willing to own it in three years?

Dealing with this in your own group?

We answer scoping questions before there's a contract in sight — including the ones about cost and data handling.

Book a demo See what we've shipped

Questions

Short answers,
in full.

The questions this article gets asked most, answered so each one stands on its own.

Talk to us
When should we self-host an AI model instead of using an API?

Three cases justify it: data that genuinely cannot leave your boundary for regulatory or contractual reasons, very high sustained volume on a narrow task, or a hard local-latency or offline requirement. Cost alone is usually not a good reason — below high sustained utilisation, self-hosting frequently costs more than API pricing once idle capacity and engineering time are counted.

Are small open models good enough for business use?

For bounded tasks, often yes — classification, extraction, routing, tagging against a fixed taxonomy, summarising short documents. The gap to frontier models remains wide for multi-step reasoning, complex instruction-following, long-context work and tasks needing broad world knowledge. Small models are good at doing one thing to a piece of text; frontier models are good at deciding what to do.

Is self-hosting an AI model cheaper?

Often not. The usual comparison counts GPU rental against API spend and omits idle capacity, engineering time for serving and scaling, model upgrades, on-call ownership and evaluation. Idle capacity decides most cases: APIs bill per call, a rented GPU bills continuously, which favours self-hosting for continuous batch workloads and disfavours it for peaky interactive ones.

Can we combine local models and API models?

That is usually the right shape. Route bounded high-volume work to a small local model and reasoning or exceptions to a frontier API. One particularly useful pattern is running a local model as a pre-processing step that strips or tokenises identifiers before anything reaches an external provider — frontier capability on the hard part, sensitive fields never crossing your boundary.

Keep reading

More from the desk.

CommercialsHow to measure whether AI is actually paying back ArchitectureContext engineering: the part that decides whether it works OperationsVoice AI on the phone: what works and what still doesn't CommercialsCan agents replace software seats? Sometimes, and not how you think ArchitectureMulti-agent systems: when they help and when they hurt OperationsHow to tell whether your AI system actually works