Where Your LLMs Run: Cloud APIs, Routers, and Self-Hosting

"Which model should we use?" is the wrong first question. The better one is "where should our models run?" A production AI system rarely wants a single model from a single provider. It wants a frontier model for the hard reasoning, a small cheap model for the routine work, a private model for the sensitive data, and the freedom to move between them as prices, latencies, and capabilities change month to month. The way to get all of that without rewriting your application is to treat the provider as a runtime decision, not a hard-coded dependency.

This paper lays out the main options, what each is good for, and the one design choice that keeps them interchangeable. It is deliberately generic: the right mix depends on your workload, your data-sensitivity requirements, and your budget.

The provider is a runtime decision

Most modern model backends speak the same dialect: an OpenAI-compatible chat and embeddings API. That is a gift, because it means your application can talk to many different backends through one interface, changing only a base URL, an API key, and a model name. We build our systems around exactly this abstraction. The code asks for "a chat completion from the configured model"; configuration decides whether that resolves to a frontier cloud API, a routing service, a dedicated inference host, or a model running on our own hardware.

Once the provider is configuration rather than code, the interesting choices open up. You can run different providers per environment, per tenant, or per task. You can fail over from one to another. You can pilot a cheaper backend on a slice of traffic without touching the application. The rest of this paper is about what to put behind that interface.

Frontier cloud APIs

The default, and often the right default, is a frontier cloud API from a lab like OpenAI or Anthropic. You get the most capable models, the simplest operational story (no GPUs to manage), fast iteration, and features like large context windows and strong tool use. For hard reasoning, nuanced writing, and anything where quality is the constraint, this is where you start.

The trade-offs are cost, control, and data flow. You pay per token, which adds up on high-volume workloads; you are subject to the provider's rate limits and availability; and your prompts leave your network, which matters when the data is sensitive or regulated. None of these is disqualifying, but each is a reason not to route every call to a frontier API by reflex. A lot of production traffic is simple classification and extraction that a far cheaper model handles perfectly well.

Routers and aggregators

A routing layer such as OpenRouter sits in front of many providers and exposes them through a single API and key. Instead of integrating each lab separately, you point at the router and select a model by name. That buys you a few things: access to a broad catalog of models across vendors, easy price and performance comparison, and failover when one provider is degraded or rate-limiting you.

The cost is an extra hop and an additional party in the request path, which matters for latency-sensitive and privacy-sensitive workloads. Routers are most useful when you want breadth and flexibility, for example while you are still deciding which models suit which tasks, or when you want a single integration that can reach a long tail of models without a separate contract for each. Because the router is also OpenAI-compatible, it slots into the same abstraction as everything else.

Dedicated inference hosts

Between the frontier labs and running your own hardware sits a category of dedicated inference hosts, such as Fireworks and similar services, along with the newer GPU-focused clouds sometimes called neo-clouds. These serve open-weight models at high speed and low cost, often with better latency and price than a frontier API for the same open model, and without you having to operate the infrastructure.

This is the sweet spot for high-volume work on capable open models: bulk classification, summarization, embeddings, and the routine steps of an agent workflow. You get much of the cost profile of self-hosting with much less of the operational burden, and again through the same OpenAI-compatible interface. When a workload is large, latency matters, and a good open model is enough, a dedicated host is frequently the most economical answer.

Self-hosting with Ollama

Sometimes the model should run on infrastructure you control. Tools like Ollama make it straightforward to run open models on your own server, for example a GPU instance on EC2, and expose them through the same OpenAI-compatible interface as everything else. We use this pattern for shorter, simpler tasks and smaller models: intent classification, lightweight extraction, embeddings, and drafting, where a compact open model is sufficient and the volume or the data-sensitivity makes a self-hosted option attractive.

The reasons to self-host are concrete. Your data never leaves your network, which is decisive for regulated or confidential workloads and for air-gapped deployments. Costs become fixed and predictable rather than per-token, which pays off at high volume. And you are not exposed to a third party's rate limits or outages. The counterweight is operations: you own the instance, the scaling, and the GPU bill, so self-hosting earns its place on steady, well-understood workloads rather than spiky or frontier-capability ones. In practice, small self-hosted models handle the routine tier while calls that genuinely need a frontier model still go to the cloud.

Match the model to the job

None of these options is the answer on its own. The real design is a mix, chosen per task. A cheap, fast, possibly self-hosted model classifies an incoming request and decides where it goes; a frontier model handles the small fraction of work that genuinely needs its capability; a dedicated host serves the high-volume middle. This mirrors the two-stage routing we use in multi-agent systems, where a cheap classifier fronts expensive execution, and it keeps the total cost proportional to the difficulty of the work rather than to its volume.

Deciding the mix is an empirical exercise, not a one-time verdict. Route by the properties that actually matter for each task: required capability, acceptable latency, cost per call at your volume, and whether the data can leave your network. Measure quality on your own inputs, because a smaller or open model that looks weaker on a benchmark is often indistinguishable from a frontier model on a narrow, well-scoped task, at a fraction of the price.

Keep it swappable

The single decision that makes all of this practical is the abstraction we started with. When every model call goes through one OpenAI-compatible interface, and the provider, base URL, and model are configuration rather than code, moving between a frontier API, a router, a dedicated host, and a self-hosted model is a config change, not a rewrite. You can shift a workload from a frontier API to a self-hosted model when volume grows, fail over to a router when a provider degrades, or pilot a cheaper backend on part of your traffic, all without touching application logic.

That flexibility is worth designing for from the start, because the landscape moves quickly. New models, cheaper hosts, and better open weights arrive constantly, and the mix that is optimal today will not be optimal in six months. The teams that benefit are the ones who can adopt each improvement by changing configuration, and who tested against more than one backend early enough that no single provider is load-bearing. Own the abstraction, and where your LLMs run becomes a lever you can pull, rather than a decision you are stuck with.