"Can we just run the model ourselves?" is a question that arrives the moment an AI pilot gets a real bill. The honest answer is a short piece of arithmetic, plus a few questions about the customer.
Here is the picture I use, then the sums.
Taxi or car?
- An API is a taxi. You pay per ride, which here means per token. Use nothing and you pay nothing.
- Self-hosting is a car. You rent GPUs (the chips that run the model) and run an open-weight model yourself. The cost is fixed each month, and extra trips are almost free.
Running a model to get an answer is called inference. The question is which side of the crossover point you are on.
There is a middle option too: a managed service that runs inside your own cloud account. Think of a rental car with a driver. The data stays with you, and someone else keeps the GPUs running.
Ask these questions in order
- Data rules first. If text may not leave the customer's own cloud account, a public API is out before cost matters.
- Then skills. Who can run GPUs in production? If nobody, lean towards a managed service in their account.
- Then volume. High and steady use favours the car.
- Not measured yet? Start on the API and measure.
The break-even sum
All numbers below are made up for the example. Use your own.
- API price: $2 per million tokens
- Pilot use: 800 million tokens a month, so the bill is 800 x $2 = $1,600
- GPUs: $4,000 a month for one set
- People (part-time engineer and on-call time): $2,000 a month
- One set of GPUs can serve 4,000 million tokens a month
Self-hosting costs $4,000 + $2,000 = $6,000 a month, used or not.
break-even = fixed cost / API price per million tokens = $6,000 / $2 = 3,000 million tokens a month The pilot uses 800 million, far below 3,000. Stay on the API.
Now try ten times the use, which is 8,000 million tokens:
- API: 8,000 x $2 = $16,000
- One set serves only 4,000 million, and GPUs come in whole sets, so you need two: 2 x $4,000 = $8,000
- Add $2,000 for people: $10,000
Self-hosting saves about $6,000 a month on paper. Check quality and on-call before you celebrate.
Owning the car is three layers of work
- Runtime: the model running on a GPU.
- Infrastructure: many copies, shared load, GPU supply.
- Tooling: releases, watching for problems, tests, cost counting.
The weakest layer sets the limit. And someone has to be on call at 3 a.m. That person's time is a real cost.
Caveats
- The sum is a first filter, not a quote. It ignores quality, traffic spikes and setup time.
- Idle GPUs cost the same as busy ones. Low use means you pay for hardware that does nothing.
- "Open-weight means free" is a mix-up. The download is free. GPUs and people are not.
- Prices change, and tokens are priced per million. Re-run the sum with current numbers.
- A cheaper model that fails your test questions is not cheaper.
- You do not have to pick one for good. Many teams self-host the steady load and rent the spikes.
Try it with your own numbers
The full chapter has a decision tree, diagrams of the three layers, and a break-even calculator where you can change the price, GPU cost and volume and watch the answer move. It is free with no signup: Why run models yourself?. It is one of the free chapters of Deployed Lab, a course on AI engineering and the forward deployed engineer role.

