A company with a few hundred employees usually reaches the same point about a year after the first chatbot pilot. The pilot worked, legal does not want customer data leaving the country, the API bill keeps growing, and someone asks whether it would be cheaper to "just buy a GPU server". The honest answer is that it depends less on the hardware than on which workload you are talking about.

This post walks through the decision the way an engineering team should make it: split the workloads, size memory from the model, calculate cost per hour actually used, and only then look at price lists. There is a small Python model you can run with your own quotes, a serving config, and the pitfalls that tend to show up after the purchase order is signed.

Two workloads with opposite profiles

Most AI infrastructure discussions go wrong by treating "AI" as one workload. It is at least two.

Training and fine-tuning are bursty. You run a LoRA fine-tune for a day, evaluate for a week, maybe run another job next month. While a job runs it wants as much memory and interconnect bandwidth as you can give it, because gradients, optimizer state and activations all compete for VRAM, and multi-GPU jobs spend real time moving tensors between cards.

Inference is steady. It runs every working hour, often around the clock, and it is judged on two numbers: latency per request and cost per response. A single mid-range card can serve a quantized 8B model to a few dozen concurrent users comfortably. What matters is that it is always on and always close to the data.

Those profiles map naturally onto the buy-or-rent question:

Workload Utilization pattern Usual answer
Production inference High, predictable Buy, or rent long-term with fixed capacity
Fine-tuning experiments Low, bursty Rent by the hour or week
Full pre-training Very high, short-lived Almost never a mid-size company's job
Embeddings and batch jobs Medium, schedulable Run on inference hardware off-peak

If you buy one big configuration for everything, you pay for training-class interconnect that sits idle while serving requests, or you get a cheap inference box that cannot fit a fine-tuning job. Planning the two separately is the single biggest cost lever.

Size memory from the model, not from the spec sheet

VRAM is the constraint that decides everything else. If the model does not fit on one card, you have to split it with tensor or pipeline parallelism, and splitting across cards over PCIe costs throughput. So start from the model.

Weights are easy: parameters times bytes per parameter.

  • 8B model: about 16 GB at FP16, 8 GB at FP8, roughly 5 GB at 4-bit
  • 70B model: about 140 GB at FP16, 70 GB at FP8, roughly 40 GB at 4-bit

The part people underestimate is the KV cache, which grows with context length and the number of concurrent requests. For a transformer with grouped-query attention:

def kv_cache_bytes_per_token(layers, kv_heads, head_dim, bytes_per_value=2): # 2 = one tensor for keys, one for values return 2 * layers * kv_heads * head_dim * bytes_per_value def kv_cache_gb(layers, kv_heads, head_dim, context_len, concurrent, bytes_per_value=2): per_token = kv_cache_bytes_per_token(layers, kv_heads, head_dim, bytes_per_value) return per_token * context_len * concurrent / 1e9 # Llama-style 8B: 32 layers, 8 KV heads, head_dim 128 print(kv_cache_gb(32, 8, 128, context_len=8192, concurrent=16)) # ~17.2 GB # Llama-style 70B: 80 layers, 8 KV heads, head_dim 128 print(kv_cache_gb(80, 8, 128, context_len=8192, concurrent=16)) # ~42.9 GB 

So an 8B model at FP16 with 16 users each holding an 8K context needs around 16 GB for weights plus 17 GB for cache, plus a few GB of runtime overhead. That is a 48 GB card, not a 24 GB one. Quantize the weights to FP8 (8 GB) and store the cache in FP8 (about 9 GB) and it fits on a 24 GB card again. These trade-offs are worth making on paper before hardware arrives.

The 70B case is the classic trap. At FP8 the weights alone are 70 GB, so an 80 GB card leaves almost nothing for cache. In practice you either quantize to 4-bit, split across two cards, or move to a card with more memory. Each option has a different cost per response, which is the number you should compare.

The break-even model

The question "buy or rent" reduces to cost per hour of useful GPU time. Owned hardware has a fixed monthly cost whether you use it or not; rented hardware costs roughly the same per hour regardless of how much you use it.

HOURS_PER_MONTH = 730 def owned_cost_per_used_hour(capex, months_of_life, opex_per_month, utilization): """capex: server price; opex: power, rack space, support, admin time.""" monthly = capex / months_of_life + opex_per_month return monthly / (HOURS_PER_MONTH * utilization) def breakeven_utilization(capex, months_of_life, opex_per_month, rent_per_hour): monthly = capex / months_of_life + opex_per_month return monthly / (HOURS_PER_MONTH * rent_per_hour) # Illustrative inputs only. Replace with your own quotes. capex = 60_000 # server with GPUs, in your currency units life = 36 # months before it is obsolete for your workload opex = 900 # power + colocation + share of an engineer rent = 3.5 # comparable rented server, per hour print(f"break-even utilization: {breakeven_utilization(capex, life, opex, rent):.0%}") for u in (0.15, 0.4, 0.8): print(u, round(owned_cost_per_used_hour(capex, life, opex, u), 2)) 

With these placeholder inputs the break-even sits around 100%, which means renting wins at any utilization. Change the life to 48 months and rent to 5.0 and it drops to roughly 60%. The exact numbers are not the point; the shape is. Three things move the result much more than the GPU price:

  1. Utilization. A production inference server running 18 hours a day sits at 75%. A fine-tuning box used for two jobs a month sits under 10%. The same hardware is a good purchase for the first and a poor one for the second.
  2. Useful life. Teams plan for five years and replace in three because a new model family needs more memory than the card has. Be pessimistic here.
  3. Opex you forget. Power at full load, cooling, rack space, spare parts, the engineer who patches drivers at night. If you do not have a server room built for this density, colocation fees go into opex too.

Then convert to the metric the business cares about: cost per response. Measure tokens per second at your real concurrency on a candidate card, divide hourly cost by responses per hour, and compare cards on that number. A cheaper card that serves half the throughput is often the more expensive option.

Serving configuration that respects the budget

Whatever you buy or rent, the serving layer decides how much of that VRAM turns into throughput. A typical vLLM launch for an 8B model on one 48 GB card:

vllm serve meta-llama/Llama-3.1-8B-Instruct \ --gpu-memory-utilization 0.90 \ --max-model-len 8192 \ --max-num-seqs 32 \ --kv-cache-dtype fp8 \ --port 8000 

And for a 70B model split across two cards:

vllm serve <your-70b-awq-checkpoint> \ --tensor-parallel-size 2 \ --quantization awq \ --max-model-len 8192 \ --max-num-seqs 16 \ --gpu-memory-utilization 0.92 

Notes from running this in practice:

  • --max-model-len is a memory decision, not a feature flag. Setting it to the model's maximum (often 128K) reserves capacity for contexts your users never send. Set it to what your prompts actually need, measured from logs.
  • --max-num-seqs caps concurrency. Too high and requests queue inside the engine with poor latency; too low and the card idles. Load test with your real prompt length distribution.
  • Tensor parallelism over PCIe works, but expect noticeably lower scaling than on cards linked with a high-bandwidth bridge. If a model fits on one larger card, that is usually faster than two smaller ones.
  • Pin your stack. Write down the driver version, CUDA version, PyTorch version and serving engine version that passed load tests, and build images from that. A driver update that silently changes behaviour is a common cause of the "it was fast last week" ticket.

Pitfalls that appear after delivery

Power and cooling. A 4-GPU inference server can draw 2 to 3 kW under load; an 8-GPU training-class machine can pass 10 kW. A typical office server room circuit will not carry that, and air conditioning sized for a few file servers will not remove the heat. Check this before ordering, not when the breaker trips.

Noise and physical space. GPU servers are loud. "We will put it in the office" ends the first day.

Lead times. High-end cards can have delivery times measured in months. If the project has a deadline, rent for the first phase and buy once the workload is proven.

Driver and environment work. Getting CUDA, cuDNN, PyTorch and the serving engine to agree is a day of work on a good day. Budget for it or buy hardware delivered with a tested environment.

Data lifecycle. For owned hardware, plan disk sanitization at end of life. For rented hardware, get the provider to confirm in writing that storage is wiped on release, and agree how long you have to copy data out before shutdown.

Data residency. In banking, healthcare and the public sector, regulations often forbid sending personal data abroad. That rules out many hosted APIs regardless of price, and it is often the actual reason on-prem or in-country hosting gets approved. Check this first, since it can make the cost comparison irrelevant.

A sensible default architecture

For most mid-size companies the pattern that holds up looks like this:

  1. Inference on dedicated capacity you control, owned or rented long-term, sized from the KV cache math above, inside the jurisdiction your data must stay in.
  2. Fine-tuning on rented capacity by the hour or week, spun up when there is a measured quality gap between a generic model and your domain, and released when the job is done.
  3. Start with an open model as-is. Fine-tune only after you have an evaluation set that shows the generic model failing on your tasks. Many projects never need the second step.
  4. Review utilization quarterly. If the rented inference server is busy 80% of the time and the workload is stable, that is the moment to price a purchase. If an owned box sits at 20%, consolidate.

Buying hardware is a bet on a workload staying stable for three years. Renting is paying a premium for the right to change your mind. For production inference with steady traffic the bet often pays; for experiments it rarely does.

If you are working through this decision and want a second opinion on sizing for a specific model, request volume and data residency requirement, here is how we plan AI server infrastructure with training and inference sized separately, including hosting in a Kazakhstan data centre.