Inference is not training: sizing for serving, not for the run

A team finishes a fine-tune, deploys it on the same instance type they trained on, and is surprised twice: the GPU is mostly idle, and tail latency is still bad. Both surprises have the same cause. Training and serving are different workloads with almost nothing in common except the hardware.

Training is one long job you want to finish as fast as possible. Throughput is everything, latency is meaningless, and the machine runs flat out for hours. Serving is thousands of short jobs arriving unpredictably, where the number that matters is what the slowest 1% of users experience, and the machine is idle most of the time and saturated occasionally.

Here is how to size for the second one.

1. Memory is the constraint, not compute

For training you size compute and fit memory around it. For serving it is the reverse: memory decides whether the thing runs at all, and how many concurrent requests you can hold.

Two consumers, and people budget for one:

Weights. Parameters × bytes per parameter. A 7B model at fp16 is roughly 14 GB. At int8, 7 GB. At int4, around 3.5 GB.

KV cache. This is the one that gets missed. Every concurrent request holds a cache proportional to its context length, and it grows as the response is generated:

kv_bytes ≈ 2 × layers × kv_heads × head_dim × seq_len × bytes_per_element

For a 7B-class model that is often 0.4–0.5 MB per token. At 4,000 tokens of context that is around 2 GB per concurrent request. Thirty-two concurrent requests want 64 GB of KV cache alone — four times the weights.

The practical consequence: your maximum concurrency is set by leftover memory after weights, and your context limit and your concurrency limit trade directly against each other. Decide which one your product needs before choosing a card.

2. Batching is the whole game

A single request uses a tiny fraction of an accelerator's compute. Generation is memory-bandwidth bound — you read the entire weight matrix to produce one token — so serving one request at a time wastes almost all of the hardware.

Continuous batching (as in vLLM, TGI and similar) fixes this by adding and removing requests from the running batch every step, rather than waiting to assemble a fixed batch. Throughput improvements of 5–20× over naive per-request serving are normal.

The tradeoff is real but small: a request may wait for the current step to finish. Tens of milliseconds against several-fold better utilisation. Take it.

What to avoid is the naive fixed-window batcher — "wait 200ms, batch what arrived" — which adds a fixed latency floor to every request including the ones that arrive into an empty queue.

3. Quantisation, honestly

Int8 typically costs very little quality on most tasks and halves the weight memory. Int4 halves it again and the quality cost is task-dependent — often invisible on summarisation and classification, sometimes clearly visible on reasoning and code.

What quantisation buys is not primarily speed. It is headroom: smaller weights leave more memory for KV cache, which raises concurrency, which raises throughput per instance. The speedup is a side effect of better batching.

Evaluate it on your own task with your own prompts. Published benchmark deltas do not transfer.

4. Autoscaling, and why cold start is different here

Scaling a web service is cheap: start a container, it is ready. Scaling a model server is not. Cold start is dominated by loading weights — reading tens of gigabytes from storage into GPU memory, which is anywhere from thirty seconds to several minutes.

That changes the design:

• Scale on queue depth or time-to-first-token, not CPU. CPU tells you nothing about a GPU server.
• Scale up early and down slowly. A conservative scale-down beats paying the cold-start penalty on the next burst.
• Keep one warm replica if any traffic is user-facing. Scale-to-zero is for batch endpoints and demos.
• Cache weights on local NVMe, not object storage. It is the difference between a 40-second start and a four-minute one.

5. When you should not use a GPU at all

Worth saying plainly, because it is a common and expensive mistake.

An embedding model, a classifier, a reranker or a model under about 1B parameters, serving low or intermittent QPS, is very often cheaper and simpler on CPU. Modern CPU inference runtimes are good, the memory is cheap, autoscaling behaves normally, and there is no cold-start cliff.

The crossover is roughly where sustained utilisation would keep a GPU meaningfully busy. Below that, you are renting an accelerator to be idle.

A sizing method that works

Five steps, in order, before choosing anything:

1. Measure your real prompt and response length distribution. Use p95, not the average — the average is a fiction created by short prompts.
2. Compute weight memory at your chosen precision.
3. Compute KV cache at p95 context × target concurrency.
4. Add 15% overhead. Pick the smallest card those three fit in.
5. Load-test at target concurrency and read p50 and p99 time-to-first-token and tokens/second. Adjust concurrency until p99 meets your requirement, then work out how many replicas your peak QPS needs.

This takes an afternoon and it replaces a quarter of guessing. Most importantly it produces a number you can defend when someone asks why the inference bill looks the way it does.

Antyxsoft GPU instances are launching soon, with the vCPU, memory and NVMe around the accelerator configurable rather than fixed — which matters more for serving than for training. Join the waitlist.