How Much vRAM Does Your AI Model Need: A Sizing Guide

Why almost everyone under-sizes

The question always arrives the same way: "I want to serve a 70B model, will a 48 GB card do?" The honest answer is "it depends", but not on anything mysterious: it depends on how many bits you store the weights in and how much context you want to serve at once. Those are two numbers you can work out in thirty seconds, and most failed sizings come from having worked out only the first.

Memory used by a model at inference has three parts: the weights, the KV cache and the runtime overhead. The first is fixed and easy. The second grows with context length and with concurrent requests, and it is the one that surprises people. The third is small but not zero.

The easy part: the weights

Weights take up parameter count × bytes per parameter. That is all. An 8-billion-parameter model at 16-bit precision takes 8 billion × 2 bytes = 16 GB. The table writes itself:

Weight memory by precision. Weights only, no KV cache or overhead.
Model	FP16 / BF16	FP8 / INT8	INT4
8B	16 GB	8 GB	4 GB
13B	26 GB	13 GB	6.5 GB
32B	64 GB	32 GB	16 GB
70B	140 GB	70 GB	35 GB

The first cut is already visible: a 70B at FP16 does not fit in any 96 GB card. Either you quantise it or you split it across several GPUs.

Quantisation changes the equation

Quantising means storing each weight in fewer bits. Going from FP16 to INT4 divides memory by four and, for most text generation, the quality loss is modest with a modern method. It is not free: quality drops more on long reasoning and on code than on conversation, and it is worth measuring with your own evaluation set before accepting it.

Rule of thumb: if the model fits at FP8 within your memory budget, stay at FP8. Drop to INT4 only when it is the only way to make it fit, or when you need the free space to serve more concurrent requests.

The KV cache, the forgotten cost

While generating, the model stores the attention keys and values of every token it has already processed, so it does not recompute them. That is the KV cache, and it grows linearly with context length and with concurrent requests. The formula is:

KV cache = 2 × layers × kv_heads × head_dim × context × bytes × requests

The 2 is the key and the value. For a Llama 3.1 70B — 80 layers, 8 KV heads under grouped-query attention, head dimension 128 — at FP16 with a single request:

8,192 tokens of context: 2 × 80 × 8 × 128 × 8,192 × 2 = 2.5 GiB
32,768 tokens: 10 GiB

That looks small until you serve several users. Ten concurrent requests at 32k context on that same model is 100 GiB of KV cache alone, more than the whole card. Which is why sizing a service with real concurrency looks nothing like a lab test with a single conversation.

There are two legitimate shortcuts: quantise the KV cache to 8 bits as well, which halves it, and cap the maximum context per request at what you actually use. A limit of 8k instead of 128k saves more memory than any other decision on this list.

What the sum leaves out

On top of the total you have intermediate activations, runtime buffers and allocator fragmentation. With vLLM or TensorRT-LLM, reserving an extra 10-15% over the sum of weights and KV cache is a reasonable estimate. If your calculation lands exactly on the card's limit, it does not fit.

What fits in 24, 48 and 96 GB

Putting the three parts together, at 8k context, one request and 15% overhead, this is what fits in each GPU instance size:

Weights + KV cache at 8k + 15% overhead. More concurrency or more context pushes this up.
Instance	Comfortable	Tight
24 GB	8B at FP8 (10 GB) · 13B at INT4 (9 GB)	8B at FP16 (20 GB) · 13B at FP8 (16 GB) · 32B at INT4 (20 GB)
48 GB	13B at FP16 (31 GB) · 32B at FP8 (39 GB)	70B at INT4 (43 GB)
96 GB	32B at FP16 (75 GB) · 70B at INT4 (43 GB)	70B at FP8 (83 GB)

The "tight" column is exactly where deployments break: the model loads, the first requests answer, and on the third concurrent user the server runs out of memory. If your case sits there, size up.

When you need more than one card

When the model will not fit in a single GPU even quantised, it has to be split. Tensor parallelism divides each layer across several cards and demands fast interconnect between them; pipeline parallelism tolerates less bandwidth but leaves GPUs waiting. Either way you are no longer talking about an instance but about a dedicated GPU server with two, four or eight cards.

The other reason to move up is concurrency. A small model serving a hundred users at once consumes more KV cache memory than a large model serving one, and there the bottleneck is not the weights but how many requests you want to keep alive.

How to measure it yourself

Back-of-the-envelope is for picking a size; to confirm, measure it. Load the model with your runtime, send the load you expect in production and watch actual consumption:

nvidia-smi --query-gpu=memory.used,memory.total --format=csv -l 1

Your model's exact parameters — layers, KV heads, head dimension — are in the config.json of its repository, so you can substitute them into the formula above instead of trusting a generic table. The Hugging Face model size estimator does the weights part automatically.

And if the card is shared, make sure the split is in hardware and not by turns: in a MIG partition the memory assigned to you is yours and nobody can encroach on it, whereas with a software split your neighbour's consumption does reach you. That difference is what makes a 24 GB instance genuinely 24 GB.

If you are still deciding whether to host the model or keep paying per token, the other half of the sum is in Local LLM vs API, where we compare cost, latency and privacy. This guide tells you how much GPU you need; that one tells you whether it is worth owning.

Frequently asked questions

How much vRAM does Llama 3.1 70B need? expand_more

It depends on precision. At FP16 the weights alone take 140 GB, so it will not fit in a single 96 GB card. At FP8 it is 70 GB and at INT4 around 35 GB, plus the KV cache — 2.5 GiB at 8,192 tokens of context and one request — and 10-15% of runtime overhead. In practice a 70B quantised to INT4 runs comfortably on a 96 GB instance and fits tightly on a 48 GB one.

What is the KV cache and why does it matter so much? expand_more

It is the memory where the model stores attention keys and values for already-processed tokens so it does not recompute them at every step. It grows linearly with context length and with concurrent requests, so ten users at 32,000 tokens of context can consume more memory than the model weights themselves. It is the item that wrecks the most sizings.

Do I lose much quality quantising to INT4? expand_more

Less than many people fear, but it is not free. With modern methods the degradation on conversational text generation is usually small, while on long reasoning and on code it shows more. The recommendation is to stay at FP8 if your memory budget allows and drop to INT4 only when it is the only way to make the model fit or to free space for more concurrency.

Is 24 GB of vRAM enough for me? expand_more

For an 8B model at FP8, a quantised 13B, or embedding generation and RAG, comfortably yes. For an 8B at FP16 or a 32B at INT4 it fits, but with no headroom for concurrency. If you will serve several users at once with long contexts, count the KV cache before deciding: that is what separates a 24 GB instance from a 48 GB one.

How do I calculate the KV cache for my specific model? expand_more

With the formula 2 × layers × kv_heads × head_dim × context × bytes × requests. The first three values are in the model repository config.json: num_hidden_layers, num_key_value_heads, and head dimension, usually hidden_size divided by num_attention_heads. Bytes is 2 at FP16 and 1 if you quantise the KV cache itself to 8 bits.

AI GPU vRAM Inference LLM MIG