The weights formula
A model's size is given in parameters: 7B means about 7 billion. Each parameter is stored as a number with a fixed number of bits. Multiply and you have the memory the weights need:
At 16 bits, two bytes, each billion parameters takes 2 GB. At 8 bits, 1 GB. At 4 bits, half a GB. Mistral 7B, which Mistral AI announced with 7.3 billion parameters, therefore needs about 15 GB at 16-bit, 7.3 GB at 8-bit and 4.1 GB as a Q4_K file.
All figures on this page are weights only. A running model needs more memory on top, covered below.
Memory by model size
Weights only, in GB of 109 bytes:
| Model | FP16 / BF16 | 8-bit | Q4_K | 4-bit |
|---|---|---|---|---|
| 1B | 2.0 GB | 1.0 GB | 0.6 GB | 0.5 GB |
| 3B | 6.0 GB | 3.0 GB | 1.7 GB | 1.5 GB |
| 7B | 14 GB | 7.0 GB | 3.9 GB | 3.5 GB |
| 8B | 16 GB | 8.0 GB | 4.5 GB | 4.0 GB |
| 13B | 26 GB | 13 GB | 7.3 GB | 6.5 GB |
| 14B | 28 GB | 14 GB | 7.9 GB | 7.0 GB |
| 32B | 64 GB | 32 GB | 18 GB | 16 GB |
| 70B | 140 GB | 70 GB | 39 GB | 35 GB |
Operating systems and GPU tools often report GiB, 230 bytes, which is 7.4 percent larger than a GB. A 14 GB file shows as about 13.0 GiB. The data storage converter converts between the two.
What fits in a given amount of memory
Turned around: the largest parameter count whose weights alone fit, without any room for the cache or the runtime. Use it as an upper bound, not a target.
| Memory | FP16 / BF16 | 8-bit | Q4_K |
|---|---|---|---|
| 8 GB | 4.0B | 8.0B | 14B |
| 12 GB | 6.0B | 12B | 21B |
| 16 GB | 8.0B | 16B | 28B |
| 24 GB | 12B | 24B | 43B |
| 48 GB | 24B | 48B | 85B |
The same memory holds a model about 3.6 times larger in Q4_K than in FP16, and twice as large at 8-bit.
What quantization means
The Hugging Face documentation describes it plainly: quantization lowers the memory needed to load and use a model by storing the weights at lower precision while trying to preserve as much accuracy as possible. Weights are typically stored in 32-bit floating point, with 16-bit formats (fp16 and bf16) increasingly popular, and some methods go further to integers such as int8 or int4. Some methods need a calibration step for better accuracy and extreme compression down to 1 or 2 bits; others quantize on the fly as the model loads.
The common formats you will meet, with the bit widths Hugging Face lists for them: bitsandbytes (4 and 8 bits), AWQ (4), GPTQ (2, 3, 4 and 8) and GGUF, the format used by llama.cpp (1 to 8). The llama.cpp project itself lists 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit and 8-bit integer quantization.
GGUF names decoded
GGUF is a binary file format that stores both the tensors and a standard set of metadata, built for quick loading. Its quantization types have names like Q4_K. The number is the bit width of the stored integers; the real cost per weight is a little higher because each block of weights also stores scales. The values Hugging Face lists:
| Type | Bits per weight | 7B weights |
|---|---|---|
| FP32 | 32 (full precision) | 28 GB |
| FP16 / BF16 | 16 (half precision) | 14 GB |
| 8-bit | 8 (int8) | 7.0 GB |
| Q6_K | 6.5625 (GGUF 6-bit) | 5.7 GB |
| Q5_K | 5.5 (GGUF 5-bit) | 4.8 GB |
| Q4_K | 4.5 (GGUF 4-bit) | 3.9 GB |
| 4-bit | 4 (int4, no overhead) | 3.5 GB |
| Q3_K | 3.4375 (GGUF 3-bit) | 3.0 GB |
| Q2_K | 2.625 (GGUF 2-bit) | 2.3 GB |
Fewer bits means a smaller model in memory, and quantization is a trade between size and accuracy. Which level is good enough depends on the model and the task, so compare a couple of levels on your own prompts.
Download size and disk space
A GGUF file stores the quantized tensors plus metadata, so its size on disk is close to the weights figure for its type: a 7B model in Q4_K is a file of about 3.9 GB, the same model at 16-bit about 14 GB. Model pages often offer the same model in many quantization levels, and each one is a separate download. Keeping three levels of a 13B model, at Q4_K, Q5_K and 8-bit, takes about 29 GB of disk.
Download speed follows the same arithmetic. At 100 megabits per second, about 12.5 megabytes per second, the 3.9 GB file takes about 5 minutes.
Memory the tables leave out
- The KV cache. While generating, a model keeps the key and value states of earlier tokens so it does not recompute them. Hugging Face notes that the default cache grows as generation progresses and can take a significant portion of memory for long contexts. A long document or a long chat costs memory on top of the weights. The token counter tells you how long your input is.
- The runtime. The program running the model, its buffers and the graphics driver take memory too.
- Your other apps. On a computer with shared memory, the operating system and open programs use part of the same pool.
The llama.cpp README adds two options that change the picture. It treats Apple silicon as a first-class target, optimized through Metal and other Apple frameworks. And it supports CPU and GPU hybrid inference, which partially accelerates models larger than the total VRAM by running part of the model on the processor, slower but possible.
A quick fit check
- Find the parameter count on the model card.
- Pick a format and multiply: billions × bits ÷ 8 = GB of weights.
- Compare with your GPU memory, or with system memory on a machine with shared memory.
- Leave room for the KV cache and the runtime; the longer your prompts, the more room.
- If it does not fit, choose a lower-bit file, a smaller model, or hybrid CPU and GPU inference.
Before downloading, check the license as well as the size; the guide open weights vs open source AI explains what the labels on model pages mean.
Frequently asked questions
How much VRAM do I need to run a 7B model?
For the weights alone: about 14 GB at 16-bit precision, 7 GB at 8-bit and roughly 4 GB with a 4-bit GGUF quantization such as Q4_K at 4.5 bits per weight. Add room for the KV cache, which grows with the length of the conversation, and for the runtime itself.
What does quantization do?
It stores the weights at lower precision. Hugging Face describes models as typically stored in 32-bit floating point, with 16-bit increasingly common, and quantization going further to integer formats such as 8 or 4 bits. Fewer bits per weight means a smaller file and less memory, while the method tries to keep as much accuracy as possible.
Why is Q4_K 4.5 bits and not 4?
Each block of weights also stores scale values needed to turn the small integers back into usable numbers. In Q4_K, super-blocks of 8 blocks of 32 weights carry 6-bit scales and minimums, which averages out to 4.5 bits per weight. Q6_K comes to 6.5625 and Q2_K to 2.625.
Can I run a model that is bigger than my GPU memory?
With some runtimes, yes, more slowly. The llama.cpp project lists CPU and GPU hybrid inference to partially accelerate models larger than the total VRAM: part of the model runs on the graphics card and the rest on the processor using system memory.
Is GB the same as GiB here?
No. This page uses GB as 10^9 bytes, because parameter counts are in billions. Tools and operating systems often show GiB, 2^30 bytes, which is about 7 percent larger, so 14 GB is about 13 GiB.
Sources
- Hugging Face Transformers docs: Quantization overview (lower precision to reduce memory; fp32 full precision, fp16 and bf16 half precision, int8 and int4; methods and their supported bits, including bitsandbytes 4/8, AWQ 4, GPTQ 2/3/4/8 and GGUF 1 to 8)
- Hugging Face Hub docs: GGUF (binary format with tensors and metadata, developed by the llama.cpp author; quantization types table with bits per weight for Q6_K, Q5_K, Q4_K, Q3_K, Q2_K and IQ types)
- llama.cpp on GitHub, README (1.5-bit to 8-bit integer quantization; Apple silicon via Metal; CPU+GPU hybrid inference for models larger than the total VRAM)
- Hugging Face Transformers docs: Cache strategies (the KV cache stores key and value states; the default cache grows as generation progresses; it can take a significant portion of memory for long contexts)
- Mistral AI, Announcing Mistral 7B, September 27, 2023 (7.3B parameters)