Before you download a model, you need to answer one question: will it fit? LLM VRAM requirements are the main reason a model runs smoothly on one computer and crawls on another. If you guess wrong, you either waste a large download or end up with a model that is painfully slow.
This article explains where memory goes when you run a model locally. You will learn the difference between system RAM and GPU VRAM, how to estimate the size of a model’s weights, why the context window adds a second cost, and when a hybrid CPU and GPU setup makes sense. Every number here is an estimate with stated assumptions, because real use varies by model, runtime, and settings.
RAM, VRAM, and Unified Memory Explained
Three kinds of memory matter for local LLMs. Knowing which one your computer uses tells you which limit applies to you.
System RAM
System RAM is the main memory your CPU uses. It is usually larger and cheaper than GPU memory, so it can hold bigger models. The cost is speed: the CPU reads memory more slowly than a GPU does, so generation is slower.
GPU VRAM
VRAM is the memory on your graphics card. It is fast, and it sits right next to the GPU cores that do the math. Language model generation is largely limited by how quickly the processor can read the weights from memory, which is why VRAM speed matters so much. A card with more VRAM can hold bigger models, and a card with faster memory generates text more quickly.
Unified Memory on Apple Silicon
Apple Silicon Macs share one memory pool between the CPU and GPU. A Mac with 64 GB of unified memory can give a large part of that pool to the GPU, which makes big models practical on a laptop. The operating system still keeps some memory for itself, so you cannot use all of it for a model.
What Uses Memory When You Run an LLM
Four things take up space. Beginners often count only the first one.
Model Weights
The weights are the numbers the model learned during training. They are the biggest cost, and their size depends on two things: the parameter count and the number of bits stored per parameter. A model with 8 billion parameters stored in 16-bit format needs about 16 GB. The same model stored in a 4-bit format needs roughly 4 to 5 GB.
The Context Window and the KV Cache
When a model generates text, it stores intermediate results for every token it has already seen. This storage is called the KV cache, short for key-value cache. It grows with the length of your conversation, so a long prompt or a long document costs more memory than a short one.
The size depends on the model’s architecture. A rough formula for a model that stores the cache in 16-bit format is:
KV cache bytes = 2 x layers x KV heads x head size x context tokens x 2 bytesTake a hypothetical model with 32 layers, 8 KV heads, and a head size of 128. At an 8,192-token context, the cache is about 1 GiB. At 32,768 tokens, it is about 4 GiB. This example is not any specific model, so check the architecture of the one you plan to use.
Runtime Overhead
The runtime needs working memory for temporary calculations, buffers, and the program itself. Ollama, LM Studio, and llama.cpp each handle this differently. A safe habit is to reserve one to two extra gigabytes on top of weights and cache.
Everything Else on Your Computer
Your desktop, browser, and other programs also use memory. A GPU that drives your display gives up some VRAM to it. Close heavy programs before you test a model that barely fits.
How to Estimate LLM VRAM Requirements From Parameters
You can estimate the size of the weights with a simple rule:
weights in GB = parameters in billions x bits per weight / 8Quantized formats use slightly more than their nominal bit count because they store extra scaling values. The table below uses approximate average bits per weight: about 4.8 for Q4_K_M, 5.7 for Q5_K_M, and 8.5 for Q8_0. These are rounded estimates, and real files differ a little by model. Read our GGUF quantization guide for the meaning of these names.
| Model size | FP16 (16-bit) | Q8_0 | Q5_K_M | Q4_K_M |
|---|---|---|---|---|
| 7B to 8B | About 14 to 16 GB | About 7.4 to 8.5 GB | About 5 to 5.7 GB | About 4.2 to 4.8 GB |
| 14B | About 28 GB | About 15 GB | About 10 GB | About 8.4 GB |
| 32B | About 64 GB | About 34 GB | About 23 GB | About 19 GB |
| 70B | About 140 GB | About 74 GB | About 50 GB | About 42 GB |
These figures cover weights only. Add the KV cache and one to two gigabytes of overhead to get the total. As a reference point, the Ollama library lists a 4-bit Llama 3.3 70B download at about 43 GB, which matches the estimate for 70B at Q4_K_M.
Mixture-of-Experts Models Still Need Full Memory
A mixture-of-experts (MoE) model activates only a small share of its parameters for each token. A 30B model with 3B active parameters can generate quickly, but it must still hold all 30B parameters in memory. Use the total parameter count for memory planning and the active count when you think about speed.
CPU-Only, GPU-Only, and Hybrid Inference
You can run a model in three ways, and each has a different memory requirement.
GPU-Only Inference
The whole model sits in VRAM. This gives the fastest generation, but the model plus its cache must fit on the card. If it does not fit, the runtime either fails or switches to a hybrid setup.
CPU-Only Inference
The model runs from system RAM on the CPU. This works on computers without a capable GPU, and it lets you run models far larger than any consumer graphics card can hold. Expect slower generation. Small models in the 3B to 8B range are the most practical choice here.
Hybrid Inference
The runtime puts some layers on the GPU and keeps the rest in system RAM. This lets you run a model that is bigger than your VRAM. Speed falls as more layers move to the CPU, and the slowest part of the chain sets the pace. In Ollama, the ollama ps command shows what share of a loaded model runs on the GPU. In llama.cpp, the -ngl option sets how many layers go to the GPU.
Practical Memory Guidelines by Hardware Tier
The table below turns the LLM VRAM requirements estimates into starting points. It assumes 4-bit quantization and a context window of a few thousand tokens. Treat each row as a rough guide, and test before you commit.
| Memory available to the model | Models that usually fit |
|---|---|
| 4 to 6 GB | 1B to 4B models, or a 7B to 8B model with limited context |
| 8 GB | 7B to 8B models at 4-bit with a moderate context |
| 12 GB | Up to about 14B at 4-bit with a short context |
| 16 GB | 14B models with room for context |
| 24 GB | Around 27B to 32B at 4-bit, or a 30B MoE model |
| 48 GB | 32B at a higher-quality quantization, or a 70B at heavy compression with limited context |
| 64 GB or more | 70B at 4-bit with room for context |
The edge cases need attention. A model that fits only by a hair will often fail when you raise the context length, because the KV cache grows. Leave a margin of at least 10 to 15 percent.
How to Check Your Real Memory Use
Estimates get you to a shortlist. Measuring tells you the truth. Follow these steps.
- Load the model with the context length you plan to use. In Ollama, you can set it with the
num_ctxoption or theOLLAMA_CONTEXT_LENGTHenvironment variable. - Run a prompt that fills a good part of that context, not just a short hello.
- On an NVIDIA GPU, run
nvidia-smiin a second terminal and read the memory column. - On Ollama, run
ollama psand check the size and the processor split between CPU and GPU. - On a Mac, open Activity Monitor and watch the memory pressure graph.
- If memory is tight, reduce the context length first, then try a smaller quantization, then a smaller model.
Ways to Reduce Memory Use
When a model goes over your LLM VRAM requirements budget, you have several options, and they cost different things.
- Choose a lower-bit quantization. Q4_K_M uses about half the memory of Q8_0 at some cost in quality.
- Shorten the context window. Many tasks work fine with 4,096 tokens, and a shorter window shrinks the KV cache.
- Quantize the KV cache. Some runtimes can store the cache in 8-bit format, which saves memory at a small quality risk. Support depends on the runtime and settings.
- Pick a smaller model. A well-trained 8B model often beats a heavily compressed 32B model on your hardware.
- Offload layers to the CPU. This keeps a big model usable but makes it slower.
The article on how to run local LLMs faster covers these options in more detail, including how each one affects speed.
Limitations of These Estimates
The LLM VRAM requirements in this article are guides, not guarantees. Memory use changes with the model architecture, the quantization method, the context length, the batch size, and the runtime version. Some models use grouped-query attention, which makes the KV cache much smaller than older designs. Others add vision encoders that need extra memory.
For a specific model, trust the file size on the model page and your own measurements before any formula. Also remember that adding memory does not fix slow generation if the memory itself is slow.
Conclusion
Planning for LLM VRAM requirements comes down to a short checklist. Estimate the weights from the parameter count and the bits per weight. Add the KV cache for your context length. Add one to two gigabytes for the runtime. Compare the total with your VRAM, or your unified memory on a Mac, and keep a margin.
If the model does not fit, shorten the context, choose a lower-bit quantization, or pick a smaller model before you accept a slow hybrid setup. Once you know your limits, you can read about the best local LLMs for different tasks and choose with confidence.