GGUF Quantization Explained: Q4_K_M vs Q5_K_M vs Q8

Open a model page on Hugging Face and you will often see a long list of files with names like Q4_K_M, Q5_K_M, and Q8_0. They all contain the same model, yet they differ in size, speed, and quality. GGUF quantization is the reason, and understanding it helps you pick the right file without trial and error.

This guide explains what quantization does, what the GGUF format is, and how the three most common choices compare. You will also learn how to decide between them and how to test the difference on your own computer. We avoid exact quality scores and speeds, because they depend on the model, the hardware, and the test.

What Is Quantization?

A language model stores billions of numbers called weights. In the original training format, each weight usually takes 16 bits of space. Quantization stores each weight with fewer bits, so the model file gets smaller and uses less memory.

A Simple Analogy

Think of a photo. A high-quality image keeps every shade of color, and a compressed copy keeps fewer shades but looks nearly the same to most eyes. Quantization works in a similar way. It rounds each weight to the closest value that a smaller number format can store. The model loses a little precision, and you gain a lot of space.

Why Quantization Helps Local Users

Quantization helps in two ways. First, a smaller model fits in less memory, so you can run larger models on the same hardware. Second, generation is often limited by how fast the processor reads the weights from memory. Fewer bytes to read can mean faster output, although the speed gain varies. Our article on LLM VRAM requirements shows how this changes the memory math.

What Is the GGUF Format?

GGUF is a file format for storing models that run in llama.cpp and tools built on it, including Ollama and LM Studio. It replaced an older format called GGML. A GGUF file holds the quantized weights, plus the tokenizer and metadata the runtime needs, in a single file.

That single-file design makes GGUF easy to share. You download one file, point your runtime at it, and run it. The llama.cpp project documents the format and includes a tool called llama-quantize that converts models to GGUF quantization levels.

How to Read GGUF Quantization Names

The names look like code, but each part has a meaning.

Q means the weights are quantized.
The number after Q is the approximate bit count per weight. Q4 is around 4 bits, Q5 around 5, and Q8 around 8.
K means the K-quant method, a newer scheme that groups weights into blocks and handles them with more care than older methods.
S, M, and L mean small, medium, and large. They describe a mix of precision across different parts of the model. In a medium mix, some sensitive tensors keep more bits than others.
_0 in Q8_0 marks a simpler, older block-based scheme.

Q4_K_M vs Q5_K_M vs Q8_0 Compared

These three are the names you will meet most often. The table uses approximate figures, so expect small differences from one model to the next.

Format	Approx. bits per weight	Size for an 8B model	Quality	Typical use
Q4_K_M	About 4.8	About 4.8 GB	Good for most tasks, with some loss	The common default for limited memory
Q5_K_M	About 5.7	About 5.7 GB	Closer to the original	A balance when you have a little more room
Q8_0	About 8.5	About 8.5 GB	Very close to the original	When quality matters more than size

These bit counts are rounded averages. Quantized formats store small scaling values with each block of weights, so the real average is slightly above the number in the name.

Q4_K_M: The Practical Default

Q4_K_M uses about a quarter of the memory of the 16-bit original. Many people use it as the standard choice because it keeps good quality while fitting large models on modest hardware. It tends to work well for chat, summaries, and general tasks. On tasks that need precision, such as math or exact code, the quality loss is more likely to show.

Q5_K_M: A Middle Step

Q5_K_M adds roughly one bit per weight. The file is about 18 percent larger than Q4_K_M, and the output stays closer to the original model. If you have a few spare gigabytes of memory, this is a sensible upgrade for coding and careful writing.

Q8_0: Near-Original Quality

Q8_0 stores about 8 bits per weight. Most people cannot tell its output from the full-precision model in everyday use. It needs about twice the memory of Q4_K_M, so it suits smaller models where the extra memory is affordable. For a 70B model, Q8_0 is out of reach for most home computers.

Other Quantization Levels You May See

You will find more options in a typical repository. A few are worth knowing.

Q6_K sits between Q5_K_M and Q8_0 and gives near-original quality at a smaller size than Q8_0.
Q3_K and Q2_K variants save even more memory but lose noticeable quality. They are a last resort for fitting a large model into a small space.
IQ formats, such as IQ4_XS, use an importance-based approach that tries to keep quality at low bit counts. Their speed on some hardware can differ from K-quants.
F16 and BF16 are the unquantized 16-bit formats. They are the quality reference, and they use the most memory.

Some model authors also publish quantizations made with extra calibration data, which can improve quality at the same size. These are separate from the file names above, so read the model card for details.

How Quantization Affects Quality and Speed

The effect of GGUF quantization depends heavily on the model. Large models usually survive heavy quantization better than small ones, because they have more redundancy. A 4-bit 32B model often beats a 16-bit 8B model, but that is not a rule, so test your own case.

Quality Loss Is Uneven

Quantization does not damage every task equally. Casual chat and summaries rarely show problems. Multi-step math, strict code generation, and tasks in less common languages often show more loss. If your task is sensitive, choose a higher-bit file for it.

Speed Is Not Always Faster

Smaller files usually generate faster when memory bandwidth is the limit. But the speed depends on the processor, the backend, and how well it supports a given format. On some hardware, a lower-bit format needs extra work to unpack, so the speed gain is smaller than the file-size drop. Do not assume that fewer bits always means more tokens per second. The article on how to run local LLMs faster shows how to measure this.

How to Choose a GGUF Quantization Level

Use this short process to pick a file.

Decide which model you want and check its memory needs, including context, using our memory guide.
Start with the highest quantization that fits comfortably in memory, with a margin for the context window.
If Q8_0 fits, use it for quality-sensitive work. If it does not, try Q6_K or Q5_K_M.
If Q5_K_M does not fit, use Q4_K_M.
Go below Q4 only when nothing else fits, and test the result carefully.
If you are choosing between a lower quantization of a bigger model and a higher quantization of a smaller one, run both on your own prompts.

How to Test the Difference Yourself

You can compare two GGUF quantization levels in a few minutes. In Ollama, many models offer several tags, such as a Q4_K_M tag and a Q8_0 tag. Pull both and run the same prompts through each.

ollama pull llama3.1:8b-instruct-q4_K_M
ollama pull llama3.1:8b-instruct-q8_0

ollama run llama3.1:8b-instruct-q4_K_M --verbose
ollama run llama3.1:8b-instruct-q8_0 --verbose

Tag names differ by model, so check the Tags tab on the model’s library page. The --verbose flag prints timing statistics after each answer, including the generation rate. Use a short list of prompts that match your real work, and compare both the answers and the rates.

If you use llama.cpp directly, the llama-perplexity tool measures how well a quantized model predicts a text sample, and lower perplexity means a result closer to the original. It is a useful sanity check, but it does not replace testing on your own tasks.

Common Mistakes

Choosing by file name alone. Two Q4_K_M files from different models can differ widely in quality.
Ignoring the context window. A file that fits at 2,048 tokens may not fit at 16,384.
Comparing different models and calling it a quantization test. Change only one thing at a time.
Using a very low quantization on a small model. Small models have less redundancy, so heavy compression hurts them more.
Trusting a single prompt. One answer tells you little. Use several.

Conclusion

GGUF quantization trades a little precision for a lot of memory, which is why it matters so much for home hardware. Q4_K_M is the practical default for most people, Q5_K_M is a good upgrade when you have spare memory, and Q8_0 gives results very close to the original at about twice the size of Q4_K_M. Lower formats exist for tight spots, and they cost more in quality.

Pick the highest level that fits your memory with room for context, then confirm with a short test on your own prompts. After that, you can look at runtimes in our comparison of Ollama, LM Studio, and llama.cpp.

GGUF Quantization Explained: Q4_K_M vs Q5_K_M vs Q8_0