How to Make Local LLMs Run Faster: Practical Optimization Techniques

A model that answers in two seconds feels like a tool. One that takes thirty feels like a chore. If you want to run local LLMs faster, the good news is that most slowdowns come from a handful of causes, and you can find them with a few simple checks.

This guide shows how to diagnose where your time goes, then walks through practical fixes: quantization, context length, GPU offload, runtime settings, generation settings, and memory monitoring. Each fix has a cost, so we explain the trade-offs. We do not give speed numbers, because your results depend on your hardware, model, and runtime.

Understand What Makes Local LLMs Slow

Generation happens in two phases, and they have different bottlenecks. Knowing which phase is slow tells you which fix to try.

Prompt Processing

First, the model reads your whole prompt. This phase is called prefill, and it is limited mostly by raw computing power. A long prompt, such as a pasted document, makes this phase slow. You notice it as a delay before the first word appears.

Token Generation

Next, the model writes its answer one token at a time. For each token it must read most of its weights from memory, so this phase is limited mostly by memory bandwidth. You notice it as the speed at which words stream onto the screen.

This split explains a common surprise. A faster processor helps prefill more than it helps generation, and faster memory helps generation more than prefill. When you read about a speedup, ask which phase it improves.

Measure Before You Change Anything

Do not tune blind. Take a baseline first, so you can tell whether a change helped you run local LLMs faster or only felt different.

1. Choose one prompt that is typical for your work, and keep it the same for every test.
2. Run it several times, because the first run includes the time to load the model.
3. Record the prompt processing rate and the generation rate in tokens per second.
4. Change one setting at a time and repeat the test.

In Ollama, add the verbose flag to see timing details after each response:

ollama run llama3.1:8b --verbose

The output includes the prompt evaluation rate and the evaluation rate for generation. In llama.cpp, the llama-bench tool runs repeatable tests for prompt processing and generation. LM Studio shows tokens per second in its chat view.

Check Where the Model Is Running

The biggest slowdown is often a model that has quietly fallen back to the CPU. Before you tune anything else, confirm that the model uses your GPU.

Check GPU Use in Ollama

Load a model and run this in a second terminal:

ollama ps

The processor column shows how the loaded model is split. A value such as 100% GPU means the whole model is on the graphics card. A split such as 40% CPU and 60% GPU means some layers sit in system RAM, which slows generation.

Check GPU Use on NVIDIA Hardware

Run nvidia-smi while the model generates. You should see memory in use and the GPU utilization above zero. If utilization stays near zero, the work is happening on the CPU.

Common Reasons the GPU Is Not Used

• The model plus its context does not fit in VRAM, so the runtime offloads layers to the CPU.
• The GPU drivers are missing, outdated, or not supported by your runtime build.
• You are running a CPU-only build of the software.
• Another program is holding VRAM.

If memory is the problem, read our guide to LLM VRAM requirements to see what fits.

Choose a Smaller Model or a Better Quantization

Because generation reads the weights for every token, a smaller file usually means faster output. You have two ways to shrink the file.

Lower the Quantization Level

Moving from Q8_0 to Q5_K_M or Q4_K_M reduces memory traffic and can raise generation speed. It also lowers quality a little. The article on GGUF quantization explains the trade-off. The gain varies by hardware, so measure it instead of assuming it.

Pick a Smaller or Sparser Model

An 8B model generates faster than a 32B model on the same machine. A mixture-of-experts model can also be faster than its total size suggests, because it uses only part of its parameters for each token. It still needs enough memory for all of them.

Pick the smallest model that does your task well. Running a larger model than you need is the most common way to waste time.

Right-Size the Context Window

The context window affects both memory and speed. A larger window uses more memory for the KV cache, and long prompts take longer to process. Many tasks do not need a huge window.

In Ollama, you can set the context length for a request through the num_ctx option:

ollama run llama3.1:8b
/set parameter num_ctx 4096

In llama.cpp, use the -c flag. If you do not need to read long documents, a window of 4,096 tokens is often enough, and it leaves more room for the model. If your context does not fit in VRAM, the runtime may push work to the CPU, so a smaller window can bring the whole job back onto the GPU.

Use GPU Acceleration Correctly

When the GPU can hold the model, make sure the runtime uses it fully.

Offload All Layers You Can

In llama.cpp, the -ngl option sets the number of layers on the GPU. A high value such as 99 asks for all layers. If the model does not fully fit, lower the number until it loads. Each layer moved back to the CPU reduces speed.

Turn On Flash Attention Where Supported

Flash attention is an optimized way to compute attention. It can reduce memory use and speed up long contexts on supported hardware. In llama.cpp, the -fa option controls it, and Ollama exposes the OLLAMA_FLASH_ATTENTION environment variable. Support depends on your GPU and version, so check the current documentation.

Quantize the KV Cache

Some runtimes can store the KV cache in 8-bit format instead of 16-bit. This halves the cache memory and can free enough space to keep the model on the GPU. In Ollama, set OLLAMA_KV_CACHE_TYPE to q8_0 after you enable flash attention. The cost is a small quality risk, so test it on your own tasks.

Tune the Runtime

A few runtime settings matter more than the rest when you want to run local LLMs faster.

• Keep the model loaded. Loading a large model takes time. In Ollama, the keep_alive setting or the OLLAMA_KEEP_ALIVE variable controls how long a model stays in memory after a request. A longer value avoids repeated reloads.
• Set CPU threads sensibly. For CPU inference, more threads help up to the number of physical cores. Using every logical thread on a hyper-threaded CPU can hurt. In llama.cpp, the -t option sets this.
• Avoid running many models at once. Two models in memory compete for the same space. Ollama has a limit on loaded models, and you can lower it if memory is tight.
• Adjust batch sizes for long prompts. In llama.cpp, the -b and -ub options control batch sizes, which can affect prompt processing speed and memory use.
• Update your software. Runtimes improve quickly, and a newer version may speed up your model or hardware. Read the release notes first, because updates can also change behavior.

Tune the Generation Settings

Settings that control the output can also affect how long you wait.

Limit the Output Length

A model that writes a long answer takes longer, so set a maximum number of output tokens when you only need a short one. In Ollama, this is the num_predict option. A clear prompt that asks for a brief answer also helps.

Watch for Reasoning Tokens

Reasoning models write a long chain of thought before the answer. That thinking counts as generated tokens, so it adds time. If a task does not need deep reasoning, use a standard instruct model or turn thinking off where the model allows it.

Understand Sampling Settings

Settings such as temperature and top-p change how the model picks words. They affect output style much more than speed, so do not expect them to make generation faster. Lower temperature gives more consistent answers, which is useful for code and extraction.

Advanced Options

These techniques can help in the right situation, but they add complexity.

Speculative Decoding

Speculative decoding uses a small draft model to guess several tokens ahead, and the large model checks the guesses in one step. When the guesses are right, generation speeds up. llama.cpp supports it, and the gain depends on how well the draft model matches the large one. Benefits vary by task, and short or creative outputs may see less.

Prompt Caching

If you reuse the same long prefix, such as a system prompt or a document, some runtimes can cache its processed state and skip the work next time. This cuts the delay before the first token in repeated workflows. Check how your runtime handles this before you rely on it.

Handle Multiple Users

If several requests arrive at once, the runtime may process them in parallel, which divides memory and speed among them. In Ollama, OLLAMA_NUM_PARALLEL controls this. For a personal setup, leaving parallelism low keeps single-request speed higher.

A Checklist to Run Local LLMs Faster

When a model feels slow, work through this list in order.

1. Confirm the model runs on the GPU with ollama ps or nvidia-smi.
2. Check that the model and context fit in memory with room to spare.
3. Shorten the context window if you do not need a long one.
4. Try a lower quantization or a smaller model.
5. Enable flash attention and a quantized KV cache if your setup supports them.
6. Keep the model loaded between requests.
7. Update the runtime and graphics drivers.
8. Re-measure after each change and keep only the changes that help.

Know the Trade-Offs

To run local LLMs faster without hurting your results, remember that every speedup costs something. Smaller quantization costs quality. Shorter context costs reach. KV cache compression costs a little accuracy. A smaller model costs capability. The best setup is not the fastest one but the fastest one that still does your task well.

Also remember that your hardware sets a ceiling. If your memory is slow or small, no setting will make it behave like a high-end card.

Conclusion

To run local LLMs faster, find the bottleneck first. Check that the model uses your GPU, that it fits in memory, and that your context window matches your task. Then try a lower quantization, a smaller model, flash attention, and a quantized KV cache, and measure each change.

Keep a baseline, change one thing at a time, and accept only the trade-offs you can live with. With those habits you can often get a good speedup without buying new hardware. For choosing the right tool to apply these settings, see our comparison of Ollama, LM Studio, and llama.cpp.