LLM Inference Explained: How It Works, Bottlenecks, and Optimization

Every time you send a prompt to a chatbot and receive an answer, a large language model (LLM) performs a process called LLM inference. The model uses what it learned during training to understand your input and generate a response. It does not learn new model weights during this process. Instead, it runs its trained parameters to predict the next token, again and again, until it finishes the response.

Inference may look simple from the user’s side, but it requires careful coordination between model weights, GPU memory, processors, system memory, and serving software. A powerful GPU does not automatically guarantee fast responses. The system can still slow down because of memory bandwidth, CPU overhead, data transfers over PCIe, or inefficient request scheduling.

This guide explains what LLM inference means, how it works, where performance bottlenecks appear, and which techniques engineers use to make inference faster and more efficient.

What Is LLM Inference?

LLM inference is the process of using a trained large language model to generate predictions or responses from new input data. When you enter a question, the model converts the text into tokens, processes those tokens through its neural network, and predicts what token should come next.

For example, imagine asking an AI assistant, “Why do GPUs matter for artificial intelligence?” The model does not retrieve a complete answer from a fixed database. It processes your prompt and calculates a probability distribution over possible next tokens. It selects a token according to its decoding strategy, adds that token to the context, and repeats the process.

The model’s learned weights remain fixed during ordinary inference. This is the main difference between inference and training.

LLM inference vs. LLM training

Aspect	Training	Inference
Main purpose	Learn patterns from data	Generate predictions or responses
Model weights	Updated through optimization	Normally remain unchanged
Typical computation	Forward pass and backpropagation	Forward pass
Memory needs	Weights, activations, gradients, optimizer states	Weights, activations, and KV cache
Performance goals	Training speed and stability	Low latency, high throughput, and low cost

Training a model can require enormous computing resources, but inference also becomes expensive when millions of users send requests. A model may be trained once and then serve requests continuously. That is why inference efficiency is a major part of modern AI infrastructure.

How Does LLM Inference Work?

Most widely used generative language models follow an autoregressive generation process. They predict one token at a time, using the prompt and previously generated tokens as context. The process has two important stages: prefill and decoding.

1. Tokenization: Converting text into tokens

Before the model can process a prompt, a tokenizer converts the text into numerical token IDs. A token may represent a complete word, part of a word, punctuation, or another text unit.

For example, the sentence “Inference makes AI useful” is converted into a sequence of token IDs. The exact IDs depend on the tokenizer used by the model. The model processes these numbers rather than ordinary text.

2. Prefill: Processing the input prompt

During prefill, the model processes the prompt’s input tokens through its transformer layers. It calculates the attention and other neural network operations needed to build the internal context for generation. It also creates the initial key-value (KV) cache.

Because the full input prompt is already available, the model can process many prompt tokens in parallel. This makes prefill relatively compute-intensive, particularly when users submit long prompts or large documents.

Prefill strongly affects time to first token (TTFT), which measures how long a user waits before seeing the first generated token. Long context windows, large input batches, and prompt processing overhead can increase this time.

3. Decoding: Generating the response

After prefill, the model begins decoding. It predicts the next token, adds it to the sequence, and performs another forward pass to predict the following token. It repeats this process until it reaches a stopping condition or the output limit.

Unlike prefill, decoding is sequential for each individual request. The next token depends on the tokens that came before it. This makes it difficult to parallelize the generation of one response in the same way as prompt processing.

At low batch sizes, decoding often becomes memory-bandwidth-bound. The GPU must access model weights and the relevant KV-cache data while producing each token. The arithmetic units may have unused capacity because the workload cannot feed them with data quickly enough.

4. KV cache: Reusing previous attention calculations

The key-value cache stores attention-related information from tokens that the model has already processed. Without this cache, the model would need to recompute key and value representations for earlier tokens at every generation step.

Reusing the cache avoids much of that repeated work and makes autoregressive generation practical. However, the cache consumes GPU memory, and its size grows as the sequence becomes longer and as more requests run concurrently.

This creates an important trade-off: a larger KV cache can support more context and requests, but it also increases memory pressure. Inference engines use cache management techniques to allocate and reuse this memory efficiently.

Prefill vs. Decode: Why the Difference Matters

Feature	Prefill	Decode
Input	Processes the available prompt	Generates one new token per sequence step
Parallelism	High across prompt tokens	Limited within one sequence
Common bottleneck	Compute and memory capacity	Memory bandwidth and KV-cache access
Main latency metric	Time to first token	Time per output token
Optimization focus	Efficient prompt processing	Weight reuse, batching, and cache management

These stages have different hardware requirements. A configuration that improves prompt processing may not improve token generation. For example, a large prefill batch can use GPU compute efficiently, but it may delay decode requests that are waiting to produce their next tokens.

Some serving systems therefore use chunked prefill to divide long prompts into smaller pieces and schedule them alongside decoding work. Larger deployments can also separate prefill and decode onto different GPU groups, allowing each stage to scale independently.

What Are the Main LLM Inference Bottlenecks?

A slow model does not always mean that the GPU is too weak. The actual bottleneck can appear in several parts of the system. Engineers need to identify the limiting resource before choosing an optimization.

1. GPU memory bandwidth

GPU memory bandwidth determines how quickly data can move between high-bandwidth memory (HBM) and the GPU’s processing units. During decoding, the model repeatedly accesses its weights and KV cache. If the GPU cannot supply data quickly enough, its compute units may remain underused.

This explains why a GPU with impressive theoretical computing power may not reach its advertised peak performance during small-batch LLM generation. The workload may be waiting for data rather than performing arithmetic.

2. CPU bottlenecks

The CPU handles important tasks around inference. These can include tokenization, request validation, scheduling, sampling, data preparation, and communication with the GPU. If the CPU cannot prepare and submit work quickly enough, the GPU may wait between operations.

CPU bottlenecks become more visible when an application handles many small requests, uses inefficient data pipelines, or performs too much work in its request-handling process. A slow tokenizer or a single overloaded scheduling thread can reduce overall throughput even when the GPU has spare capacity.

To investigate this problem, monitor CPU utilization per core, request queue time, tokenization time, and the time between GPU operations. Total CPU utilization alone can hide a bottleneck in one busy thread.

3. PCIe bandwidth and data transfer

PCI Express (PCIe) connects the CPU and system memory to a discrete GPU. It carries data such as model weights, input tensors, outputs, and other information between the host and accelerator.

PCIe bandwidth can become a serious LLM inference bottleneck when the system frequently transfers large amounts of data between CPU memory and GPU memory. This can happen when the model does not fit entirely in VRAM and the system offloads some weights to RAM, when the application repeatedly copies tensors, or when a multi-GPU setup uses a slow communication path.

However, PCIe is not automatically the cause of low GPU utilization. If the model and its active KV cache fit in GPU memory and the runtime keeps the necessary data on the device, the GPU may spend most of its time limited by its own memory bandwidth or by the sequential nature of decoding. In that situation, upgrading the PCIe connection may produce little improvement.

For diagnosis, check host-to-device and device-to-host transfer volume, transfer duration, PCIe link generation and width, and whether the model is using CPU offloading. Compare those measurements with GPU activity rather than assuming that a slow response means PCIe is saturated.

4. Low GPU utilization

Low GPU utilization means the GPU is not continuously occupied with work. It can happen because the workload is too small, the batch size is low, the CPU cannot submit work quickly enough, data transfers are slow, or the inference engine schedules requests inefficiently.

There is an important distinction between low compute utilization and poor performance. During memory-bound decoding, a GPU can show modest compute utilization while still operating close to the limit imposed by memory bandwidth. Therefore, a low utilization percentage does not prove that the GPU is idle or that it needs replacing.

Look at GPU compute activity, memory bandwidth, VRAM use, power draw, CPU activity, and tokens per second together. Use a profiler when basic monitoring cannot reveal where time is spent.

5. GPU memory capacity and KV-cache pressure

Model weights occupy a large part of GPU memory, but they are not the only requirement. The runtime also needs memory for the KV cache, temporary activations, computation buffers, and other runtime data.

Long prompts, long responses, and high concurrency increase KV-cache requirements. When memory becomes scarce, the serving system may reduce the number of active requests, evict cached data, or move work to slower memory. Each choice can affect latency and throughput.

6. Inefficient batching and request scheduling

When a GPU handles only one short request at a time, it may not have enough parallel work to use its resources efficiently. Batching combines multiple requests so the system can process more tokens together and reuse model weights more effectively.

Static batching waits for a fixed group of requests before processing them. Continuous batching is more flexible: the serving engine can add new requests as other sequences finish. This keeps the GPU supplied with work and can improve overall throughput.

How to Diagnose Low GPU Utilization During LLM Inference

Before changing hardware or rewriting the application, measure the system. The following checks help narrow down the cause.

1. Check GPU activity and VRAM: Use tools such as NVIDIA System Management Interface (nvidia-smi) to inspect utilization, memory use, power, temperature, and running processes.
2. Measure CPU activity: Check individual CPU cores, tokenization time, scheduling overhead, and whether one thread is overloaded.
3. Measure PCIe transfers: Find out whether the runtime moves weights or tensors between system RAM and VRAM repeatedly.
4. Separate prefill and decode: Measure time to first token and time per output token independently. They point to different performance problems.
5. Test different batch sizes: Increase concurrency gradually and observe throughput and latency. Stop when the additional requests cause unacceptable delays or memory pressure.
6. Profile the runtime: Use GPU profiling tools to identify kernel gaps, memory stalls, synchronization, and data-transfer overhead.

A useful diagnostic principle is to change one variable at a time. If increasing batch size improves aggregate throughput but makes individual responses slower, the system is trading latency for throughput. If moving the model fully into VRAM removes long transfer delays, CPU offloading was likely part of the problem.

LLM Inference Optimization Techniques

LLM inference optimization combines model-level, GPU-level, and serving-level improvements. The best approach depends on the model size, hardware, context length, and workload.

1. Quantization

Quantization represents model weights with fewer bits. For example, a model may use 16-bit floating-point weights or a lower-bit format such as 8-bit or 4-bit weights.

Lower precision can reduce VRAM usage and memory traffic. This may allow a model to fit on a smaller GPU or leave more memory available for larger batches and KV caches. Quantization can also improve speed when the hardware and inference engine support efficient kernels for the selected format.

However, quantization is not a guaranteed speed boost. Some formats may require extra processing, and aggressive quantization can reduce output quality. Test the model’s accuracy and real generation speed on the target hardware.

2. Continuous batching

Continuous batching dynamically combines requests that arrive at different times. Instead of waiting for every request in a fixed batch to finish, the engine can add new sequences as earlier ones complete.

This approach improves GPU utilization and overall throughput under concurrent workloads. It is especially useful for API services and chat applications that receive a changing stream of user requests.

3. PagedAttention and efficient KV-cache management

PagedAttention manages KV-cache memory in smaller blocks rather than requiring one large continuous memory allocation for every request. This reduces memory fragmentation and helps an inference server share GPU memory across active sequences more efficiently.

The vLLM serving engine uses PagedAttention as a central part of its memory management approach. Efficient cache allocation can support more concurrent requests without requiring the system to reserve excessive memory in advance.

4. FlashAttention and optimized attention kernels

Attention operations can move large amounts of data between memory and processing units. FlashAttention uses a more memory-efficient approach to compute exact attention, reducing unnecessary memory movement and avoiding the need to store some large intermediate matrices in the usual way.

Its benefits depend on sequence length, hardware, model architecture, and the implementation used by the inference engine.

5. Speculative decoding

Speculative decoding uses a smaller, faster draft model to propose several tokens. A larger target model then checks those proposed tokens in parallel and accepts the valid sequence according to the method’s verification rules.

When the draft model makes useful predictions, the system can generate multiple accepted tokens with fewer sequential target-model steps. The technique can improve generation speed, although the result depends on how well the draft model predicts the target model and on the overhead of verification.

6. Prefix caching

Many applications repeatedly send the same system instructions, templates, or document prefixes. Prefix caching allows the inference engine to reuse the KV cache for matching prompt sections instead of processing the same tokens from scratch.

This can reduce prefill work and improve time to first token for workloads with repeated context. It is particularly useful for applications that use long system prompts or shared agent instructions.

7. Chunked prefill and disaggregated serving

Chunked prefill divides long prompt processing into smaller pieces. An inference engine can schedule these chunks alongside active decode requests, helping prevent one long prompt from delaying token generation for other users.

At larger scale, disaggregated serving places prefill and decode on separate workers or GPU groups. Prefill workers focus on processing prompts, while decode workers generate tokens. This lets operators scale each stage according to its own workload, although moving KV-cache data between the groups introduces communication overhead that must be managed.

8. Use an optimized inference engine

An inference engine manages model loading, memory allocation, request scheduling, batching, and GPU execution. Different engines offer different strengths, hardware support, and deployment options.

• vLLM: Designed for high-throughput LLM serving, with features such as PagedAttention and continuous batching.
• Hugging Face Transformers: A widely used library for loading and running models, with several inference and optimization integrations.
• llama.cpp: A practical option for running supported models on CPUs and different GPU backends, including quantized models.
• TGI: Hugging Face’s Text Generation Inference project provides serving features for deploying language models.

Choose an engine based on model compatibility, hardware, workload, deployment needs, and measured performance rather than relying on a single benchmark.

How to Measure LLM Inference Performance

Engineers use several metrics because no single number describes the full user experience.

Time to first token (TTFT)

TTFT measures the time between submitting a request and receiving its first generated token. It is strongly affected by prompt processing, queueing, and scheduling.

Time per output token (TPOT)

TPOT measures the average time needed to generate each output token after the first token. It helps describe the speed of the decoding stage.

Tokens per second

Tokens per second measures generation speed. It can describe the speed of one request or aggregate throughput across many concurrent requests. Always state which one you are measuring.

Throughput

Throughput measures how much work the serving system completes over a period, often expressed as input or output tokens per second across all requests. Higher throughput can reduce cost per token, but aggressive batching may increase the waiting time for an individual user.

Latency and cost per request

End-to-end latency includes queueing, prompt processing, generation, and communication with the application. Cost per request or per million tokens helps operators understand whether a faster configuration is also economically practical.

LLM Inference on GPUs, CPUs, and NPUs

GPUs are widely used for large-scale LLM inference because they offer substantial parallel computing power and high memory bandwidth. They are well suited to the matrix operations that transformer models perform.

CPUs can also run LLMs, particularly smaller or quantized models. CPU inference may be useful when a compatible GPU is unavailable, when memory capacity matters more than speed, or when an application needs to run locally. However, large models often run much faster on suitable accelerators.

Neural processing units (NPUs) are specialized accelerators designed for AI workloads. Some smartphones and personal computers use NPUs to run supported models locally with lower power consumption. For example, phones such as the Samsung Galaxy S25 Ultra include dedicated AI processing hardware. Actual LLM performance depends on model size, supported operations, available memory, software, and whether the application can use the NPU efficiently.

Why LLM Inference Optimization Matters

Inference optimization affects more than response speed. It determines how many users a server can support, how much hardware an application needs, and how much each generated response costs.

• Better user experience: Lower latency makes chatbots and AI assistants feel more responsive.
• Lower infrastructure costs: Better memory use and higher throughput can reduce the hardware required for a given workload.
• More concurrent users: Efficient scheduling and cache management help a server handle more requests.
• Local AI deployment: Quantization and optimized runtimes make it possible to run more models on consumer hardware.
• Scalable AI applications: A well-designed serving system can handle changes in traffic without wasting resources.

Frequently Asked Questions About LLM Inference

What does LLM inference mean?

LLM inference means running a trained large language model on new input to generate a prediction or response. The model uses its existing weights rather than updating them through training.

What is inference in an LLM?

Inference is the process through which an LLM converts input tokens into output tokens. It usually includes a prefill stage that processes the prompt and a decode stage that generates the response one token at a time.

Why is LLM inference slower than expected?

Slow inference can result from limited GPU memory bandwidth, CPU overhead, PCIe transfers, long prompts, KV-cache pressure, low batch sizes, or inefficient scheduling. Measuring prefill and decode separately helps identify the actual cause.

Does low GPU utilization always mean a bottleneck?

No. Low compute utilization can occur during memory-bound decoding, when the GPU waits for data rather than arithmetic work. Check memory bandwidth, GPU activity, CPU usage, and tokens per second before deciding what to upgrade.

Does PCIe bandwidth affect LLM inference?

Yes, when the workload transfers substantial data between system RAM and GPU memory. CPU offloading and repeated tensor transfers can make PCIe bandwidth important. If the model and active data stay in VRAM, GPU memory bandwidth or compute may be the more relevant limit.

What is the best way to optimize LLM inference?

There is no single best method for every workload. Start by measuring TTFT, TPOT, throughput, GPU memory use, and CPU activity. Then test suitable techniques such as quantization, continuous batching, efficient KV-cache management, prefix caching, and optimized inference engines.

Conclusion

LLM inference is the process that turns a trained language model into a working AI application. It involves tokenizing the prompt, processing it during prefill, and generating the response through autoregressive decoding. The KV cache makes this process more efficient, but it also creates memory demands as context and concurrency grow.

Fast inference requires more than a powerful GPU. CPU scheduling, PCIe transfers, GPU memory bandwidth, batching, and serving software all influence performance. The most reliable approach is to measure the complete system, identify the real bottleneck, and apply the optimization that addresses it.

For further technical reading, see the vLLM optimization guide, NVIDIA’s inference optimization guide, and IBM’s overview of LLM inference.