LLM Inference Explained: How It Works, Bottlenecks, and Optimization
Every time you send a prompt to a chatbot and receive an answer, a large language model (LLM) performs a process called LLM inference. The model uses what it learned during training to understand your input and generate a response. It does not learn new model weights during this process. Instead, it runs its trained parameters to predict the next token, again and again, until it finishes the response. Inference may look simple from the user’s side, but it requires careful coordination between model weights, GPU memory, processors, system memory, and serving software. A powerful GPU does not automatically guarantee fast responses. The system can still slow down because of memory bandwidth, CPU overhead, data transfers over PCIe, or inefficient request scheduling. This guide explains what LLM inference means, how it works, where performance bottlenecks appear, and which techniques engineers use to make inference faster and more efficient. What Is LLM Inference? LLM inference is the process of using a trained large language model to generate predictions or responses from new input data. When you enter a question, the model converts the text into tokens, processes those tokens through its neural network, and predicts what token should come next. For example, imagine asking an AI assistant, “Why do GPUs matter for artificial intelligence?” The model does not retrieve a complete answer from a fixed database. It processes your prompt and calculates a probability distribution over possible next tokens. It selects a token according to its decoding strategy, adds that token to the context, and repeats the process. The model’s learned weights remain fixed during ordinary inference. This is the main difference between inference and training. LLM inference vs. LLM training Aspect Training Inference Main purpose Learn patterns from data Generate predictions or responses Model weights Updated through optimization Normally remain unchanged Typical computation Forward pass and backpropagation Forward pass Memory needs Weights, activations, gradients, optimizer states Weights, activations, and KV cache Performance goals Training speed and stability Low latency, high throughput, and low cost Training a model can require enormous computing resources, but inference also becomes expensive when millions of users send requests. A model may be trained once and then serve requests continuously. That is why inference efficiency is a major part of modern AI infrastructure. How Does LLM Inference Work? Most widely used generative language models follow an autoregressive generation process. They predict one token at a time, using the prompt and previously generated tokens as context. The process has two important stages: prefill and decoding. 1. Tokenization: Converting text into tokens Before the model can process a prompt, a tokenizer converts the text into numerical token IDs. A token may represent a complete word, part of a word, punctuation, or another text unit. For example, the sentence “Inference makes AI useful” is converted into a sequence of token IDs. The exact IDs depend on the tokenizer used by the model. The model processes these numbers rather than ordinary text. 2. Prefill: Processing the input prompt During prefill, the model processes the prompt’s input tokens through its transformer layers. It calculates the attention and other neural network operations needed to build the internal context for generation. It also creates the initial key-value (KV) cache. Because the full input prompt is already available, the model can process many prompt tokens in parallel. This makes prefill relatively compute-intensive, particularly when users submit long prompts or large documents. Prefill strongly affects time to first token (TTFT), which measures how long a user waits before seeing the first generated token. Long context windows, large input batches, and prompt processing overhead can increase this time. 3. Decoding: Generating the response After prefill, the model begins decoding. It predicts the next token, adds it to the sequence, and performs another forward pass to predict the following token. It repeats this process until it reaches a stopping condition or the output limit. Unlike prefill, decoding is sequential for each individual request. The next token depends on the tokens that came before it. This makes it difficult to parallelize the generation of one response in the same way as prompt processing. At low batch sizes, decoding often becomes memory-bandwidth-bound. The GPU must access model weights and the relevant KV-cache data while producing each token. The arithmetic units may have unused capacity because the workload cannot feed them with data quickly enough. 4. KV cache: Reusing previous attention calculations The key-value cache stores attention-related information from tokens that the model has already processed. Without this cache, the model would need to recompute key and value representations for earlier tokens at every generation step. Reusing the cache avoids much of that repeated work and makes autoregressive generation practical. However, the cache consumes GPU memory, and its size grows as the sequence becomes longer and as more requests run concurrently. This creates an important trade-off: a larger KV cache can support more context and requests, but it also increases memory pressure. Inference engines use cache management techniques to allocate and reuse this memory efficiently. Prefill vs. Decode: Why the Difference Matters Feature Prefill Decode Input Processes the available prompt Generates one new token per sequence step Parallelism High across prompt tokens Limited within one sequence Common bottleneck Compute and memory capacity Memory bandwidth and KV-cache access Main latency metric Time to first token Time per output token Optimization focus Efficient prompt processing Weight reuse, batching, and cache management These stages have different hardware requirements. A configuration that improves prompt processing may not improve token generation. For example, a large prefill batch can use GPU compute efficiently, but it may delay decode requests that are waiting to produce their next tokens. Some serving systems therefore use chunked prefill to divide long prompts into smaller pieces and schedule them alongside decoding work. Larger deployments can also separate prefill and decode onto different GPU groups, allowing each stage to scale independently. What Are the Main LLM Inference Bottlenecks? A slow model does not always mean that the GPU is too weak.
