How to Fine-Tune an LLM Locally With LoRA and QLoRA

By /

You do not need a data center to teach a language model a new skill. With a technique called LoRA, and its memory-saving variant QLoRA, you can fine-tune an LLM locally on a single consumer GPU. The model can learn a writing style, a response format, or a narrow task, using a few hundred to a few thousand examples.

This guide explains what fine-tuning is, how LoRA and QLoRA work, and what hardware you realistically need. It then walks through a beginner workflow: install the tools, prepare a dataset, run training, evaluate the result, and use the adapter. We also cover troubleshooting and the limits of the method. The code follows current Hugging Face libraries, but these libraries change often, and we have not run this exact script for you, so check their documentation if an argument has changed.

Fine-Tuning, RAG, and Training From Scratch

People mix up three different things, and picking the wrong one wastes weeks.

Training From Scratch

Training from scratch builds a model from random weights. It needs huge datasets and large clusters of GPUs. It is not a realistic local project, and you almost never need it.

Fine-Tuning

Fine-tuning starts with a trained model and adjusts its weights on your examples. It is best for changing behavior: a consistent tone, a fixed output format, or skill at a narrow task. It is a poor way to add facts, because the model does not reliably memorize them.

Retrieval-Augmented Generation

RAG leaves the model unchanged and gives it relevant documents at question time. It is the better choice when you need the model to answer from your files, especially when those files change. Our tutorial on how to build a local RAG system covers it. Many projects use both: RAG for knowledge and fine-tuning for style.

LoRA and QLoRA Explained

Full fine-tuning updates every weight in the model, and that needs a lot of memory. LoRA and QLoRA avoid most of that cost.

What LoRA Does

LoRA stands for low-rank adaptation. Instead of changing the original weights, it freezes them and adds small extra matrices that learn the changes. These small matrices are called an adapter. Because the adapter holds only a tiny fraction of the parameters, training needs far less memory, and the saved file is small. The method comes from the paper LoRA: Low-Rank Adaptation of Large Language Models.

What QLoRA Adds

QLoRA goes one step further. It loads the frozen base model in 4-bit quantized form, then trains LoRA adapters on top. This cuts memory use even more, so larger models fit on smaller GPUs. The authors of the QLoRA paper reported fine-tuning a 65B model on a single 48 GB GPU. This is a different kind of quantization from the GGUF files you use for inference, which our GGUF quantization guide covers.

Key Terms to Know

  • Adapter: the small set of trained weights saved after fine-tuning.
  • Rank (r): the size of the adapter matrices. A higher rank can learn more but uses more memory. Values like 8, 16, and 32 are common starting points.
  • Alpha: a scaling factor for the adapter. A common habit is to set it to about twice the rank.
  • Target modules: the parts of the model that receive adapters. Many setups apply them to all linear layers.
  • Epoch: one full pass through the training data.

What Hardware Do You Need to Fine-Tune an LLM Locally?

Training uses much more memory than inference, because it stores gradients and optimizer values and works on batches of text. These are rough estimates, and the real numbers depend on sequence length, batch size, and settings.

Model size Typical approach GPU memory to plan for (estimate)
1B to 3B QLoRA or LoRA Around 8 GB or more
7B to 8B QLoRA Often 16 to 24 GB
13B to 14B QLoRA Often 24 GB or more
30B and above QLoRA 48 GB or more, or several GPUs

Platform support matters too. The common QLoRA stack relies on the bitsandbytes library, which works best with NVIDIA GPUs. On Windows, many people use WSL2 for a smoother setup. Apple Silicon users should look at the MLX tools, which have their own LoRA support, and AMD users should check current ROCm support. If you lack suitable hardware, rent a cloud GPU for a few hours. The workflow is the same.

Set Up Your Environment

Before you fine-tune an LLM locally, create a clean Python environment, then install the libraries. You need a recent NVIDIA driver and a PyTorch build with CUDA support, so follow the official PyTorch installation page first.

python -m venv .venv
source .venv/bin/activate
pip install transformers datasets peft trl accelerate bitsandbytes

Check that PyTorch sees your GPU:

python -c "import torch; print(torch.cuda.is_available())"

If this prints False, fix the driver or the PyTorch install before you go further. The PEFT documentation and the TRL documentation list current versions and options.

Prepare Your Dataset

To fine-tune an LLM locally with good results, your dataset matters more than any setting. A small set of clean, consistent examples beats a large messy one.

Choose a Format

For chat models, store each example as a conversation in JSON Lines format, with one JSON object per line:

{"messages": [{"role": "system", "content": "You write short, polite support replies."}, {"role": "user", "content": "My order has not arrived."}, {"role": "assistant", "content": "I am sorry about the delay. Please share your order number and I will check the status right away."}]}

Save the training examples in train.jsonl and a separate set in val.jsonl. Keep about 10 percent of your data for validation, and never train on it.

Follow Data Quality Rules

  • Write the answers you want. The model copies what it sees, including mistakes.
  • Keep the format consistent. If every answer should be three sentences, make every example three sentences.
  • Remove private data. Names, emails, and secrets can leak into the model’s outputs.
  • Cover edge cases. Include examples where the right response is to refuse or ask a question.
  • Check the license. Make sure you can legally use the data and the base model for your purpose.

For a narrow task, a few hundred good examples can give a visible change. Start small, test, and add more data where the model fails.

Write the Training Script

The script below loads a small base model in 4-bit form, attaches a LoRA adapter, and trains with the TRL library. It uses a small model so you can test the workflow on modest hardware. Save it as train.py.

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"
OUTPUT_DIR = "outputs/my-adapter"

# 1. Load the base model in 4-bit (QLoRA)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",
)

# 2. Describe the LoRA adapter
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",
    task_type="CAUSAL_LM",
)

# 3. Load the data
dataset = load_dataset(
    "json", data_files={"train": "train.jsonl", "validation": "val.jsonl"}
)

# 4. Set training options
training_args = SFTConfig(
    output_dir=OUTPUT_DIR,
    num_train_epochs=2,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=50,
    bf16=True,
    gradient_checkpointing=True,
    report_to="none",
)

# 5. Train and save the adapter
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["validation"],
    processing_class=tokenizer,
    peft_config=lora_config,
)
trainer.train()
trainer.save_model(OUTPUT_DIR)
tokenizer.save_pretrained(OUTPUT_DIR)

What the Important Settings Do

  • load_in_4bit and nf4: store the base weights in a 4-bit format designed for model weights.
  • r and lora_alpha: set the adapter size and scale. Raise r only if the model underfits.
  • per_device_train_batch_size and gradient_accumulation_steps: together set the effective batch size. Here it is 2 times 8, or 16 examples per update. Lower the first number if you run out of memory, and raise the second to keep the effective batch size.
  • learning_rate: 2e-4 is a common starting point for LoRA. Too high can break the model, and too low can leave it unchanged.
  • gradient_checkpointing: trades speed for memory by recomputing values during training.
  • bf16: needs a GPU that supports bfloat16, such as recent NVIDIA cards. On older cards, use fp16=True and change the compute dtype to torch.float16.

Library arguments change between versions. For example, older TRL releases used a tokenizer argument where newer ones use processing_class. If you see an error about an unexpected argument, check the documentation for your installed version.

Run and Monitor Training

Start training from the terminal:

python train.py

Watch the loss values that print every ten steps. The training loss should fall over time. The validation loss tells you more, because it shows how the model handles examples it has not trained on. Training that keeps lowering training loss while validation loss rises is a sign of overfitting. In that case, stop earlier, use fewer epochs, or add more varied data.

Evaluate the Fine-Tuned Model

A falling loss does not prove that the model is useful. You need to look at real outputs.

Compare Against the Base Model

Write 20 to 50 test prompts that are not in your training data. Run each through the base model and the fine-tuned model, and compare the answers side by side. Score them against the criteria you care about, such as correct format, tone, and accuracy.

Test for Regressions

Fine-tuning on a narrow task can make a model worse at other things, a problem sometimes called forgetting. Include a few general questions in your test set to see whether the model still behaves well outside your task.

Load the Adapter and Generate

This script loads the saved adapter on top of its base model and generates a reply:

import torch
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

ADAPTER_DIR = "outputs/my-adapter"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER_DIR)
model = AutoPeftModelForCausalLM.from_pretrained(ADAPTER_DIR, device_map="auto")

messages = [
    {"role": "system", "content": "You write short, polite support replies."},
    {"role": "user", "content": "I was charged twice for my subscription."},
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=150)

reply = output[0][inputs["input_ids"].shape[1]:]
print(tokenizer.decode(reply, skip_special_tokens=True))

Use the Adapter With Ollama or llama.cpp

Many people want to run the result in their usual local tools. The adapter works only with the exact base model it was trained on. Ollama can apply a LoRA adapter through the ADAPTER instruction in a Modelfile, and llama.cpp includes a script to convert a LoRA adapter to its own format. Supported architectures and formats change over time, so read the current Ollama import guide and the llama.cpp documentation before you try this. A common alternative is to merge the adapter into the base model, convert the merged model to GGUF, and then quantize it.

After you have a model that runs in Ollama, you can call it from code as shown in our Ollama API and Python tutorial.

Troubleshooting Common Problems

  • Out-of-memory error. Lower the batch size, shorten the maximum sequence length, keep gradient checkpointing on, or choose a smaller model.
  • bitsandbytes errors on install or import. This usually points to a mismatch between the CUDA version, the driver, and the library. Check the bitsandbytes documentation for your platform.
  • Loss does not move. Check that the data loads correctly, that the learning rate is not tiny, and that the examples really contain the pattern you want.
  • Loss becomes NaN or the output is garbage. Lower the learning rate, and confirm that your GPU supports the precision setting.
  • The model ignores your training. You may have too little data, too few steps, or a rank that is too low. Try more examples before you increase the rank.
  • The model repeats training examples word for word. That is overfitting. Use fewer epochs and more varied data.
  • Wrong chat format. Use the base model’s own chat template, as the tokenizer does by default, so training matches how you will prompt it.

Limitations of Local Fine-Tuning

Fine-tuning is powerful, but when you fine-tune an LLM locally you still face real limits. It does not reliably teach new facts. It can reduce safety behavior or general ability if your data pushes in that direction. Results depend heavily on data quality, and a weak base model stays weak. Finally, you are responsible for the license of both the base model and your data, so read the terms before you share or sell the result.

Conclusion

To fine-tune an LLM locally, start with a small instruct model, prepare a clean dataset, and use QLoRA to keep memory low. Train a LoRA adapter, watch the validation loss, and judge the result with your own test prompts instead of trusting the loss alone.

Remember what fine-tuning is for. Use it to shape behavior and format, and use RAG when the model needs your documents. Once you have a working adapter, you can build it into a larger system, such as the local AI agent in the next tutorial.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top