Language models know a lot, but they do not know your files. They have never read your notes, your company handbook, or your research papers. A local RAG system fixes this by searching your documents when you ask a question and giving the best passages to the model, all on your own computer.
RAG stands for retrieval-augmented generation. This tutorial explains how it works, then builds a small but complete version with Ollama for the models and Chroma as the vector database. You will learn embeddings, chunking, indexing, retrieval, and generation, and you will see how to check whether your system actually works. We have not run this exact code for you, so test it on a small folder first.
How a Local RAG System Works
A RAG pipeline has two stages that run at different times. Understanding them keeps the rest of the tutorial simple.
The Indexing Stage
You prepare your documents once, or whenever they change. The pipeline reads each file, cuts the text into smaller pieces called chunks, and converts each chunk into a list of numbers called an embedding. It then stores the chunks and embeddings in a vector database.
The Question Stage
When you ask a question, the pipeline converts the question into an embedding, searches the database for the chunks whose embeddings are closest, and places those chunks in a prompt. The language model then writes an answer using that material.
Why Use RAG Instead of Pasting Documents
If you only have one short document, pasting it into the prompt works. RAG helps when the collection is too large to fit in the context window, or when you want the model to look only at relevant parts. It also keeps your data local, and you can update the index without retraining anything. Note that RAG is different from fine-tuning, which changes the model itself. Our guide to fine-tuning an LLM locally explains when to use each.
Key Concepts Explained
Four terms come up constantly, so it helps to define them first.
Embeddings
An embedding is a list of numbers that represents the meaning of a piece of text. Texts with similar meaning get similar numbers. An embedding model creates them, and it is separate from the chat model. Embedding models are small. For example, the nomic-embed-text model in the Ollama library is a few hundred megabytes.
Chunking
Chunking means splitting documents into pieces. Chunks must be small enough to be specific, and large enough to carry meaning. A chunk of a few hundred words often works as a starting point, but the best size depends on your documents.
Vector Databases
A vector database stores embeddings and finds the nearest ones quickly. Chroma is a popular open-source choice that runs inside your Python program and saves data to a local folder, so it needs no separate server.
Similarity Search
The database compares the question’s embedding with stored embeddings and returns the closest matches. Distance measures such as cosine distance score the match. A smaller distance means a closer match.
What You Need Before You Build a Local RAG System
- Ollama installed and running. See our Ollama API and Python tutorial if you need setup help.
- Python 3.10 or newer.
- A chat model and an embedding model. This tutorial uses
llama3.1:8bandnomic-embed-textas examples. - A folder of text or Markdown files to search.
Download the models and install the Python packages:
ollama pull nomic-embed-text
ollama pull llama3.1:8b
python -m venv .venv
source .venv/bin/activate
pip install ollama chromadbOn Windows, activate the environment with .venv\Scripts\activate. Choose a chat model that fits your memory, using our guide to LLM VRAM requirements. Embedding models need very little.
Build the System Step by Step
We will write one file, rag.py, that can index a folder and answer questions. The code is short enough to read in one sitting.
Step 1: Set Up and Split Text Into Chunks
The first part sets configuration values and splits text. It breaks text at blank lines so paragraphs stay together, and it adds a small overlap between neighboring chunks, which helps when an answer sits across a boundary.
import sys
from pathlib import Path
import chromadb
import ollama
EMBED_MODEL = "nomic-embed-text"
CHAT_MODEL = "llama3.1:8b"
CHUNK_SIZE = 800 # approximate characters per chunk
CHUNK_OVERLAP = 120 # characters carried into the next chunk
TOP_K = 4 # chunks to retrieve per question
client = chromadb.PersistentClient(path="chroma_db")
collection = client.get_or_create_collection(
name="docs", metadata={"hnsw:space": "cosine"}
)
def split_long(paragraph: str) -> list[str]:
if len(paragraph) <= CHUNK_SIZE:
return [paragraph]
step = CHUNK_SIZE - CHUNK_OVERLAP
return [paragraph[i:i + CHUNK_SIZE] for i in range(0, len(paragraph), step)]
def chunk_text(text: str) -> list[str]:
paragraphs = []
for block in text.split("\n\n"):
block = block.strip()
if block:
paragraphs.extend(split_long(block))
chunks, current = [], ""
for paragraph in paragraphs:
if len(current) + len(paragraph) + 2 <= CHUNK_SIZE:
current = f"{current}\n\n{paragraph}" if current else paragraph
else:
if current:
chunks.append(current)
tail = current[-CHUNK_OVERLAP:] if current else ""
current = f"{tail}\n\n{paragraph}" if tail else paragraph
if current:
chunks.append(current)
return chunksStep 2: Create Embeddings and Index the Files
Next, add the functions that turn text into embeddings and store everything in Chroma. The nomic-embed-text model works best when you add a short prefix that marks the text as a document or a query. Other embedding models use different conventions, so read the model card for the one you choose.
def embed(texts: list[str], prefix: str) -> list[list[float]]:
result = []
for start in range(0, len(texts), 32):
batch = [prefix + t for t in texts[start:start + 32]]
response = ollama.embed(model=EMBED_MODEL, input=batch)
result.extend(response.embeddings)
return result
def index_folder(folder: str) -> None:
root = Path(folder)
files = [f for f in root.rglob("*") if f.suffix.lower() in {".txt", ".md"}]
if not files:
sys.exit("No .txt or .md files found.")
for file in files:
text = file.read_text(encoding="utf-8", errors="ignore")
chunks = chunk_text(text)
if not chunks:
continue
relative = str(file.relative_to(root))
collection.upsert(
ids=[f"{relative}::{i}" for i in range(len(chunks))],
documents=chunks,
embeddings=embed(chunks, "search_document: "),
metadatas=[{"source": relative, "chunk": i} for i in range(len(chunks))],
)
print(f"Indexed {relative}: {len(chunks)} chunks")Using upsert means you can re-run the indexing after you edit a file without creating duplicates. If you shorten a file, old chunks with higher numbers will remain, so delete the collection when you need a clean rebuild.
Step 3: Retrieve and Generate an Answer
The last part searches the database and builds the prompt. The prompt tells the model to use only the supplied context and to admit when the answer is missing. That instruction reduces made-up answers, though it does not remove them.
def ask(question: str) -> None:
query_embedding = embed([question], "search_query: ")[0]
results = collection.query(query_embeddings=[query_embedding], n_results=TOP_K)
documents = results["documents"][0]
metadatas = results["metadatas"][0]
distances = results["distances"][0]
context = "\n\n".join(
f"[{i + 1}] (source: {m['source']})\n{d}"
for i, (d, m) in enumerate(zip(documents, metadatas))
)
prompt = (
"Answer the question using only the context below. "
"If the context does not contain the answer, say you do not know. "
"Cite the sources you use, like [1].\n\n"
f"Context:\n{context}\n\nQuestion: {question}"
)
response = ollama.chat(
model=CHAT_MODEL,
messages=[{"role": "user", "content": prompt}],
options={"temperature": 0.1, "num_ctx": 8192},
)
print(response.message.content)
print("\nRetrieved chunks:")
for i, (m, dist) in enumerate(zip(metadatas, distances)):
print(f"[{i + 1}] {m['source']} (chunk {m['chunk']}, distance {dist:.3f})")
if __name__ == "__main__":
if len(sys.argv) >= 3 and sys.argv[1] == "index":
index_folder(sys.argv[2])
elif len(sys.argv) >= 3 and sys.argv[1] == "ask":
ask(" ".join(sys.argv[2:]))
else:
print('Usage: python rag.py index FOLDER | python rag.py ask "QUESTION"')Step 4: Run It
Put a few text files in a folder named docs, then run:
python rag.py index docs
python rag.py ask "What is our refund policy?"The program prints the answer and the retrieved chunks with their distances. Always look at the retrieved chunks. They show you whether the system found the right material, which is the most common point of failure.
Improve Retrieval Quality
Most problems in a local RAG system are retrieval problems. If the right chunk never reaches the model, even a strong model cannot answer well. These changes often help.
- Tune the chunk size. Very small chunks lose context, and very large chunks bury the answer. Try a few sizes and compare results on real questions.
- Keep structure. Split on headings and sections where possible. Adding the document title or heading to each chunk gives the embedding more context.
- Store metadata. File names, dates, and sections let you filter results, and they make citations possible.
- Retrieve more, then narrow. Fetch ten chunks and use a reranker to pick the best few. Reranking models score each chunk against the question and often improve precision.
- Combine keyword and vector search. Keyword search catches exact terms like product codes, which embeddings sometimes miss.
- Choose a stronger embedding model. Models differ in quality and language support, so test more than one.
Handle Different Document Types
Real collections include more than text files. Each type needs its own handling.
For PDFs, a library such as pypdf can extract text from pages that contain real text. Scanned PDFs hold images, so they need optical character recognition (OCR) first. Check the extracted text, because tables and multi-column layouts often come out jumbled.
For web pages and Word files, convert them to clean text or Markdown before you chunk them. Remove menus, footers, and repeated headers, since they add noise to your index.
For tables and spreadsheets, plain text chunking usually works poorly. Consider converting each row or section into a short sentence-like description before indexing.
Limitations of RAG
A local RAG system is useful, but it is not magic. Know these limits before you rely on it.
- The model can still be wrong. It may ignore the context, misread it, or blend it with its own knowledge. Citations help you check.
- Retrieval can miss. Questions that need information from many places, such as “summarize every complaint this year,” often fail with a small top-k.
- Bad documents give bad answers. Outdated or conflicting files produce outdated or conflicting responses.
- Context is limited. Every retrieved chunk uses space in the prompt, and a larger window needs more memory.
- Sensitive data needs care. A local setup keeps data on your machine, but the index files hold your content in readable form, so protect the folder.
How to Evaluate Your System
Do not judge a local RAG system by a few good-looking answers. Measure it in two parts, retrieval and generation.
Test Retrieval First
Write 20 to 30 questions that your documents can answer, and note which file should contain each answer. Then check how often the right file appears among the retrieved chunks. This is a simple hit rate. A short script can run it:
tests = [
("What is the refund window?", "policies.md"),
("How do I reset my password?", "faq.md"),
]
hits = 0
for question, expected in tests:
query_embedding = embed([question], "search_query: ")[0]
results = collection.query(query_embeddings=[query_embedding], n_results=TOP_K)
sources = [m["source"] for m in results["metadatas"][0]]
hits += expected in sources
print(f"Hit rate: {hits}/{len(tests)}")If the hit rate is low, fix retrieval before you touch the prompt or the model.
Then Test the Answers
Read the generated answers for a sample of questions. Check three things: whether the answer is correct, whether it stays within the retrieved text, and whether it admits uncertainty when the context lacks the answer. Include a few questions that your documents cannot answer, and see whether the model says it does not know.
Repeat both tests after every change, so you can tell which change helped. For the model side, our guide to the best local LLMs for different tasks can help you pick a stronger generator.
Conclusion
A local RAG system has five working parts: documents split into chunks, embeddings that capture meaning, a vector database, a retrieval step, and a model that writes the answer. With Ollama and Chroma, you can run all of it on one computer and keep your files private.
Start small, inspect the retrieved chunks, and build a short test set before you tune anything. Most gains come from better chunking and retrieval, not from a bigger model. Once your search works well, you can give it to an assistant that decides when to look things up, as shown in our tutorial on a local AI agent.