Embed and Store Locally with Chroma

You have chunks of your documents ready. Now we turn each chunk into an embedding and save it in a local database so it can be searched later. We will use Chroma, a free, open-source vector database that runs on your machine and stores everything in a folder on your disk. By the end of this lesson, your knowledge base will be built and saved.

What You'll Learn

• What Chroma is and why it is perfect for local RAG
• How to create a persistent Chroma database that saves to disk
• How to connect Chroma to Ollama's embedding model
• How to add your document chunks so they are embedded and stored

What Is Chroma?

A vector database is a database built to store embeddings (those lists of numbers from the embeddings lesson) and find the closest ones fast. Chroma is one of the simplest. It needs no separate server, installs with one pip command, and saves all your data in a local folder you choose. That keeps the whole system local and private, exactly what we want.

When you ask Chroma to store a chunk, it can automatically call an embedding model to turn that chunk into numbers, then save both the original text and its embedding together. Later, when you search, Chroma embeds your question and finds the nearest stored chunks for you. You never touch the math.

Step 1: Create a Persistent Database

By default, a database that lives only in memory disappears when your program ends, so you would have to rebuild it every time. We want it saved to disk so you build it once and reuse it forever. That is what PersistentClient does. Create a file called build.py:

import chromadb

# Saves the database to a folder named "db" in your project
client = chromadb.PersistentClient(path="db")

The path="db" tells Chroma to store everything in a folder named db. After you run this, you will see that folder appear, and your knowledge base survives between runs.

Step 2: Connect Chroma to Ollama's Embedding Model

Chroma needs to know how to turn text into embeddings. We point it at the nomic-embed-text model running in Ollama on your machine. Chroma ships with a ready-made embedding function for Ollama. Add this to build.py:

from chromadb.utils import embedding_functions

ollama_ef = embedding_functions.OllamaEmbeddingFunction(
    url="http://localhost:11434/api/embeddings",
    model_name="nomic-embed-text",
)

That url points to the embeddings endpoint of the local Ollama service from the Ollama lesson (localhost:11434). This line is what keeps embedding local: when Chroma needs numbers for a chunk, it asks the model running on your machine, not any cloud service.

Step 3: Create a Collection

A collection is like a table: a named place to keep a related group of chunks. We attach our embedding function to it so everything added is embedded the same way. Using get_or_create_collection means you can run the script again without errors; it reuses the collection if it already exists.

collection = client.get_or_create_collection(
    name="my_notes",
    embedding_function=ollama_ef,
)

Step 4: Add Your Chunks

Now we feed in the chunks from the previous lesson. Chroma's add method needs three things: the chunk texts (documents), and a unique id for each one. We will reuse the loading and chunking code from before. Here is the full build.py brought together:

import chromadb
from chromadb.utils import embedding_functions
from pypdf import PdfReader

# --- 1. Load and chunk (from the previous lesson) ---
def load_pdf(path):
    reader = PdfReader(path)
    return "\n".join(page.extract_text() or "" for page in reader.pages)

def chunk_text(text, chunk_size=800, overlap=100):
    assert overlap < chunk_size, "overlap must be smaller than chunk_size"
    chunks, start = [], 0
    while start < len(text):
        chunks.append(text[start:start + chunk_size])
        start += chunk_size - overlap
    return chunks

text = load_pdf("lecture.pdf")
chunks = chunk_text(text)

# --- 2. Set up Chroma + Ollama embeddings ---
client = chromadb.PersistentClient(path="db")
ollama_ef = embedding_functions.OllamaEmbeddingFunction(
    url="http://localhost:11434/api/embeddings",
    model_name="nomic-embed-text",
)
collection = client.get_or_create_collection(
    name="my_notes",
    embedding_function=ollama_ef,
)

# --- 3. Embed and store every chunk ---
collection.add(
    documents=chunks,
    ids=[f"chunk-{i}" for i in range(len(chunks))],
)

print(f"Stored {len(chunks)} chunks in the local database.")

When you run python build.py, here is what happens behind the scenes for each chunk: Chroma sends the text to Ollama's nomic-embed-text model, gets back the embedding, and saves the text plus its embedding into the db folder on your disk.

Step 5: Confirm It Worked

You can check how many chunks are stored with one line:

print(collection.count())   # should match the number of chunks

If the count matches the number of chunks you created, your knowledge base is built and saved. The db folder now holds an embedded, searchable copy of your document, entirely on your machine.

A Note on Running It Twice

Because we use get_or_create_collection and reuse the same ids, running the script again will not create duplicates for the same chunks; Chroma updates entries with matching ids. If you want a completely fresh start, simply delete the db folder and run the script again.

This is intentionally the simplest possible setup. Tuning how embeddings are indexed for speed and scale, choosing distance metrics, and handling millions of records are covered in the Vector Databases course. For a private study helper, the defaults here work well.

Key Takeaways

• Chroma is a free, local vector database that stores embeddings in a folder on your disk; install with pip install chromadb.
• Use PersistentClient(path="db") so your knowledge base is saved to disk and built only once.
• Connect Chroma to Ollama with OllamaEmbeddingFunction pointed at localhost:11434 and nomic-embed-text, keeping embedding local.
• A collection holds your chunks; collection.add(documents=..., ids=...) embeds and stores them automatically.
• Check collection.count() to confirm everything was stored. Your private knowledge base is now built.