Embeddings Inside Document Q&A: Why Search Sometimes Misses
"Chat with your documents" tools, company knowledge bots, and assistants that answer from your files all work the same basic way. They search your documents for the parts related to your question, then hand those parts to a language model to write the answer. This approach is called RAG, short for retrieval-augmented generation.
The overall steps are covered in the lesson Paste, Upload, or Retrieve in the Context Engineering course. This lesson zooms in on the part that uses embeddings: the search step. When document Q&A gives a poor answer, the search step is very often where it went wrong.
What You'll Learn
- How documents are prepared so they can be searched by meaning
- Why documents are split into chunks, and why chunk size matters
- The main reasons the search step misses the right text
- How builders improve search, and what that means for you as a user
Preparing documents: chunk, embed, store
Before any question is asked, the documents are prepared:
- ChunkSplit each document into small pieces
- EmbedGive every chunk a location on the map
- StoreSave chunks and embeddings for fast search
Then, for each question, the question is embedded, the nearest chunks are found, and only those chunks are given to the language model.
The model only sees the chunks the search returns. If the right chunk is not among them, the model cannot use it, however capable it is. It may then say it does not know, or worse, answer from general knowledge as if it had checked your documents.
Why chunks, and why size matters
Recall from lesson 1 that one embedding for a whole long document blurs its details together. So documents are split into chunks, often a few paragraphs each. Chunk size is a real trade-off:
Good chunking keeps each piece about one idea, with enough context to stand on its own.
| Criteria | Chunks too small | Chunks too large |
|---|---|---|
| Example | One sentence each | Ten pages each |
| Problem | Loses context: "it" and "this rule" lose their meaning | Blurs many topics into one location |
| Effect on search | Finds a fragment without the key detail | Matches loosely, misses the specific part |
Chunks too small
- Example
- One sentence each
- Problem
- Loses context: "it" and "this rule" lose their meaning
- Effect on search
- Finds a fragment without the key detail
Chunks too large
- Example
- Ten pages each
- Problem
- Blurs many topics into one location
- Effect on search
- Matches loosely, misses the specific part
Many tools also let chunks overlap a little, or attach the document title and section heading to each chunk, so a piece still makes sense on its own.
Why the search step misses
Here are the most common causes, all linked to how embeddings work:
The question and the document use different words. Semantic search handles most paraphrases, but specialist terms, internal jargon, and abbreviations can land in unexpected places. Asking about "PTO" when the handbook says "annual leave" may or may not match, depending on the model.
The answer depends on an exact detail. Codes, names, dates, and numbers barely move a chunk's location. "What is the price of plan B-12?" may retrieve chunks about pricing in general instead of B-12 specifically.
The fact is split across chunks. A rule is in one chunk and its exception is in the next. The search returns the rule but not the exception, so the answer is confidently incomplete.
Related but not relevant. The search returns chunks on the right topic that do not contain the answer, like the "how long refunds take" example from lesson 2.
The question is too broad. "What are the main themes across all our reports?" needs every document. Search returns a handful of chunks, so the answer covers only a sliver.
The sources are stale. If an old version of a policy is still stored alongside the new one, search may return the old one. Embeddings measure meaning, not freshness.
How builders improve search
If you ever set up or choose a document Q&A tool, these are the techniques to look for:
- Hybrid search: combining meaning search with keyword search, so exact codes and names still match.
- Re-ranking: taking the top results and using a second, more careful model to reorder them by how well they actually answer the question.
- Better chunking: keeping headings, adding context to each chunk, and splitting at natural section breaks.
- Metadata filters: limiting search to a document type, date range, or department before searching by meaning.
- Keeping sources clean: removing outdated versions so they cannot be retrieved.
What it means for you as a user
You usually cannot change how a tool is built, but you can work with it:
- Ask specific, one-part questions, using the terms your documents use.
- Include exact names or codes in the question, since hybrid systems can match them.
- Ask for sources and quotes, then check them.
- Rephrase and ask again if the answer seems thin. Different wording lands in a different place on the map and can retrieve different chunks.
- Do not trust whole-collection answers ("summarize everything") from a tool that only retrieves a few chunks.
If you want to build a small private document search yourself, Local RAG for Beginners walks through it step by step.
Key Takeaways
- Document Q&A (RAG) searches your documents by meaning, then gives only the retrieved chunks to the model.
- Documents are split into chunks and embedded ahead of time. Chunk size is a trade-off between context and focus.
- Search misses because of different wording, exact details, split facts, related-but-irrelevant chunks, broad questions, and stale sources.
- Builders improve it with hybrid search, re-ranking, better chunking, filters, and clean sources.
- As a user: ask specific questions in the documents' own words, include exact names, ask for sources, and rephrase when answers are thin.