Document Ingestion Pipelines
Ingestion is the offline prep that turns messy PDFs and pages into clean chunks with vectors and metadata, so your model can find the right passage at answer time.
Why Does This Exist?
You ask about the parental leave policy and the model answers with total confidence. The policy says twelve weeks. The model says six. You check the index and the memo was never parsed, so retrieval had nothing to hand the model. That is an ingestion failure, not a prompting failure. No prompt tweak fixes it.
Your source files do not look like training data. They are two column PDFs where the reading order jumps across the page, scanned images with no text layer, wiki pages full of macros and warning banners, chat threads that change topic mid scroll. An embedding expects one clean passage about one thing, not a raw file. Feed it the raw file and you get a blurry vector that matches nothing precisely. Retrieval quality caps everything downstream, so the ceiling of your answers is set here, before any query arrives.
This page covers the whole offline path from file to searchable store: parse, clean, split, embed, and load with metadata. The cut itself gets its own page in chunking strategies, storage gets vector databases, and the live query half gets RAG architecture. The one idea underneath everything is simple. An embedding is a list of numbers where distance means similarity, so related passages land close together. The short glossary form is embedding.
Think of It Like This
A prep kitchen before the dinner rush
You run a small prep kitchen. In the morning, farmers drop off muddy crates: carrots with tops still on, flour in torn sacks, fish on ice. You do not hand those crates to the line cooks during service. You wash and trim, you portion everything into small labeled containers, and you stack those containers on freezer shelves with dates and contents facing out. At dinner rush, the cooks grab a container and cook. Fast. No washing, no guessing.
Raw PDFs and wiki pages are the muddy crates. Parsing and cleaning are the wash and trim. A chunk is one portioned container, sized for a single use. Its vector is the label that says what is inside in a language the shelves understand, and the shelf is the vector store. The cooks are the model at query time.
Where it stops: a real kitchen hates duplication because it wastes food. Here you duplicate on purpose. Each container shares a scoop with the next one, so nothing important sits exactly on a cut line. And a cook can taste a sauce and adjust. Your model cannot. If a container never made it to the shelf, the model cannot taste around the gap. It just answers without it.
How It Actually Works
Ingestion runs offline, once per document, not once per query. You pay the cost up front and spread it across every future question that might touch that document. Five steps, in order, each with one job. The diagram above walks the same fixture you will compute below.
Parse the file into text that keeps its structure
Parsing turns bytes into usable text. Plain text and markdown are easy. HTML needs tag stripping that keeps headings and lists, because those boundaries later make good cut points. PDFs are the hard case. Two column layouts, footnotes, tables, and page numbers break naive extractors, and a scanned page has no text at all until OCR or a vision model reads it.
Keep two things from this step. First, the text in reading order. Second, the source record: filename or URL, title, page numbers, and a stable document id. You will need that id later to update or delete. If a table comes out as scattered cells, fix parsing before you touch anything else. No chunking trick rescues text that was already scrambled.
Clean without deleting meaning
Cleaning removes what would pollute search. Repeating headers and footers, navigation chrome, excess whitespace, hyphenation from line wraps, and normalized unicode quotes and dashes. You have seen the symptom of skipping this: every chunk starts with Confidential Internal and the query for vacation policy matches the footer instead of the policy.
Be conservative. Strip boilerplate you can name exactly, and keep dates, names, and numbers even when they look noisy. The assumption here is that your rules remove chrome, not content. When that assumption fails, usually on legal or medical text with odd formatting, the failure is silent. Chunks look clean and retrieval quietly misses the clause you deleted.
Cut into overlapping chunks
Now split the cleaned text into chunks, passage sized pieces that each get their own vector. Call the total words in the cleaned text, the chunk size in words, the overlap in words, the stride or step forward each time, and the number of chunks you will store. In plain words: stride is how far you move before cutting again, and overlap is how much you repeat so a sentence on a boundary survives inside the next chunk.
The relations are:
when is larger than and the numbers divide evenly. Total stored words are , which equals plus repeated words. Overlap is deliberate duplication. You pay extra storage to protect boundaries.
Size is the trade off. Small chunks match precisely but carry little context, so the model gets a sharp fragment. Large chunks carry context but average many ideas into one vector, so they match vaguely. Overlap in the 10 to 25 percent range is the usual compromise, and the full treatment with structure aware cuts lives in chunking strategies. Two named variants solve the extremes: semantic chunking cuts where the topic shifts, and parent-child chunking searches small pieces but hands the model the larger parent. The glossary form is chunking.
Embed each chunk and load it with metadata
An embedding model turns each chunk into an embedding vector, a list of numbers where closeness means similar meaning. Call the vector for chunk the value . Use the same model for documents and for queries. Vectors from different models live in different spaces and do not compare, so swapping models means re embedding everything.
Loading stores more than . Each record holds the vector, the chunk text itself so the model has something to read, plus metadata: source id, chunk index, section title, timestamp, and access tags like team or role. Metadata is what lets retrieval filter before it ranks, for example only this user's documents, or only policies newer than January. That store is a vector database, covered in vector databases. Skip metadata and you can still search, but you cannot explain where an answer came from or restrict who sees it.
Worked example: 1,050 words to four vectors
Take a cleaned policy memo with words. Choose words and words. Then words per step, and:
Starts fall at words 0, 250, 500, and 750. In one based counting the chunks cover words 1 to 300, 251 to 550, 501 to 800, and 751 to 1050. Each boundary repeats 50 words: words 251 to 300 appear in chunks one and two, and so on. Total stored is words, which is plus repeated words, about 14 percent overhead for boundary safety.
Embed each chunk with one model call per chunk to get . Load four records. Record three might read: text words 501 to 800, source memo parental leave v3, chunk index 2 of 4, date, team hr. At query time the retriever compares the query vector against these four, not against the whole memo, which is why each chunk needs to read sensibly on its own.
When should you skip all this? If your corpus is small, clean, and stable enough to paste into the prompt, do that. Ingestion pays off when the corpus is large, messy, or changing. And if your source changes hourly and stale answers are worse than slow answers, an index you rebuild nightly will hurt you. Prefer live retrieval or a shorter refresh loop. Debugging a bad answer by stage, cheapest check first, is in RAG failure diagnosis.
Code
The point here is the counting, not the model call. This snippet builds the four chunks from the worked example with plain Python, so you can see the starts, the sizes, and the 50 word repeat.
def chunk_words(words: list[str], chunk_size: int = 300, overlap: int = 50) -> list[list[str]]: stride: int = chunk_size - overlap chunks: list[list[str]] = [] start: int = 0 while start < len(words): end: int = min(start + chunk_size, len(words)) chunks.append(words[start:end]) if end == len(words): break start += stride return chunks
words: list[str] = [f"w{i}" for i in range(1050)]chunks: list[list[str]] = chunk_words(words)
print(len(chunks)) # -> 4print([len(c) for c in chunks]) # -> [300, 300, 300, 300]print(chunks[1][0]) # -> w250Four chunks, each 300 words, and the second chunk starts at word 250, which is exactly the 50 word repeat of words 250 through 299 from the first chunk. Change overlap to zero and you get no safety net. Change it to 150 and you pay double the duplication for little extra protection.
Watch Out For
A boundary cuts the one sentence that mattered
Symptom: retrieval looks healthy, scores look fine, but the model quotes half a thought or misses the condition attached to a rule. You open the stored chunks and the key sentence is split. Chunk two ends with employees are eligible after, chunk three starts with six months of service. Neither half matches the query well, and the model only receives one half.
Fix: open the stored chunks near the miss before you retune anything else. Add 10 to 25 percent overlap if you have none, and cut on structure first: headings, then paragraphs, then sentences, with fixed size only as the last resort for oversize pieces. For tables and procedures, keep the whole unit together or switch to parent child retrieval so the small match still delivers the full context.
The index goes stale while the docs move on
Symptom: the wiki was corrected last week but the model still quotes the old numbers, or a filter for team sales returns engineering docs. Reingesting everything nightly did not help because old chunk records were never deleted, so two versions of the same memo now compete.
Fix: do not treat ingestion as append only. Key every record by document id plus chunk index, store source, timestamp, and access tags at load time, and make updates delete then insert for that document id. Keep a content hash so you can tell a real edit from a no op. And keep one embedding model version for the whole index. A new model version means a full re embed, because old and new vectors do not compare.
The Quick Version
- Ingestion is the offline ETL for unstructured files: parse, clean, chunk, embed, and load with metadata before any query arrives.
- A 1,050-word memo with 300-word chunks and 50-word overlap becomes four vectors with starts at words 0, 250, 500, and 750.
- Stride equals chunk size minus overlap, so overlap is paid duplication that keeps split sentences whole inside the next chunk.
- Each stored record needs the vector, the chunk text, and metadata with source, chunk index, timestamp, and access tags.
- Small corpora that fit the prompt do not need this machinery, and fast changing sources need a refresh loop or live retrieval instead.
- When answers go wrong, read the stored chunks first, then check retrieval, then generation, because ingestion failures look like model failures.