RAG Pipeline Development: Lessons from Production

Every month someone declares RAG dead — usually right before shipping a system that quietly contains a retrieval pipeline. The truth from production is less dramatic: RAG pipeline development in 2026 is a solved-enough engineering discipline with known best practices, known failure modes, and one big strategic shift (context engineering) that changes how you should think about it.

We build and operate retrieval systems for clients — document-heavy businesses where the answer has to be right, not plausible. Here is the pipeline we actually ship, what the research says about each choice, and the mistakes we see teams make.

Why "just use a bigger context window" keeps failing

The most common objection to building a retrieval pipeline: modern models take a million tokens, so why not paste everything in? Because long context degrades in ways benchmarks long hid:

• "Lost in the Middle" showed models use information at the start and end of context far better than the middle.
• NoLiMa removed literal word-overlap between question and needle — and 13 models claiming 128K+ context degraded sharply at lengths far below their claimed windows.
• Chroma's "Context Rot" report tested 18 models and found performance drops non-uniformly as input grows even on trivially simple tasks — and focused ~300-token prompts beat full ~113k-token dumps on long-conversation benchmarks.
• A follow-up study found degradation persists even when retrieval within the long context is perfect. Length itself hurts.

This is why "context engineering" — the term Shopify's Tobi Lütke coined in June 2025 and Karpathy amplified — took over from prompt engineering: the game is filling the window with just the right information. Retrieval is not competing with long context. Retrieval is how you keep contexts small, and small contexts are how you keep answers right.

The pipeline we ship

After 25+ delivered projects, our default RAG architecture is deliberately unexciting. Excitement is for demos; production needs layers you can debug:

1. Ingestion with structure awareness. Chunk along document structure (sections, tables, headings), roughly 500 tokens with overlap, and enrich every chunk with metadata — source, date, access rights. Access rights at ingestion, not as an afterthought: your retrieval layer is also your permission layer.

2. Contextualized chunks. Anthropic's contextual retrieval numbers are still the benchmark worth knowing: prepending short LLM-generated context to each chunk cut failed retrievals by 49%, and 67% combined with reranking. We apply the same idea wherever the corpus allows it.

3. Hybrid search. Dense vectors miss exact identifiers; BM25 misses paraphrases. We run both and fuse rank-based (reciprocal rank fusion), because BM25 scores and cosine similarities are not comparable numbers. Qdrant made this a first-class query feature — with weighted fusion since 1.17 — and pgvector's iterative index scans fixed the classic "filtered query returns three rows" problem for the Postgres-native route. Which one we pick depends on the client's ops reality, not ideology: Qdrant when vector search is the workload, pgvector when the data already lives in Postgres.

4. Reranking. A cross-encoder over the top ~50 candidates is the cheapest quality upgrade in the whole pipeline. It also caps what enters the context window — context engineering enforced by architecture.

5. Evaluation before launch, not after. RAGAS established the component metrics (faithfulness, context precision/recall), but know its limit: it checks whether the answer is consistent with what was retrieved — not whether what was retrieved is true. So we build a golden set of real client questions with verified answers and gate releases on it. When retrieval quality is not measured, it is imagined.

What changed recently (and what we ignore)

Worth adopting:

• Better open embeddings. Qwen3-Embedding topped multilingual MTEB with open weights — important for us, because German-language corpora and EU data residency often rule out API embedding calls. Small on-device models like EmbeddingGemma make fully-local RAG realistic.
• Re-embeddable model families. Voyage 4 ships models sharing one embedding space — embed cheap, upgrade later without re-indexing everything. Expect this pattern to spread.
• Cheaper memory. Qdrant 1.18's TurboQuant roughly doubles compression at similar recall — directly relevant when everything runs on EU infrastructure and RAM is the cost driver.

Worth skepticism:

• "Agentic retrieval" as a product checkbox. Letting an agent plan and decompose queries genuinely helps on complex questions — the Agentic RAG survey maps the patterns, and cloud vendors now ship it natively. But an agent looping over a bad index produces confident garbage faster. Fix retrieval first; add agency second.
• Skipping straight to knowledge graphs. Graphs pay off for genuinely relational questions — we wrote a full decision guide in GraphRAG vs RAG. They do not fix a broken chunking strategy.

A real example

For Weltberg Immobilien, documents, property data and communication history feed both a SaaS dashboard and a team of five AI agents running 12+ workflows in production. The retrieval layer is what makes the agents trustworthy: every answer traces back to retrieved sources, and every agent decision is logged. Their leadership's verdict — a system that "runs in production every day and makes decisions we genuinely trust" — is, in engineering terms, a statement about the retrieval pipeline as much as the model.

A worked example: from document chaos to trusted answers

Here is what the pipeline looks like applied, using a composite of the document-heavy projects we ship. A business sits on tens of thousands of PDFs, emails and records: contracts, protocols, invoices, correspondence. The team answers questions by memory and full-text search, and onboarding a new employee takes months because the knowledge lives in heads.

Week one is corpus triage, not code. We inventory document types, owners and permissions, and — most importantly — collect fifty real questions the team actually asks, with the correct answers and where they came from. That golden set becomes the project's definition of truth; every architecture decision afterwards is measured against it.

Week two is ingestion. Structure-aware chunking per document type (contracts split by clause, protocols by agenda item), metadata on every chunk — client, project, date, access level. This is where most of the eventual answer quality is decided, and it is the least glamorous work in the project.

Week three is retrieval assembly. Hybrid search over the collections, rank fusion, a reranker, and contextualized chunks where document types are dense with cross-references. Then we run the golden set and look at retrieval metrics before anyone judges answer prose: did the right chunks surface in the top ten? If not, no prompt will save the answer.

Week four is hardening. Permission filters tested with adversarial queries (can an intern's account retrieve management-level contracts? — this must fail), source citations rendered in the UI, and the evaluation wired into CI so every future change re-runs the golden set.

The result is not a chatbot; it is an answering system the team quotes in meetings because every answer carries its sources. That sourcing discipline is also what makes the same retrieval layer reusable later — the agent workflows we build on top inherit a foundation that is already measured and permission-aware.

Common mistakes we get called in to fix

1. One giant index for everything — mixing contracts, emails and marketing copy, then wondering why answers cite the wrong register. Separate collections, separate metadata.
2. No golden questions. If you cannot answer "did last week's change make retrieval better or worse?", you are steering blind.
3. Ignoring permissions. A RAG system that answers from documents users are not allowed to see is a GDPR incident with extra steps. EU-hosted, permission-aware retrieval is our default — data never leaves Europe.
4. Embedding once, forever. Corpora drift. Schedule re-ingestion like you schedule backups.
5. Betting the answer quality on one retriever. Hybrid + rerank exists precisely because every single method has embarrassing failure cases.

Questions clients keep asking us

"Which embedding model should we pick?" For German-and-English corpora with EU residency requirements, a strong open multilingual model served inside your infrastructure is our default recommendation — and more important than the specific pick is wiring your evaluation set so you can swap models in an afternoon when a better one ships. The model is a component; the evaluation harness is the asset.

"Can we skip the reranker to save latency?" Measure before you decide: a reranker over 50 candidates adds tens of milliseconds and routinely rescues the answers your users would have complained about. If latency genuinely matters (live chat), rerank asynchronously for the citation view while streaming the first answer — you rarely have to choose.

"How much data do we need before RAG makes sense?" Less than teams assume. The threshold is not corpus size — it is question frequency. A 200-document policy handbook that the team queries daily beats a million-document archive nobody asks about. Start where the questions already are.

The short version

RAG pipeline development in 2026 means: structure-aware chunking, contextualized embeddings, hybrid search with rank fusion, a reranker, and an evaluation set you actually trust — inside an architecture where retrieval keeps contexts small, because the research is unambiguous that big contexts rot. That system is not glamorous. It is the difference between an assistant your team quotes in meetings and one they quietly stop using.

This post is part of our practical guide to AI automation for businesses. If you want a retrieval system built for your documents, your language and EU compliance — book a call with Nexuri. We respond within 24 hours.