Production RAG for Enterprise Search: Chunking, Retrieval, and Evals That Catch Regressions

Enterprise RAG often looks convincing in a demo. Upload a handful of documents, ask a question, and the model returns a plausible answer with a source link.

Production is a different system.

The corpus is larger and changes constantly. Permissions are uneven. Documents contain tables, boilerplate, scans, revisions, and contradictory policies. A query may span several systems. And a small change to parsing, embedding, or ranking can quietly make answers less grounded.

The hard part is rarely the prompt. It is building a retrieval system that selects the right evidence, preserves access boundaries, and makes failures visible before users find them.

This article covers the engineering choices that matter most: chunking, retrieval and reranking, citation hygiene, and evaluation.

What breaks when RAG moves from demo to production

A demo corpus is usually clean, small, and manually selected. Production corpora are not.

A policy repository may contain several versions of the same document. A knowledge base may mix HTML pages, PDFs, spreadsheets, tickets, and slide decks. Search results can be polluted by navigation text, repeated headers, legal footers, or template language. A retrieved passage can be semantically similar to the query while still being the wrong policy, region, product, or effective date.

That creates a common failure mode: the model writes a fluent answer based on weak evidence. Better prompt instructions can reduce some of the damage, but they cannot make irrelevant context relevant.

Treat RAG as a search and evidence-selection problem first. The generator should receive a small set of authoritative, accessible, and query-relevant passages. Its job is then to synthesize or abstain—not to compensate for poor retrieval.

There are several production concerns that a prototype often skips:

• Document lifecycle. New, updated, and deleted source documents must be reflected in the index predictably.
• Identity and authorization. Retrieval must enforce the caller’s access rights before content reaches the model.
• Source provenance. Each returned passage needs a stable source identifier, location, version, and timestamp.
• Observability. Teams need to inspect which chunks were retrieved, their scores, and the final citations.
• Change management. Parser, chunking, embedding, and ranker changes need regression testing just like application code.

If these foundations are missing, a system may appear reliable until the first policy change, permission incident, or high-stakes query.

Chunking strategies that reduce retrieval noise

Chunking is not just a preprocessing detail. It defines the units that retrieval can find and the units the model can cite.

The basic trade-off is straightforward. Small chunks improve precision: a matched chunk contains less unrelated text. But chunks that are too small lose the surrounding definition, exception, or table context needed to answer correctly. Large chunks preserve context but dilute relevance and consume more of the model’s context window.

There is no universal chunk size. The right boundary depends on the source structure and the questions users ask.

Start with document-aware boundaries. Split at headings, paragraphs, list groups, table sections, or code blocks before falling back to a token limit. A chunk from an HR policy should retain the section heading and, where relevant, the parent heading. A chunk from an API guide should preserve the endpoint, method, parameters, and nearby constraints. A table should not be separated from its headers without explicitly carrying those headers into the chunk.

Use overlap carefully. Modest overlap can preserve a sentence or definition that crosses a boundary. Excessive overlap creates near-duplicate vectors, crowds out diverse retrieval results, and makes it harder to tell which passage is truly useful. Measure the effect against representative queries rather than assuming more overlap is safer.

Metadata does as much work as the chunk text. At ingestion, attach fields such as:

• document ID and canonical URL or repository path;
• title and heading path;
• document type and business domain;
• owner, version, effective date, and ingestion timestamp;
• language and region where applicable;
• access-control attributes; and
• source offsets or page numbers for citations.

This metadata supports filtering and makes investigation possible. For example, a query about an expense policy should be able to prefer the current policy, restrict to the user’s country, and cite the relevant section—not an archived PDF with similar wording.

Avoid treating all documents alike. A slide deck, contract, support ticket, and wiki page require different extraction and chunking rules. Build a small set of source-specific ingestion paths. Then test them with real examples, especially documents with tables, repeated layout elements, scanned pages, and revisions.

Retrieval, reranking, and citation hygiene

A production retrieval path usually needs more than one nearest-neighbor lookup.

Dense retrieval is useful when a user’s wording differs from the source. Keyword or lexical retrieval remains valuable for exact product names, policy codes, error messages, and identifiers. A hybrid approach can give candidates from both methods, then apply metadata filters and a reranker to choose the final evidence set.

The sequence matters:

1. Authenticate the caller and resolve document-level permissions.
2. Normalize the query and apply mandatory filters such as tenant, region, or source scope.
3. Retrieve a broad candidate set using lexical, semantic, or hybrid search.
4. Rerank candidates against the full query.
5. Deduplicate overlapping chunks and preserve source diversity where appropriate.
6. Send only the selected evidence, with source metadata, to the generator.

Reranking is often where relevance improves most visibly. Initial retrieval is designed for recall: it should avoid missing good candidates. A reranker can then make a more precise judgement about which candidate actually answers the question. The best configuration depends on latency, corpus size, and query mix, so evaluate it on your own queries.

Citation hygiene belongs in the interface between retrieval and generation. Do not ask the model to invent references after it has composed an answer. Instead, pass stable citation IDs with each evidence passage and require claims to map only to those IDs. The application should render the underlying source title, location, and version from trusted metadata.

Also test whether the cited passage supports the sentence next to it. A model can attach a real citation to a nearby but unsupported conclusion. This is a citation entailment problem, not merely a formatting problem.

When evidence is insufficient or sources disagree, the correct behavior is to say so. Build an abstention path: explain what was not found, link to the closest sources if useful, and provide a route for escalation. Unsupported confidence is not a feature.

Evals for grounded answers and regression detection

RAG quality cannot be managed from anecdotal chat transcripts. Build an evaluation set before making major pipeline changes.

A useful set contains representative user questions, expected evidence, and expected answer behavior. Include easy questions, ambiguous questions, queries with exact identifiers, cross-document questions, permission-sensitive queries, and questions that should be refused or escalated because the source corpus does not contain enough evidence.

Evaluate the pipeline at separate layers.

Retrieval evaluation asks whether the relevant source appears in the candidate set and in the final context. Track measures such as recall at a chosen cutoff, ranking quality, and the share of queries where all required evidence is retrieved. Segment results by document type, business domain, language, and query category. An aggregate score can hide a serious drop for one important source.

Grounded-answer evaluation asks whether the answer is supported by the retrieved evidence. Review factual correctness, completeness, citation placement, and whether each citation entails the associated claim. Explicitly label unsupported claims, fabricated citations, and stale-source answers.

Operational evaluation asks whether the system meets its service constraints. Track latency by stage, retrieval failures, empty-result rate, permission-filter outcomes, and index freshness. A highly relevant answer that arrives too late—or uses content a user should not see—is not production-ready.

Use a mix of automated checks and human review. Automated graders can help triage large test sets, but they should be calibrated against expert-labelled examples. For high-impact domains, keep a human-reviewed gold set and use it as the release gate.

Every change should run against the same baseline: a new embedding model, a revised parser, a different chunking rule, a reranker update, or a prompt edit. Compare results by segment, inspect failures, and define acceptable degradation in advance. That is how you catch regressions that a few hand-picked examples would miss.

Deployment considerations for enterprise infrastructure

Enterprise search requires controls at the document layer, not just at the chat interface.

Enforce authorization during retrieval. Do not retrieve broadly and rely on the model to ignore restricted passages. Propagate source-system permissions into the index, validate them during query execution, and test permission changes such as role removal and document revocation.

Plan for freshness. Define how ingestion detects updates, how old index entries are removed, and how users can see the source version or effective date. For regulated or policy-heavy use cases, maintaining an auditable ingestion record matters as much as retrieval relevance.

Design for investigation. Store structured traces for each request: normalized query, filters applied, retrieved chunk IDs, scores, reranking decisions, selected context, response, and rendered citations. Protect these logs appropriately; they may themselves contain sensitive content. The goal is to make a bad answer diagnosable without reconstructing the request from memory.

Finally, deploy incrementally. Start with a bounded corpus and a clearly defined query class. Establish a baseline evaluation set. Add sources and use cases only after the retrieval, access-control, and evaluation loops are operating reliably. Production RAG is not a one-time integration. It is a search system with an AI interface, and it needs the same operational discipline.

A reliable enterprise RAG system does not depend on a clever prompt. It depends on clean source handling, deliberate chunking, evidence-aware ranking, strict document-level access controls, and evals that detect when the system gets worse.

Talk to Bicara Labs about building and deploying an enterprise AI search system that is grounded in your documents and your infrastructure.