There is a version of Retrieval-Augmented Generation that takes an afternoon: chunk some PDFs, push embeddings into a vector store, stuff the top matches into a prompt, and ship. It demos beautifully. Then it meets production - a user asks something the corpus half-answers, the model confidently fills the gap, and now a wrong number with a fabricated citation is sitting in front of a customer. The gap between that demo and a system an auditor will sign off on is where the actual engineering lives.
At BulkBeings we build RAG for teams who cannot afford a confident wrong answer - clauses in a contract, a dosage, a policy limit, a regulatory filing. The goal is never "answer everything." It is "answer what the sources support, cite it to the sentence, and refuse the rest out loud." That reframing changes every layer of the stack. This post walks the layers we actually build, and where demos quietly cut corners.
A demo RAG vs. a RAG that survives an audit
The difference is not model quality. It is accountability. A demo answers; a production system can show its work. Before writing a line of retrieval code, we make the auditable version the spec.
- Demo: retrieves top-k by cosine similarity. Production: hybrid retrieval (dense + lexical), then a reranker, then a relevance floor that can return nothing.
- Demo: pastes chunks into the prompt. Production: passes chunks with stable IDs and forces every claim to map back to a source span.
- Demo: 'looks right' is the eval. Production: a versioned eval harness scores retrieval and generation separately, and gates deploys.
- Demo: answers confidently when unsure. Production: refuses gracefully and logs why.
- Demo: no trace. Production: every answer stores query, retrieved IDs, reranker scores, prompt, and model version for replay.
If you can't reconstruct why the model said something six weeks later, you don't have a production system. You have a liability with good UX.
Retrieval quality is 80% of the fight
Hallucination is usually blamed on the LLM. In our experience most "hallucinations" in RAG are retrieval failures wearing a costume: the right passage was never in the context window, so the model improvised. Fix retrieval first and generation problems shrink dramatically.
Chunking is a modeling decision, not a config value
Fixed 512-token windows shred tables, split a clause from its exception, and orphan the heading that gave a paragraph meaning. We chunk structurally - by section, list item, table row - and attach metadata (source, section path, effective date, version) to every chunk. We keep a small overlap and, where the document has hierarchy, store parent context so a retrieved sentence can be expanded to its enclosing section at generation time. Good chunks are self-contained enough that a human reading one in isolation would not be misled.
Embeddings, hybrid search, and the reranker that earns its latency
Dense embeddings capture meaning but miss exact tokens - part numbers, statute references, error codes - that a user often types verbatim. So we run hybrid: dense vectors alongside BM25/keyword, fused with reciprocal rank fusion. Then the single highest-leverage component in the whole pipeline: a cross-encoder reranker. Bi-encoder retrieval is fast and approximate; a reranker reads query and candidate together and reorders them with far better judgement. We retrieve 40-100 candidates cheaply, rerank to the top 5-8, and only those reach the model. This one stage moves grounded-answer accuracy more than swapping the base LLM ever does.
We are deliberately not religious about the vector store. We run Pinecone, Milvus, and Weaviate depending on scale, filtering needs, and where the data has to live. The store matters less than what you put in it and how you rerank what comes out.
Grounding and citation-to-source
A citation that points at a whole document is decoration. Real grounding means every factual sentence resolves to a specific chunk ID and, ideally, a character span within it. We pass chunks to the model with explicit IDs and instruct it to attach the supporting ID to each claim. Then - and this is the part demos skip - we verify the citation post-generation rather than trusting the model's self-report. If a sentence claims support from chunk 7, we check that chunk 7 actually entails it; unsupported sentences get flagged, dropped, or trigger a refusal. Citation becomes a checked contract, not a UI flourish.
def grounded_answer(query: str):
hits = hybrid_search(query, k=80) # dense + BM25, RRF-fused
ranked = reranker.rerank(query, hits)[:6] # cross-encoder
if ranked[0].score < RELEVANCE_FLOOR:
return Refusal(reason="no_supporting_source")
ctx = [{"id": h.id, "text": h.text, "src": h.meta} for h in ranked]
draft = llm.generate(query, context=ctx, cite=True)
# Verify each claim maps to a real, entailing source
checked = verify_citations(draft, ctx)
if checked.unsupported:
return Refusal(reason="unverifiable_claim",
partial=checked.supported)
return Answer(text=checked.text, sources=checked.ids)When to reach for a knowledge graph
Vector search is associative - it finds things that read similarly. It is weak at questions with structure: multi-hop reasoning ("which suppliers are two tiers below a sanctioned entity"), aggregation, and relationships that span many documents. When the value of the answer lives in connections rather than passages, we add a knowledge graph and let the LLM traverse typed entities and edges, using retrieval to ground each node. This is not the default - a graph is real modeling and maintenance cost. We reach for it when questions are relational and precision on those relationships is worth the build. Most corpora do not need one; the ones that do fail badly without it.
Evaluation: two harnesses, not one
You cannot improve what you cannot measure, and "we tried some questions" is not measurement. We evaluate retrieval and generation as separate systems, because they fail for different reasons and a blended score hides both.
Retrieval evals
On a labelled set of query-to-relevant-chunk pairs we track recall@k, MRR, and nDCG. This tells us whether the right evidence is even reaching the model. If recall@k is low, no prompt engineering will save you - the answer isn't in the room. We run this on every index change: new chunker, new embedding model, new reranker.
Generation evals
On top of good retrieval we score faithfulness (is every claim supported by the retrieved context?), answer relevance, and citation correctness. We use LLM-as-judge with rubric prompts, calibrated against human labels so the judge is itself validated, plus adversarial cases designed to bait hallucination - questions the corpus almost but not quite answers. Faithfulness is the number we defend most fiercely; a fluent, relevant, unsupported answer is the exact failure we exist to prevent. These evals live in CI and gate deploys the same way unit tests do.
Guardrails and graceful refusal
The single most underrated feature in production RAG is the ability to say "I don't know." A system that refuses cleanly when evidence is thin is worth more than one that is right 95% of the time and dangerously wrong the other 5% with no signal. We enforce a relevance floor on reranker scores - below it, the system refuses before generation even starts. We check the answer against retrieved context after generation. And we scope refusals honestly: "I can't confirm that from the available documents" beats a fabricated specific every time. Graceful refusal is not a fallback for a weak model; it is a designed behaviour with its own tests.
Latency, cost, and the governance that makes it auditable
Reranking, citation verification, and judging add cost and milliseconds, so we spend deliberately. Cheap embedding retrieval fans out wide; the expensive cross-encoder only touches a shortlist. We cache embeddings and frequent queries, route easy questions to smaller models and hard ones to larger, and run verification concurrently with streaming so the user sees tokens while checks complete. The aim is a system that is fast because it is well-staged, not fast because it skips the checks that keep it honest.
We orchestrate these stages with LangChain and LangGraph - the graph model fits RAG naturally, because retrieve, rerank, generate, verify, and refuse are nodes with explicit edges and retries. We expose tools and data sources through MCP so retrieval and actions are governed behind a consistent, permissioned interface rather than ad-hoc glue. And every request writes a trace: the query, retrieved chunk IDs, reranker scores, the exact prompt, model and index versions, and the final citations. That trace is what turns "the AI said so" into "here is the source, the retrieval path, and the version that produced this answer." It is the difference between a system you hope is right and one you can prove was reasonable.
We don't rent AI. We engineer it - which means every answer our RAG gives is one we can trace back to a source and defend in a room full of skeptics.
None of this is exotic. It is disciplined engineering applied to a stack that rewards shortcuts with confident lies. Chunk with intent, retrieve hybrid, rerank hard, cite to the span, verify the citation, evaluate both halves, refuse gracefully, and trace everything. Do that, and the demo and the audit stop being two different systems. They become the same one.