How To Build A RAG App That Is Actually Reliable, Not Just A Demo

The morning the chatbot embarrassed the whole agency

I want to walk you through a build that started with a small disaster. Picture a mid sized insurance agency, the kind with a dozen producers and a service team that answers a steady stream of client calls all day. The owner had watched a weekend of tutorials and decided the agency needed a bot that could read its own policy documents and answer questions instantly. A junior team member wired one up over two evenings. It chatted with a single policy PDF beautifully in the demo, everyone clapped, and it went live on the internal help page that Friday.

By Monday it had already told a producer that a homeowners policy covered water backup when it plainly did not. The producer, trusting the tool, nearly repeated that to a client on the phone. That is the exact gap I keep coming back to. Standing up a bot that chats with one clean PDF is genuinely easy, and that ease is a trap. The distance between a bot that demos well and a bot that people can lean on in front of a paying customer is enormous, and almost the entire agency had assumed there was no distance at all. I am Madhuranjan Kumar, and this is the story of how that same agency rebuilt the thing into something its producers now actually trust, and the exact order we did it in.

The problem the owner walked in with

When the owner described what was happening, the symptoms all sounded like the bot was making things up at random. It answered some questions perfectly and then confidently fabricated others, with no obvious pattern. The instinct in the room was to blame the model and go shopping for a bigger, smarter one. That instinct is almost always wrong, and it was wrong here.

The real culprit was hiding at the very first step of the pipeline, long before the model ever saw a word. The agency ran on dense, ugly documents. Policy wordings, endorsements, underwriting guidelines, and carrier rulebooks, all full of tables, footnotes, columns, and fine print. The quick build had loaded those PDFs with a naive extractor, and that extractor was quietly scrambling the tables. A coverage limit that read as fifty thousand dollars in the document was landing in the system attached to the wrong peril, or with the number sheared off entirely. The model was not hallucinating so much as faithfully answering from garbage it had been handed. Once you see that, the whole project reframes. The problem was never intelligence. It was messy real world data going in raw, and no amount of prompt tuning fixes bad source text.

The first fix that paid for itself: parsing the mess into clean text

So we started where the failures started. Before touching anything clever, we replaced the crude loader with proper document parsing. A tool built for this, LlamaParse, converts a chaotic PDF into clean, consistent markdown, and its table handling is in a different league from a naive extractor. Even better, you can tell it what kind of document it is looking at, so it reconstructs the structure correctly instead of guessing. We ran every policy form and carrier guide through it and then, and this part matters, a human on the team actually read a sample of the outputs by hand to confirm the numbers and tables had survived the trip.

The agency also had knowledge trapped on carrier websites, current rate notes, eligibility rules, and bulletins that lived only online. For those we used Firecrawl, which turns a web page into clean markdown with its title, images, and metadata intact. The quiet win here is unification. Once both the internal files and the external pages come out in the same clean markdown format, you only have to optimize one pipeline instead of two. This single stage, just fixing the parsing, took the bot from confidently wrong on table heavy questions to reliably reading the right numbers. It was the cheapest fix in the whole project and it recovered the most trust. If you take one thing from this walkthrough, let it be that you fix your data before you touch anything fancy.

Learning to slice documents the right way

Clean text still has to be broken into pieces before it can be searched, and this is where the second round of tuning happened. You split the documents into chunks and store them so the system can pull back the relevant ones. The size of those chunks is a genuine tradeoff, not a setting you can copy from a blog post.

Make the chunks too large and you run into the lost in the middle problem, where a model reading a long stretch of text pays close attention to the beginning and the end and quietly ignores the material in the center. A critical exclusion buried in the middle of an oversized chunk can simply be skipped. Make the chunks too small and each piece loses the context that gives it meaning, so a clause about a coverage limit arrives without the surrounding sentence that says which coverage it belongs to. There is no universal correct size. The honest, scientific way to settle it is experiments. The team assembled a small set of real producer questions where they already knew the correct answers, then tried a few chunk sizes and measured which one got the most right. For these dense policy documents, a mid sized chunk with a little overlap won, because a clause needs enough neighbors around it to mean the right thing.

Teaching retrieval to find the exact clause, not the vaguely similar one

With clean, well sized chunks in place, the next weakness showed up in how the system searched. The default approach, pure semantic vector search, is great at finding text that means something similar to the question. Insurance, though, is stuffed with exact identifiers. Form numbers, named endorsements, specific dollar limits, and product codes that a semantic search will happily gloss over because it is chasing meaning, not exact strings.

The answer was hybrid search, running a keyword search right alongside the semantic one and merging the results. When a producer asks about a specific endorsement by its form number, the keyword side catches the exact match that semantic search alone would miss, and when they ask a fuzzier question about intent, the semantic side carries it. On top of that we added a reranker. Vector search returns its top handful of chunks in a rough order that does not reflect true relevance, so a reranking model reads that shortlist and reorders it, pushing the genuinely relevant clauses to the front and dropping the near misses. That does two things at once. It puts the best evidence directly in front of the model, and it trims the noise so fewer irrelevant tokens get sent along to confuse the answer. Between hybrid search catching exact matches and reranking sorting for real relevance, the retrieved evidence went from roughly right to precisely right.

The agent that grades its own work before it speaks

The final layer is what turned a good tool into one producers stopped double checking. Rather than blindly answering from whatever it retrieved, the system got an agent loop that behaves a bit like a careful employee. When a question comes in, the agent can rewrite a narrow, awkwardly phrased query into a more searchable one, a step back technique that turns something overly specific into a broader question that retrieval handles better. It can split a compound question into separate sub questions and search each one. Then, crucially, it grades whether the documents it pulled back are actually relevant to the question at all.

If the grade comes back weak, the agent does not shrug and answer anyway. It falls back to a web search to fill the gap, then checks its own draft answer for hallucination, making sure the response is genuinely supported by the evidence before it ever reaches the producer. This whole corrective loop can run on a local model, which mattered to the agency because policy and client data never had to leave their own machines. The behavior change was obvious in daily use. When a producer asked whether a particular policy covered a specific situation, the bot either answered with the exact clause behind it or admitted it was not sure, instead of inventing a confident, wrong reply. That honesty is the entire point. A tool that says I am not certain, check the form is infinitely more useful than one that misquotes a policy while a client waits on the line.

What the numbers looked like after ninety days

Here is the shape of the change, and I want to be clear these figures are illustrative of the kind of movement this rebuild produces, not a specific client's audited stat. In the naive first version, the team's honest read was that only around four in ten answers were trustworthy enough to act on without checking, which is why nobody trusted it. After the parsing fix alone, that jumped to roughly seven in ten, because the model was finally reading correct numbers. Once the full loop, hybrid search, reranking, and the self grading agent, was in place, trustworthy answers settled at around nine in ten, which is the threshold where people stop reflexively double checking and start relying on the tool.

The downstream effects were the real return. Service staff spent less time hunting through carrier portals for a single clause. Newer producers got senior level answers on coverage questions in seconds instead of interrupting a colleague. And the near miss that started the whole project, a producer almost quoting the wrong coverage to a client, stopped being a risk the owner lay awake over. A reliable internal knowledge tool is not flashy, but the compounding time it saves across a whole team every single day is exactly the kind of quiet leverage that pays for itself.

There was one more shift the owner did not expect. Because the bot now cited the exact clause and source behind every answer, it slowly became a training tool as well as an answer engine. New hires learned the shape of the policies by watching where the answers came from, and senior producers started trusting it enough to send it into client conversations, reading the cited clause aloud rather than promising to call back. Trust, once it crosses that line, changes behavior across the whole team, and that behavioral change is where the durable value of a reliable RAG build actually lives. A demo impresses a room for five minutes. A tool people cite to a client is one they will use for years.

The order that made it work, and where a build like this fits

If you want to attempt this yourself, copy the sequence, not just the shopping list, because the order is the lesson. Fix parsing first and read a sample of the output by hand. Then tune chunk sizes against a small set of questions you already know the answers to. Add reranking and hybrid search only once basic retrieval is solid. Layer in the self checking agent loop last. Reliable RAG is never one trick. It is clean data, plus smart retrieval, plus a loop that catches its own mistakes, applied in that order so each fix stands on a stable foundation.

As an agency, I spend most of my days helping owners turn attention into revenue through Facebook and Instagram ad campaigns and Google Ads, and a build like this fits right into that world, because a bot that answers instantly and correctly is only worth as much as the qualified traffic and clean CRM and website stack feeding people into it. You can absolutely build this in the order above, and I would push any owner to start by fixing parsing this week. If you would rather have someone parse your real documents, run the chunking experiments, wire up hybrid search and reranking, and stand up the self correcting agent so the answers hold up in front of clients, that is exactly the kind of build I take on, and you are welcome to bring me in to handle it end to end.