Four Ways to Build RAG Agents in n8n (And When to Skip the Vector Database)

The most common complaint I hear about AI agents sounds like a model problem and almost never is. My agent is not answering correctly, how do I fix it? I am Madhuranjan Kumar, and nine times out of ten the fault is not the model at all. It is the way the agent is fed its data. To show what that actually means, I want to follow one accounting firm through the mess and out the other side, because their journey lays out the four ways to feed an agent and, more usefully, when each one is the right call.

The firm's first mistake: everything into a vector database

The firm did what almost everyone does when they realize their agent needs access to company information. They reached straight for a vector database. It is the default recommendation, the thing every tutorial reaches for, so they loaded their client transaction records and their stack of policy and tax-guidance documents into it, embedded everything as vectors, and pointed the agent at it. On paper it looked complete. In practice it was the source of nearly every wrong answer that followed.

The reason is worth understanding before we watch it break. A vector database works by chopping documents into chunks, embedding each chunk as a vector, and, when a question comes in, pulling back the handful of chunks that look semantically closest. That is a genuinely good design for one specific job and a quietly terrible one for several others. The firm had unknowingly chosen it for the jobs it is bad at.

The month a summary came back wrong

The first crack showed up when a junior accountant asked the agent for a summary of a long policy document. The answer read fine and was badly incomplete. Because the retrieval only pulled a few chunks, the agent summarized those chunks, not the document, and silently dropped everything it had not happened to fetch. Nobody noticed until a partner cross-checked it.

Then came the more dangerous failure. Someone asked which month had the highest billings, with the numbers sitting in a table. The agent grabbed one chunk, found the largest value inside that single chunk, and reported it as the maximum. It was not the true maximum across all the rows, just the biggest number in the fragment it happened to see. Ask for an average and it would average one chunk, not the full set. For an accounting firm, where a confidently wrong number is worse than no number, this was the moment they realized the retrieval method, not the AI, was the problem. Chunk-based search had lost the big picture, and it lost it invisibly, which is the worst way to be wrong.

Filters fix the simple lookups

The turnaround started with the humblest method of the four. When a bookkeeper asked how many invoices a particular client paid in March, the firm stopped routing that through vector search and used a plain filter instead. The agent sets explicit conditions, client equals the name, date falls in March, and pulls only the matching rows, then counts them. It is fast, it is cheap, and it is exactly accurate, because it is doing the same thing a human does when they filter a spreadsheet.

There was one catch the firm had to learn. A filter is explicit matching, not the fuzzy semantic kind. The agent has to be told the valid client names, the exact date format, and the fields it is allowed to filter on, or the filter silently returns nothing and the agent shrugs. Once they wrote those valid options into the agent's system prompt, the simple lookups became boringly reliable, which in accounting is the highest compliment there is.

SQL takes over the rankings and totals

Filters handled the lookups, but the partners kept asking questions that filters could not close. Which five clients generated the most revenue this quarter? What is the average invoice size by service line? These are math across many rows, and making an AI read thousands of rows to compute them is both slow and error-prone. So the firm let the agent write a SQL query and handed the arithmetic to the database.

Ask for the three highest-earning clients and the agent writes a query that sums revenue, groups by client, orders the result descending, and limits it to three. The database was built for exactly this. It does the summing, grouping, and ranking in the query itself, far more reliably than a model eyeballing rows, and far more cheaply than vector search on structured data. As with filters, the firm still had to give the agent the table schema and a couple of example queries, but once it had those, the whole category of totals-and-rankings questions was solved. This is the pattern I reach for whenever the real question is a calculation rather than a lookup.

Full context for the close checklist

Not every question is a number. When a junior accountant asked for a step-by-step summary of the firm's month-end close process, neither filters nor SQL nor vector search was right, because the order of the steps is the entire point and the document was small enough to read whole. So the firm let the agent read the entire document every time, giving it full context rather than hunting for chunks.

This is the method people forget exists, usually because they assume reading a whole document is wasteful. With today's large context windows it rarely is. Reading a full policy or a complete transcript might cost a few thousand tokens against a limit of hundreds of thousands. In exchange the agent gets the complete picture and answers accurately, including anything where sequence matters, like a chronological process or a step-by-step procedure. For the firm's close checklist, full context was the only method that preserved the order, and the token cost was a rounding error against the accuracy it bought.

Vector search finds its real job

Here is the satisfying turn. The firm did not abandon vector search. They just narrowed it to the one job it is genuinely best at. Buried in their large library of tax bulletins were one-off questions whose answer lived in a single passage somewhere. Which bulletin covers a specific niche deduction? For that, chunk-based vector search is the right and cheapest tool, because you have a big body of text and you need one relevant passage out of it, like pulling a single FAQ out of a hundred.

At scale that advantage widens. The bigger the text library, the more clearly vector search beats reading everything. Its one real weakness, that it does not understand order, no longer mattered, because the firm now only used it for find-the-passage questions where order is irrelevant. The tool that had caused all their early pain became genuinely valuable the moment it was pointed at the job it was designed for.

The token bill and what it taught them

The clearest lesson showed up on the token counts, because that is where the wrong method quietly bleeds money. Consider the illustrative cost of three approaches to a query on the same data. Reading a full document into context might run around 6,577 tokens per query. A targeted filter that pulls only the matching rows might cost around 4,000. A vector search that returns just the relevant chunks might cost around 2,600. Those numbers are illustrative rather than exact, but the ranking is the point: pulling in only what you actually need keeps the system cheaper and, just as importantly, reduces hallucination, because the model is not sifting a pile of irrelevant text hoping the answer is in there.

The firm's real gain was not any single number. It was that every question now went to the method built for it. Simple lookups went to filters. Math went to SQL. Order-dependent summaries went to full context. Needle-in-a-haystack lookups went to vector search. The agent stopped being a clever tool that was occasionally, invisibly wrong, and became an assistant the partners actually trusted. That trust is what let them wire it into the CRM and website stack, so client-facing automations could pull the right record and answer without a human double-checking every figure.

Why mixing methods beats forcing one

The deeper lesson from the firm's turnaround is that there is no single best retrieval method, only a best method for a given question. The mistake at the start was not choosing a vector database. It was choosing one tool and forcing every question through it, regardless of shape. A vector database is excellent, genuinely excellent, at finding one relevant passage in a huge pile of text. It is simply the wrong instrument for counting rows, computing an average, or preserving the order of a procedure. Handing it those jobs is like using a screwdriver to hammer a nail. It sort of works, badly, and you blame the screwdriver.

Most real businesses need a blend, because most real businesses ask questions of several different shapes. The accounting firm ended up running all four methods side by side, each pointed at the question type it handles best, and the agent simply chose the right one based on what was being asked. That is the mature setup, and it is worth designing toward from the start rather than discovering the hard way through a month of wrong answers. The instinct to reach for one universal solution is understandable, because a single tool feels simpler to maintain. But the simplicity is false. A system that uses the wrong method for half its questions is not simple, it is quietly broken, and it costs you both money on wasted tokens and trust on wrong answers. The genuinely simple thing, once you accept it, is to match the tool to the job.

The rule of thumb the firm now lives by

If you strip away the jargon, the firm ended up with a beginner-friendly rule that decides the method every time: ask what a human would do. If a person would filter a spreadsheet, use a filter. If they would build a pivot table, use SQL. If they would read the whole document, give full context. If they would search a big library for one passage, use vectors. That single question routes almost any real business need to the correct retrieval method without any deep technical knowledge.

Underneath it all, this is really context engineering. An AI is only as smart as the data you hand it, and the entire craft is starting from the questions rather than the database. Write down what the agent will be asked and what it must look at to answer well, then match each pattern to a method. Always tell the agent its valid options, the product or client names, the date formats, the table schema, because explicit retrieval is not forgiving. And keep an eye on the token counts, since the cheapest correct method is almost always the one that pulls in the least irrelevant text.

This discipline pays off well beyond an accounting back office. A support team with a hundred help articles wants vector search to surface the one relevant answer, and that same clean knowledge base doubles as fuel for SEO and organic search when those articles are published. A marketing operation that needs monthly performance rankings across its Facebook and Instagram ad campaigns wants SQL, not a model guessing at totals. The method changes with the question, but the habit is constant: choose the retrieval that a competent human would choose, and the agent inherits their accuracy.

You can absolutely learn to make these calls and build these agents yourself, and the rule of thumb above will carry you most of the way. If you would rather have someone design the data pipeline so every question your agent gets routes to a fast, correct answer without the trial and error the firm went through, that is exactly the kind of build I take on.