Building Autonomous Agents with RAG in Production

Most "the agent hallucinated" complaints are actually retrieval failures wearing a model-quality costume. An LLM given the wrong three paragraphs will confidently reason its way to a wrong answer — that's not a reasoning failure, it's a retrieval failure, and no amount of prompt engineering fixes it.

Retrieval quality is the ceiling on everything downstream

A RAG pipeline has exactly one job before the model ever sees a token: find the right passages. If retrieval returns the wrong chunk, a stale chunk, or a technically-relevant-but-context-free fragment, the model is reasoning over bad input — and it will still sound confident, because nothing in the pipeline told it the input was wrong.

Chunking is where most pipelines quietly break

Fixed-size chunking (split every N tokens) is the default in most tutorials, and it's the first thing that breaks on real documents. A 500-token window cuts a table in half, separates a heading from the section it introduces, or splits a code example from the explanation of what it does.

// Naive — chunks on token count, ignoring document structure
function chunkFixed(text: string, size = 500): string[] {
  const tokens = text.split(/\s+/);
  const chunks: string[] = [];
  for (let i = 0; i < tokens.length; i += size) {
    chunks.push(tokens.slice(i, i + size).join(" "));
  }
  return chunks;
}

// Better — chunks on structure (headings, paragraphs), then merges
// undersized pieces so a heading never gets separated from its content
function chunkByStructure(sections: { heading: string; body: string }[]): string[] {
  return sections.map((s) => `${s.heading}\n\n${s.body}`);
}

The second version costs more engineering time up front — it needs a real document parser, not a token counter — but it's the difference between retrieval that works on your test queries and retrieval that works on the queries your users actually ask.

Embeddings and vector search, concretely

An embedding model turns a chunk of text into a vector — a list of numbers positioned so that semantically similar text ends up close together in that vector space. Retrieval is then a nearest-neighbor search: embed the query the same way, and find the chunks whose vectors are closest to it.

import { sql } from "./db";

// pgvector: cosine distance between the query embedding and every
// stored chunk embedding, ordered nearest-first
async function retrieve(queryEmbedding: number[], k = 5) {
  return sql`
    SELECT content, 1 - (embedding <=> ${queryEmbedding}) AS similarity
    FROM document_chunks
    ORDER BY embedding <=> ${queryEmbedding}
    LIMIT ${k}
  `;
}

The part tutorials skip: nearest-neighbor similarity is not the same thing as relevance. A chunk can be semantically close to the query and still be the wrong answer — a query about "canceling a subscription" and a chunk about "pausing a subscription" will score similar, but they're different actions. Production systems add a reranking step, or hybrid search (combining vector similarity with keyword/BM25 search), specifically to catch this.

From retrieval to an actual agent loop

RAG answers a question. An agent does something with the answer — calls a tool, takes an action, checks whether that action succeeded, and decides what to do next. The loop is retrieve, reason, act, observe, repeat, until the model decides it's done or a limit is hit.

async function runAgent(query: string, tools: ToolRegistry, maxSteps = 6) {
  const history: Message[] = [{ role: "user", content: query }];

  for (let step = 0; step < maxSteps; step++) {
    const context = await retrieve(await embed(query));
    const response = await llm.chat({
      messages: [...history, { role: "system", content: formatContext(context) }],
      tools: tools.definitions,
    });

    if (!response.toolCall) return response.content; // model is done

    const result = await tools.execute(response.toolCall);
    history.push({ role: "assistant", content: response.content, toolCall: response.toolCall });
    history.push({ role: "tool", content: result });
  }

  return "Reached max steps without a final answer."; // don't loop forever
}

That maxSteps ceiling isn't defensive pessimism — it's a required guardrail. Without it, a model stuck reasoning in a circle (call a tool, get an ambiguous result, call the same tool again) will happily do that indefinitely, and every one of those iterations is a real API call you're paying for.

Where this actually breaks in production

• Hallucinated tool calls — the model invokes a tool with arguments that look plausible but don't match what the tool actually expects, and the failure surfaces as a confusing runtime error three layers away from the real cause.

• Cost and latency runaway — a loop that should take 2 steps takes 6 because retrieval keeps returning marginally-different-but-not-actually-useful context each iteration.

• Silent retrieval drift — the underlying documents change (a policy update, a pricing change) but embeddings weren't regenerated, so the agent confidently retrieves and reasons over outdated information.

• No fallback for "I don't know" — a model that's never given permission to say the retrieved context doesn't answer the question will construct an answer anyway.

A production-safe default, not a demo default

The version of this that ships to production needs three things a weekend demo usually skips: a hard step ceiling (shown above), structured output validation on every tool call before it executes (reject malformed arguments instead of letting the tool layer throw), and an explicit escalation path — a way for the agent to hand off to a human instead of guessing when retrieved context is thin or contradictory.

An agent that can say "I don't have enough information to do this safely" is more production-ready than one that never fails — because the one that never fails is usually just failing silently.

None of this is exotic. It's the same discipline as any other backend system that calls an unreliable external dependency: validate input, bound the retry/loop count, log enough to debug the failure after the fact, and design the failure path as deliberately as the success path.