How I Made Our Internal AI Chat Assistant Stop Confidently Wrong

The problem

We built an AI chat assistant for internal use. Operations, support, and really anyone in the company can ask it about our product. Things like "how does refund work for the annual plan?" or "what does this error code mean?"

The first version was simple. One LLM agent got the question, picked which MCP tool to call, searched the knowledge base, and wrote the answer. The knowledge base had three sources: our official product docs, internal technical pages from Confluence, and work tickets.

It looked great in demos. In real use, not so much. People type short, messy questions, and the agent sent almost the same messy text straight to the search. The search returned pages that were only half related, and the model built its answer from those. The answers sounded fine but were not accurate, missed parts, or skipped the details people actually needed.

There was a second problem. Our knowledge base stored summaries of the docs, not the docs themselves. Summaries drop the small details, and the small details were exactly what Ops kept asking about.

What it should be

Ask a question, get an accurate and complete answer, and see where it came from. If the docs don't have the answer, the assistant should say so instead of making one up.

For that to work, the search has to find the right page first. If retrieval is weak, no prompt can save the answer. That was the big lesson for me.

What I changed

I made three changes:

1. Split the work. A smaller, cheaper model handles intent detection, query writing, and a prompt guard before the main agent does anything.
2. Use the real docs. No more summaries. Store the actual content with as much metadata as possible, and always include the title.
3. Add a reranker. Search wide first, then let a reranker pick the best few.

How the flow works

Here is the full flow. Notice that the small model runs before the main agent touches any tool.

sequenceDiagram
  participant U as User
  participant S as Small model
  participant A as Main agent
  participant T as KB tool
  participant R as Reranker
  U->>S: Question
  S->>S: Guard check, intent, search queries
  alt Blocked by guard
    S-->>U: Polite refusal
  else OK
    S-->>A: Intent and queries
    A->>T: search_kb(queries, source)
    T->>R: Top 30 chunks
    R-->>T: Top 5 chunks
    T-->>A: Chunks with title and URL
    A-->>U: Answer with sources
  end

The small model up front

Before, the main agent did everything: understand the question, write the search query, pick a tool, check the results, and write the answer. That is a lot for one prompt, and the search query was usually the part that got skipped.

Now a small model does the first part and returns plain JSON:

type Route = {
  safe: boolean;
  intent: "product_question" | "ticket_lookup" | "action" | "chitchat";
  queries: string[];
  source: "product_docs" | "confluence" | "tickets" | "all";
};
 
export async function route(question: string): Promise<Route> {
  const res = await smallModel.generate({
    system: ROUTER_PROMPT,
    input: question,
    responseFormat: "json",
  });
  return JSON.parse(res.text) as Route;
}

For a question like "customer wants to cancel annual plan, can refund?", it gives back something like this:

{
  "safe": true,
  "intent": "product_question",
  "queries": [
    "refund policy annual plan",
    "cancel annual subscription refund rules"
  ],
  "source": "product_docs"
}

Those clean queries search much better than the raw question. Since it's a small model, it's also fast and cheap, and I can tune its prompt without touching the main agent.

The guard part checks for prompt injection and off-topic requests. This matters more later, when the tools can change data and not only read it.

Real docs, not summaries

I re-indexed everything from the original docs. Each page is split by heading, and each chunk keeps its metadata:

{
  "title": "Refunds for Annual Plans",
  "section": "Partial refunds",
  "source": "product_docs",
  "url": "https://docs.example.com/billing/refunds#partial",
  "product": "billing",
  "updated_at": "2026-08-14",
  "text": "After 30 days, the refund amount is prorated..."
}

The title is the most important field. A chunk on its own often doesn't say what it's about. "After 30 days, the refund amount is prorated" is useless if you don't know it's about annual plans. So I put the title and section in front of the text before embedding:

const toEmbed = `${doc.title} > ${chunk.section}\n\n${chunk.text}`;

The title and URL also go back to the agent, so every answer can show its source.

Adding a reranker

Vector search is good at finding things that are "close enough", but not always the best match. So now the tool pulls 30 results, the reranker scores each one against the query, and only the top 5 go to the agent.

export async function searchKb(queries: string[], source: string) {
  const hits = await vectorSearch(queries, { source, limit: 30 });
  const unique = dedupeById(hits);
  const ranked = await reranker.rank(queries[0], unique);
  return ranked.slice(0, 5);
}

This was the easiest change and one of the biggest wins. Fewer, better chunks also means a shorter context and a less confused agent.

The result

I tested with real questions taken from the Ops chat. In the old version, most weak answers came from bad retrieval, not from the model being dumb. After the changes, the right doc showed up in the top results much more often, and the answers became more accurate, more complete, and kept the small details.

The assistant also got more honest. When nothing relevant comes back, it says it couldn't find it instead of guessing. People trust it more because they can click the source and check.

Next step: letting it do things

Once reading worked well, I added MCP tools that can create, update, and delete, like creating a ticket or changing its status. But nothing runs without a human saying yes.

This is how an action request flows:

flowchart TD
  A[User asks for a change] --> B["Small model: intent = action"]
  B --> C[Agent prepares tool call]
  C --> D[Show preview to user]
  D --> E{Approve?}
  E -- Yes --> F[Run MCP tool]
  E -- No --> G[Cancel]
  F --> H["Show result and link"]

Each write tool is marked as needing approval:

export const updateTicket = {
  name: "update_ticket",
  description: "Change status or assignee of a work ticket",
  requiresApproval: true,
  run: async (args: { id: string; status?: string; assignee?: string }) => {
    return tickets.update(args.id, args);
  },
};

What I learned

Fix retrieval before you fix the prompt. Give small jobs to a small model. Index the real docs and always keep the title. A reranker is a cheap win. And when the AI can change things, keep a human in the loop.