What is the difference between a chatbot and the AI system behind it?

A chatbot is the conversation interface. It lets someone ask a question in text or voice. Behind that interface there may be a language model, a search service, business systems, access controls and a record of what happened. A polished conversation tells you very little about whether those parts are dependable.

Kevin’s central point is that the difficult engineering sits behind the chat window. At CSM, we start by defining the job: who is asking, what information they are entitled to see, what constitutes a supported answer and what the business should do if that answer is unavailable. Those decisions determine the system we build.

What is retrieval-augmented generation, or RAG?

RAG retrieves relevant information and supplies it to a language model as context for its answer. For example, an internal assistant might find passages in an approved service manual before explaining a maintenance procedure. Retrieval supplies evidence at the time of the question; it is different from retraining the model on the documents. Microsoft’s RAG overview explains this architecture.

A useful way to follow the process is: identify the question and user; search the permitted sources; select relevant passages; generate an answer; check whether the answer is supported; and show the evidence or escalate. A failure at any stage can produce an unhelpful result even when the model writes fluently.

Why can an answer still be wrong when the documents are correct?

The system can retrieve the wrong document, miss a qualifying clause or combine incompatible versions. It might find a product warranty for the wrong territory. The model can also add a plausible statement that the supplied evidence does not support. NIST identifies confidently presented false content, or confabulation, as a generative AI risk in its Generative AI Profile.

A source link is useful, but its presence is not proof. A reviewer should be able to open the cited passage and see how it supports the specific claim. A document that mentions a warranty is not necessarily evidence for the duration, exclusions or territory stated in the answer.

Can the system say “I don’t know”?

Yes. The spoken discussion stresses the danger of unsupported answers, but models and applications can be designed to abstain. The written guidance here is more precise: neither RAG nor guardrails guarantee 100% accuracy, and an instruction to admit uncertainty is not sufficient on its own.

We would define explicit stop conditions. Missing product identity, conflicting policy versions, an unavailable approved source or an out-of-scope question should trigger a clarification or handover. The fallback should tell the user what is missing and how to resolve it. A model’s confident tone or self-reported confidence should not decide whether it is safe to proceed.

What would this look like for a warranty enquiry?

Consider an illustrative enquiry: “Is this unit covered for five years in Germany?” First establish the product, purchase date, market and relevant agreement. Retrieve the applicable approved warranty version. Check that the passage covers all the requested conditions. If the country-specific annex is missing, do not borrow terms from a different country.

A useful response might say that the available document does not establish the German terms and route the enquiry to the responsible team with the question and evidence attached. That is a completed, controlled handover. It should not be counted as a failed interaction merely because the chatbot did not invent a final answer.

Which controls matter most?

Start with a maintained knowledge set: named owners, approval status, effective dates and a process for retiring superseded material. Preserve the connection between extracted text and its original document. Where figures or eligibility rules have an authoritative structured source, retrieve them from that source rather than asking the model to infer them from prose.

Apply permissions before information enters the answer context. An employee must not gain access to confidential documents through search that they could not read directly. Microsoft describes query-time document permission checks in its document-level access guidance.

Treat retrieved content as evidence, not as authority to change system instructions. A malicious document may contain directions to disclose information or ignore restrictions. Keep tool permissions narrow and test these attacks. The OWASP RAG security guidance is a useful engineering reference. Security controls reduce risk; they do not make every retrieved passage trustworthy.

How should leaders test a chatbot before launch?

Build a test set from real enquiries with answers and evidence checked by knowledgeable staff. Include common questions, ambiguous wording, missing evidence, outdated documents, conflicting sources and attempts to access another user’s information. Hold some cases back from development so the final evaluation is not simply testing examples the team has repeatedly tuned against.

Assess retrieval and answering separately. Did it find the necessary evidence? Did it answer only what the evidence supports? Did it withhold restricted material? Did it escalate the right cases? Track unsupported claims, missed critical evidence, incorrect refusals, review time and resolution time. Aggregate accuracy alone can hide a small number of serious failures.

What is CSM’s approach to a dependable first release?

Choose a bounded knowledge domain, a responsible owner and a clear support route. Test internally, then introduce controlled use with monitoring and a rollback plan. Recheck performance when documents, retrieval settings or models change. Someone must own unresolved questions and decide whether they expose a knowledge gap, a design defect or a genuine exception.

Our aim is to reduce the effort needed to reach a supported answer. A chatbot that answers everything is a poor objective. A system that helps people find evidence, understand its limits and move the work forward is much more useful.

What should I read next?

Explore how to connect business data for AI and how software controls limit AI responsibility.