“Can we have a ChatGPT that knows our products, policies and documentation?” It is one of the most common questions businesses ask about AI. The answer is yes, but usually not by “training” a model. The standard approach is retrieval-augmented generation (RAG).
This guide explains how RAG works, how to build a chatbot on your own data step by step, and the mistakes that make many of these projects disappointing.
Training vs RAG: what is the difference?
Fine-tuning (training) changes the model’s behaviour by showing it many examples. It is good for style, format and specialised tasks. It is not a good way to teach facts that change: prices, stock, policies, documentation.
RAG keeps your information outside the model. When a user asks a question:
- The system searches your content for the most relevant passages
- It gives those passages to the language model together with the question
- The model writes an answer based on those passages, ideally with source references
Advantages of RAG:
- Update content without retraining
- Answers can cite sources
- You control which information is available
- Usually much cheaper and faster to build
For most business chatbots, RAG is the right starting point.
The architecture of a RAG chatbot
| Component | What it does |
|---|---|
| Data sources | Help center, PDFs, product data, policies, internal wikis |
| Ingestion pipeline | Loads, cleans and splits content into chunks |
| Embedding model | Turns each chunk into a vector (a numeric representation of meaning) |
| Vector database | Stores vectors and finds similar ones quickly (for example pgvector, Qdrant, Weaviate, Pinecone) |
| Retriever | Finds relevant chunks for a question, often combining vector and keyword search |
| Language model | Writes the answer from the retrieved context |
| Tools / APIs | Fetch live data (orders, stock, account information) |
| Interface | Website chat, helpdesk, Slack, internal tool |
Step 1: Define the use case
Be specific. “A chatbot for everything” fails. “Answer customer questions about shipping, returns and product compatibility in Dutch and English” succeeds. Define what it should answer, what it must never answer and when it hands over to a human.
Step 2: Prepare the data
This is where most of the quality comes from.
- Remove outdated and contradictory content
- Prefer structured, clearly written sources
- Keep metadata: title, URL, language, product, date
- Convert PDFs carefully (tables and columns often break)
Step 3: Chunking
Long documents are split into smaller chunks. Too small and context is lost; too large and retrieval becomes imprecise. Split along headings and sections where possible, and include the document title in each chunk.
Step 4: Retrieval
Pure vector search is good at meaning but weak at exact terms like product codes and error numbers. Hybrid search (vector + keyword) plus a reranking step usually gives much better results. Filter by metadata, for example language or product line, before searching.
Step 5: Prompt and guardrails
Instruct the model to:
- Answer only from the provided context
- Say clearly when it does not know
- Cite sources
- Stay within its scope and tone
Step 6: Add tools for live data
For questions like “Where is my order?”, documents are not enough. Give the assistant tools (function calling) that call your APIs, with strict authentication and permissions. Standards like the Model Context Protocol (MCP) make connecting tools easier; I wrote about MCP in 2026.
Step 7: Evaluate before launch
Create a test set of real questions with expected answers. Measure:
- Did retrieval find the right source?
- Is the answer correct and complete?
- Did it refuse correctly when it should?
Run this test after every change. Without evaluation, you are guessing.
Step 8: Launch and improve
Log conversations (with privacy in mind), review failures weekly and fix the source content. Most improvements come from better data, not a bigger model.
Common mistakes
- Dumping all documents in without cleaning. Contradictory sources produce contradictory answers.
- No evaluation. Quality is judged by a few demo questions.
- No handover. Users get stuck when the bot does not know.
- Ignoring permissions. Internal documents end up available to customers.
- Only vector search. Exact product codes and names are missed.
Costs
Costs come from:
- Development and data preparation (usually the largest part)
- Language model usage per conversation
- Embeddings (mostly one-off, plus updates)
- Hosting of the vector database and application
Model prices have dropped significantly over time, and choosing a smaller model for simple questions keeps running costs low.
Privacy and security
- Choose AI providers with suitable data processing agreements
- Do not send unnecessary personal data to the model
- Protect against prompt injection, especially when the bot can use tools
- Inform users they are talking to an AI
Frequently asked questions
Can I just upload my documents to ChatGPT?
For personal or small internal use, built-in file features can be enough. For customer-facing use with live data, access control, branding and evaluation, a custom RAG solution is more reliable.
Do I need a vector database?
For small knowledge bases, a database extension like pgvector is often enough. Dedicated vector databases help at larger scale.
How accurate are RAG chatbots?
With good data, hybrid retrieval, evaluation and clear limits, they can be very reliable within their scope. Accuracy drops quickly outside the scope, which is why handover matters.
Want a chatbot that really knows your business?
I design and build RAG assistants and AI integrations on your own data, from customer support to internal knowledge. See my AI integration service or request a consultation.