RAG vs fine-tuning: a plain guide for business leaders
Most "make AI know our business" projects are knowledge problems, and the fix starts with your documents.
BY STEFAN MARITZ··6 MIN READ
TL;DR
- RAG (retrieval-augmented generation) gives a model the right information at the moment it answers, by looking it up in your documents. Fine-tuning changes how a model behaves, by training it further on examples. They solve different problems.
- Use RAG when the model lacks knowledge that was not in its training, is out of date, or is proprietary. Use fine-tuning when the model knows enough but keeps getting the format, tone, or reasoning wrong.
- Most business problems we see are knowledge problems, and many of OpenAI's largest customer deployments used only prompt engineering and RAG.
- A small knowledge base may not need RAG at all. Below roughly 500 pages of material, the whole knowledge base can go straight into the prompt.
- Start with a clear prompt and a test set of 20 or more real questions with known answers. The failures on that test set tell you which lever to pull next.
When a leadership team asks us to make AI "know our business", the conversation turns technical quickly. Someone asks whether they need RAG or fine-tuning, and often nobody in the room is sure what either term commits them to, in cost or in effort.
This guide explains both in business terms, shows which problem each one solves, and gives you a way to decide before anyone writes code.
What is RAG?
Retrieval-augmented generation, first set out in a 2020 research paper, is a way of giving a model information it does not have. When someone asks a question, the system searches your documents, pulls out the most relevant passages, and adds them to the prompt before the model answers.
Behind the scenes, your documents are split into short chunks and converted into embeddings, which are numerical representations of meaning. These are stored in a vector database, a database built to search by meaning. At question time, the most relevant chunks are found and added to the prompt.
The practical strengths for a business are traceability and upkeep. Answers can point to the document they came from, and you update the knowledge by updating the documents, with no retraining.
What is fine-tuning?
Fine-tuning means training an existing model further on a set of examples: the inputs you give it and the outputs you want back. It shapes behaviour, such as the format of a report, the tone of a reply, or the steps it follows. A typical starting point is a dataset of 50 or more examples.
It is the right tool when the model has the knowledge it needs and keeps getting the shape of the answer wrong. It carries a cost that is easy to miss: a fine-tuned model is tied to the base model it was trained from. When a better model arrives, you retrain. That matters, because the leading model for a task changes often.
The difference, in operational terms
Think of a capable new hire in your finance team. Give them the procedures manual and access to the shared drive, and they can answer most questions correctly on day one by looking things up. That is RAG. Spend three months showing them how your team writes board reports, and they produce them in the house style without being told. That is fine-tuning. Most new hires need the manual first.
| Question | RAG | Fine-tuning |
|---|---|---|
| What it changes | What the model knows when it answers | How the model behaves |
| Best for | Policies, product details, contracts, anything proprietary or changing | Consistent format, house tone, a fixed reasoning pattern |
| How you update it | Edit or add documents | Prepare new examples and retrain |
| Can it show sources? | Yes, it can point to the passages it used | No |
| Main risk | Retrieving the wrong passage | Tying behaviour to one model version |
| First step | Organise the documents people rely on | Collect 50 or more good input and output examples |
The two can work together. They stack, and some use cases need both: a system that retrieves the right policy and answers in a fixed format every time.
When you may need neither
A clear prompt solves more than people expect. Prompt engineering alone is often enough for summarisation, translation, and code generation.
Size matters too. A token is roughly three-quarters of a word, and if your knowledge base is under about 200,000 tokens, around 500 pages, it can go directly into the prompt. Very long prompts have a known weakness, where models lose track of information in the middle, so test with your real material before relying on it.
Why RAG projects succeed or fail
A common failure happens at retrieval, before the model writes a word. If the system finds the wrong passage, the answer will be confident and wrong. Retrieval can improve a great deal: adding context to each chunk before indexing cut failed retrievals by 49%, and by 67% with a reranking step.
The bigger lever for most organisations sits upstream. A model can only work with what your systems give it. If the refund policy exists in three versions on three drives, no technique will choose the right one reliably. This is why we start with a company brain, an organised record of how the business works, before choosing any retrieval method. It is also why data and systems sit alongside strategy and people in an AI readiness assessment.
How to decide between RAG and fine-tuning
- Write down the cost of a wrong answer. A first draft someone fixes is a different problem from refunding a customer $1000 instead of $100. The cost tells you how much accuracy you need, and whether the investment returns anything.
- Build a test set of 20 or more real questions with known correct answers, written by the people who do the work.
- Start with a clear prompt and score it against the test set.
- Diagnose each failure. Missing or outdated information points to RAG. Wrong format, tone, or steps points to better examples in the prompt, then fine-tuning if that is not enough.
- Fix the documents before the model. If the answer is not written down, or is written down three ways, start there.
- Name an owner for the knowledge. Someone has to keep the documents current and retire old versions, or the system will quote last year's price list with complete confidence, and people will stop trusting it.
Frequently asked questions
What is the difference between RAG and fine-tuning?
RAG looks up relevant information in your documents when a question is asked and adds it to the prompt. Fine-tuning trains the model on examples so it behaves in a particular way. RAG changes what the model knows when it answers, and fine-tuning changes how it responds.
When should I use RAG instead of fine-tuning?
When the model is missing information, such as your policies, products, contracts, or anything that changes often. RAG is also easier to keep current, because you edit documents and leave the model alone.
When is fine-tuning worth it?
When the model already has the knowledge but keeps getting the format, tone, or steps wrong, and better prompts and examples have not fixed it. You need a solid set of good examples, often 50 or more, to start.
Can you use RAG and fine-tuning together?
Yes. They solve different problems and stack, so a system can retrieve the right documents and use a fine-tuned model to answer in a consistent format.
Does a mid-sized business need RAG?
Often not at first. If the material fits in about 500 pages it can go straight into the prompt, and a well-organised set of documents matters more than the technique you choose.
If you would rather have a knowledge layer like this designed and built with you, that is the work of our custom AI-native operating systems service.
WORK WITH PRAXES
Ready to move from reading about AI to getting your organisation ready for it? Start with the diagnostic.
Take the AI Readiness Diagnostic