Anyone adapting a language model to their own requirements has three tools: a better prompt, retrieval-augmented generation and fine-tuning. They are often presented as a ladder that eventually leads to fine-tuning. That is misleading. The three tools solve different problems, and the right choice depends on which problem you actually have.
The key question: knowledge or behaviour?
If the model lacks knowledge — about your products, contracts, internal processes or current data — RAG is the right tool. The model receives the relevant information with each request. Fine-tuning is a poor tool for knowledge: it anchors facts unreliably, cannot remove outdated information selectively and provides no sources.
If the model does not behave as desired — wrong tone, wrong format, inconsistent structure, wrong decisions on a clearly defined task — the prompt comes first and only then fine-tuning.
Prompting: the first and often last step
Modern models follow detailed instructions remarkably well. A clear task description, explicit rules, a defined output format and a few well-chosen examples solve most behavioural problems. Prompting has decisive advantages: changes take effect immediately, cost nothing but time, can be checked against an evaluation set and survive model changes comparatively well. Before considering fine-tuning, the prompt should be demonstrably exhausted — measured, not felt.
RAG: when the model needs to know something
RAG connects the model with a search over your own content. It suits knowledge bases, documentation, customer records and anything that changes or must be backed by sources. New information is available as soon as it is indexed, deleted information disappears, and every answer can cite its source. Permissions can be enforced during retrieval so each user only gets answers from documents they may see. Quality depends mainly on retrieval, not on the model.
Fine-tuning: when behaviour must be stable at scale
Fine-tuning further trains a model on many examples of input and desired output. It pays off when a narrowly defined task runs very often and the prompt does not solve it reliably or economically. Typical cases are classification into many company-specific categories, a very specific output format, or a style that cannot be achieved stably with examples in the prompt. A fine-tuned small model can then replace a large model with a long prompt and be cheaper and faster.
The cost lies less in training than in data. You need enough clean, representative examples, and they must be maintained. Every new base model generation requires retraining and re-testing. And fine-tuned models are tied to the provider where they were trained.
A decision guide
- Missing facts or current information? RAG.
- Must the answer cite sources or respect permissions? RAG.
- Wrong tone, format or structure? Improve the prompt first and measure.
- Prompt exhausted, task narrow and very frequent? Consider fine-tuning.
- Want to replace a large, expensive model with a small one? Consider fine-tuning the small model.
Combinations are the norm
In practice the tools are combined. A support assistant uses RAG for knowledge about products and tickets and a careful prompt for tone and structure. A fine-tuned classification model routes requests to the right flow beforehand. What matters is using each tool for the problem it solves.
Legal and organisational aspects
Fine-tuning with customer data is separate processing that must be covered by the mandate; in many B2B contracts it is not. Personal data used in training examples can hardly be removed selectively from a trained model. Under the EU AI Act, fine-tuning a general-purpose model does not automatically make you a provider of a general-purpose model; according to the Commission's guidelines that is only the case for a substantial modification with corresponding compute. Your obligations as provider of your own AI system remain unaffected.
Our assessment
In the large majority of projects we see, the right combination is a careful prompt with RAG. Fine-tuning is a valuable tool for narrow, frequent tasks — but rarely the first step. Innopulse decides this question in projects based on an evaluation set, not on trends.