RAG, fine-tuning or prompts: which one puts an LLM into production

There are three ways to adapt a language model to a company's work: give it better instructions, hand it the company's documents at query time, or retrain it on in-house data. Almost every project we review starts with the most expensive option. The order in which they are tried matters more than the technique, and that order is always the same.

Instructions cover more cases than people expect

The prompt is the set of instructions, examples and output format the model receives with each request. It costs hours, changes without a deployment and can be tested in an afternoon. That is why it comes first, even when technical teams find it unambitious.

In practice, a well-built prompt sets the role, describes the task with a real example, lists the expected fields and forces a strict JSON output. That covers classification, field extraction from one document type and drafting replies from a template. What it cannot do is supply information the model does not have: the product catalogue, the current returns policy, a customer's history.

The limit shows when the prompt grows into a manual. If every edge case adds a paragraph, it becomes impossible to maintain and shifts behaviour with each model release. That is the signal to move to document retrieval, and it usually arrives within the first month of real use.

RAG gives the model the company's data without retraining it

RAG (retrieval-augmented generation) means finding, for each question, the fragments of internal documentation that answer it and passing them to the model as context. The documents stay where they already live (SharePoint, the ERP, a database, a mailbox full of PDFs); they are indexed and queried at request time. The model writes from what was retrieved and can cite where each fact came from.

What changes compared with a prompt is freshness and traceability. If a product price is updated in Business Central today, tomorrow's answer reflects it. If an answer is wrong, the fragment that caused it is visible. Neither exists with a retrained model.

The problem is that RAG inherits the quality of its sources. A wiki holding three versions of the same procedure produces answers with three versions, all written with the same confidence. So most of the work in a RAG project sits in the data layer: deciding which sources are in, who maintains them and how they are split and indexed. The model itself (Claude, GPT, Gemini or an open-weight model inside the company's own Azure or AWS tenancy) is the piece that consumes the fewest hours.

Fine-tuning changes behaviour, not knowledge

Retraining a model on in-house examples adjusts how it responds: tone, format, the way it classifies. It pays off when there are thousands of labelled examples of a stable task and the two previous techniques have fallen short in the evaluation. A typical case is a classification with dozens of sector-specific categories where long prompts never reach the required accuracy.

In return, the knowledge is frozen at the training date and every update means repeating the process, with its GPU cost and data preparation. A retrained model does not cite sources either. Put differently, it is not the tool for making the model "know the company"; that is what RAG is for.

It is best treated as the last resort. In most mid-sized companies the labelled dataset does not exist, and building it costs more than the rest of the project.

The evaluation set decides the outcome

Before anything is changed, a test set is needed: fifty to a hundred real requests with the answer the team considers correct. It gives a baseline and turns each change (a new prompt, a different indexing strategy, another model) into a number rather than an impression. Without it, the debate about techniques is a debate about opinions.

The useful metrics are few: accuracy per request type, the share of answers in which the model admits it has no data, latency and cost per request. A system that answers most cases well and invents the rest is worse than one that scores slightly lower and routes its doubts to a person. The second one can be trusted with a mailbox; the first one cannot.

In production, the system also needs what any software needs: a log of every request and response, handling for errors and API limits, a fallback model and a route for a person to review low-confidence cases. That is where most of the budget goes, and where the project's survival past month two is decided.

Where to start this week

• Choose one task with volume and a verifiable result: triaging incoming email, extracting fields from one document type or drafting the first reply to an enquiry.
• Gather fifty real cases with the correct answer and record the current accuracy of the manual process.
• Try a prompt against a provider's API first and measure it against that set.
• Add retrieval only over documents someone maintains, and write down who.
• Consider fine-tuning only if the evaluation numbers justify it.

If the case fits, we at LakeTab review it in a first 60-minute call.