RAG vs Fine-Tuning for Business Documents

You want an AI assistant that answers from your own documents: policies, contracts, manuals, product specs. The question that comes up first is usually "should we fine-tune a model on our data?" For business documents the answer is almost always no. Here is why, what to do instead, and the one case where fine-tuning still earns its place.

The short answer

• Your documents fit in about 500 pages? Put them in the prompt. No retrieval system needed.
• More than that, or they change often? Use retrieval (RAG): search the documents for each question and give the model only the relevant passages.
• You need the model to behave differently (a strict output format, a classification it keeps getting wrong)? That is what fine-tuning is for, and today it is often not available anyway. Structured outputs and better prompts solve most of these cases.

The rest of this guide explains each branch.

What RAG and fine-tuning actually do

Retrieval-augmented generation (RAG) was introduced by Lewis and colleagues in a 2020 paper that combined a model's built-in knowledge with a searchable collection of documents. In practice: when a question comes in, you search your documents, take the most relevant passages, and put them into the prompt next to the question. The model answers from what it was just given, and you know exactly which passages it saw.

Fine-tuning trains an existing model further on examples of inputs and the outputs you want. It changes the model itself. OpenAI's documentation describes it as a way to "consistently format responses in a certain way or handle novel inputs", and lists classification, nuanced translation, generating content in a specific format and correcting instruction-following failures as its best uses.

Notice what is missing from that list: teaching the model your facts. Fine-tuning shapes how a model responds. Retrieval controls what it knows at the moment it answers.

Option zero: put the documents in the prompt

Before building anything, check the size of your documents. Anthropic's guidance is direct: "If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG or similar methods."

The cost of resending the same documents with every question is lower than it sounds. With prompt caching, the unchanged part of the prompt is reused between requests; Anthropic reports this reduces latency by more than 2x and costs by up to 90%.

This is the simplest system that can work, and you should rule it out before choosing anything more complex.

Why RAG usually wins for business documents

When the documents don't fit, retrieval has four practical advantages over fine-tuning for business knowledge:

1. Documents change. Update a policy and you re-index one file. A fine-tuned model would need retraining to forget the old version.
2. Answers can cite sources. Because you know which passages the model received, you can show them next to the answer so people can check it.
3. Access control stays in your hands. You can filter what the search returns by who is asking, so a user only gets answers from documents they are allowed to see.
4. It works with the newest models. Retrieval sits in front of any model, so switching or upgrading doesn't mean rebuilding anything.

Where fine-tuning fits, and what changed

Fine-tuning still has real uses: a very specific output style, a classification task with many examples, or a smaller and cheaper model that has learned one narrow job. But access to it on the major hosted models has shrunk:

• OpenAI is winding down its fine-tuning platform. Its documentation states the platform "is no longer accessible to new users", and its deprecations page says existing customers will no longer be able to create new fine-tuning jobs from January 6, 2027.
• Claude. Anthropic offered fine-tuning of Claude 3 Haiku through Amazon Bedrock from 2024. That model has since reached end of life on Bedrock, and AWS notes that once a model is in its legacy state, new fine-tuning jobs can't be created on it.

Even where fine-tuning is available, OpenAI's own advice is to start elsewhere: "The prompt engineering process may be all you need", and "Only invest in fine-tuning after setting up evals." For most business teams, prompting plus structured outputs validated in code handles the format and behaviour problems fine-tuning used to solve.

Making retrieval work well

A basic RAG system is easy to build and easy to get wrong. The usual failure is not the model: it is the search step returning the wrong passages. What helps most:

• Split documents sensibly. Chunks should follow the document's structure (sections, clauses, articles), not arbitrary character counts.
• Give each chunk its context. A clause that says "the limit is 30 days" is useless without knowing which policy it belongs to. Anthropic's Contextual Retrieval research adds a short explanation of where each chunk sits before indexing it. Combined with keyword search (BM25), this "reduced the top-20-chunk retrieval failure rate by 49%", and by 67% when a reranking step was added.
• Combine keyword and semantic search. Exact terms (product codes, clause numbers) are found by keyword search; paraphrased questions by embeddings. Use both.
• Measure with real questions. Collect questions your team actually asks, note which document should answer each one, and check whether the search finds it before you judge the answers.

Decision table

Checklist

• Estimate the size of your document set before choosing an approach
• If it fits, try the whole set in the prompt with caching first
• Split documents along their own structure
• Add context to each chunk before indexing
• Use keyword and semantic search together
• Filter retrieval by the user's permissions
• Show sources next to every answer
• Test the search step with real questions before judging answers
• Only consider fine-tuning for behaviour, not knowledge, and only after evals

Related service: AI agents and LLM apps →