A conversation you know
"We'll roll out RAG, the system will get smarter"
IT team: "We'll roll out RAG. The AI will answer based on our own documents."
CEO: "Great. And what exactly does that mean?"
IT team: "Well… the system will be smarter."
The budget is approved. Six months later the system dazzles in demos, but in daily work it confidently quotes two-year-old prices to customers. Legal asks where a particular answer actually came from — and nobody can point to it.
This article is about what went wrong — and why it was not the technology's fault, but the absence of a strategy. More precisely: about confusing one with the other.
The metaphor
The brilliant new hire and the company archive
An AI model is a brilliant new employee on day one. Graduated from the best universities in the world, speaks every language, calculates faster than your entire analytics team. But it does not know your company. It does not know what you call your products, what the board decided in March, or which customers have individually negotiated discounts.
RAG — Retrieval-Augmented Generation. In plain terms: a technique that lets an AI model look into designated company documents before answering, instead of relying solely on general knowledge. In other words: we hand the new hire a key to the company archive rather than sending them on a months-long training course. We will come back to that distinction.
It sounds like a solution. Often it is. But this is the moment when a question appears that most projects never ask: what condition is that archive in?
If it holds three contradictory versions of the price list, decks from 2019 and undated notes, the brilliant new hire will start quoting junk with full confidence. Not because they are stupid. Because nobody told them which documents are current and which ended up there by accident.
Hence the thesis of this piece: RAG is a cabinet full of documents. Strategy is the decision about what is in it, who keeps it in order, and who holds the key.
A shift in the balance of power
Why "let's buy a better model" stopped being the answer
Not long ago only a handful of giants could build a frontier AI model — training a leading model was estimated at tens, even hundreds of millions of dollars. Today the picture looks different: China's DeepSeek built a model of comparable class for a fraction of that.
Item | Order of magnitude | What it changes |
|---|---|---|
Training a frontier model — until recently | ~USD 100M | Entry barrier for a handful of players |
Training DeepSeek R1 (reported run cost) | ~USD 5.6M | The barrier drops by an order of magnitude |
Access to a frontier-class model for your company | a subscription | The same "brain" for you and your competitor |
Five and a half million dollars is — for scale — the price of a mid-sized logistics warehouse outside a large city. The conclusion for the board fits in one sentence: if you and your competitor can subscribe to exactly the same "brain", the brain has stopped being a competitive advantage.
It is a bit like spreadsheets in the nineties. Nobody won a market "because they had Excel". The winners were those who knew what to put into it.
What remains an advantage is the one thing no AI vendor can sell to your competitor: your company's data, its processes, its knowledge of customers. On one condition — that they are in a usable state. And that is where the real work begins.
The tempting shortcut
"Why not just fine-tune the model?" — researchers tested exactly that
At this point many companies float a tempting proposal: since the off-the-shelf model does not know our company, let us train it on our data. Let it learn once and for all.
It sounds sensible. Microsoft researchers decided to check it — they compared head to head two ways of giving a model new knowledge: fine-tuning (learning the facts by heart) and supplying context (RAG, handing over the right documents right before the answer).
The result was not a narrow points decision. It was a knockout. On a test of knowledge about recent events, the model with well-supplied context scored more than twice as high as the same model after fine-tuning. What is more, an ordinary model with good RAG beat even the specially fine-tuned one.
How the knowledge was delivered | Accuracy | Real-world equivalent |
|---|---|---|
Fine-tuning the model | 0.50 | Weeks of memorising facts by heart |
Supplying context (RAG) | 0.88 | Three relevant pages handed over before going on stage |
Test of knowledge about recent events. Source: Ovadia, Brief, Mishaeli, Elisha (Microsoft),
Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs, EMNLP 2024,
arxiv.org/abs/2312.05934.
Back to our brilliant new hire. It is like comparing two approaches before an important talk: you can make a professor memorise every fact about your company — weeks of tedious study, preparing materials and revising, and they will still get something wrong. Or you can hand them, right before they walk on stage, three well-chosen pages with what actually matters. The research is unambiguous: well-chosen pages beat memorisation.
One honest caveat. Fine-tuning still makes sense for style — tone of voice, format, manner. But as a way of transferring knowledge about your company it is expensive and unreliable. Which raises the stakes on the quality of what you supply. Which brings us back to the state of the archive.
Two stories from the market
The bank that won with a library. The fintech that lost on costs.
| Morgan Stanley — the example | Klarna — the counter-example |
|---|---|---|
Starting point | ~350k investment research documents. The model was taken off the shelf; the work went into putting the archive in order. | A public declaration: AI replaces customer service. Cuts, savings, headlines. |
What was settled before launch | Which documents are current, who owns their freshness, who may access what, how to check the source of every answer. | Nothing was settled about what the system does not know and when it must hand a case to a human. |
Outcome | Advisors used to find the information they needed in about one case out of five. After rollout — in four out of five. Adoption among advisors close to universal. | A return to hiring people. The CEO said publicly that focusing too much on cost led to lower quality. |
Cost missing from the business case | The librarian's work — countable, one-off, it amortises. | The reversal and the reputation — impossible to price upfront, paid in public. |
One thing separates these two stories. Morgan Stanley invested in a library and a librarian before letting the brilliant new hire in. Klarna let him in straight away — and asked him to lock the door behind the people leaving.
The lesson from Klarna is not "AI does not work". It is: a rollout without thinking through what the system knows, what it does not know, and when it must hand over to a human costs more than it saves — because the bill has to include the cost of the reversal and the reputation lost along the way.
RAG answers the question "how". Strategy answers "what", "who" and "why".
A test for your team
Four questions you can ask tomorrow morning
You do not need to understand vectors, embeddings or system architecture. It is enough that at the next conversation about "rolling out AI" you ask your team four questions — in this order.
"Where does the system get its knowledge, and who owns keeping it current?" — an AI quoting an old price list is not a technology failure, it is a vacant librarian position. If nobody in the company owns document freshness, the system starts ageing on day one.
"Can we show where every answer came from?" — when a customer, an auditor or a regulator asks why the system said what it said, a shrug is not an answer. Every answer from a production AI system should leave a trail you can follow.
"What may the system do on its own, and what requires a human?" — an example from our own practice: in the systems we design, AI never initiates a payment by itself. Not because it cannot. Because it should not. The line between assistance and autonomy is a management decision, not a technical one.
"Did we change the way we work, or just add a gadget?" — research on AI rollouts consistently points to process redesign, rather than the technology itself, as the strongest driver of financial results. AI bolted onto an old process produces a demo. AI wired into a new process produces a line in the P&L.
If the team answers these without hesitation, you have a strategy and RAG is a well-chosen tool for it. If the room goes quiet, you have a cabinet full of documents and an expensive subscription.
2 August 2026. Question number two stops being good practice and becomes an obligation. From that day the EU AI Act (Art. 50) requires, among other things, clearly informing people that they are dealing with AI content or an AI counterpart. Companies that built transparency earlier will treat the date as a formality. The rest — as a deadline.
The owner's perspective
Order in your data is a hidden asset. Mess is a hidden liability.
There is one more reason to think about this — especially if you ever consider selling the company or bringing in an investor.
Investors today assign the highest valuations not to the AI companies with the best technology, but to those that control unique knowledge embedded in their customers' daily work. The same logic works downstream — including in the valuation of a mid-sized company. An orderly company archive, together with the processes around it, is an asset that raises the valuation. Mess in your data is a liability the buyer will price for you — in their own favour.
Note the symmetry: the same four questions that protect you from a hallucination in front of a customer are also questions from the due diligence list. The librarian, the answer trail, the autonomy boundary and the redesigned process are not an "IT project". They describe how a company manages its own knowledge.
Before you approve the next budget "for RAG", ask your team three questions: who is the librarian, how will we prove every answer, and what happens on 2 August 2026. And if you would rather start with an independent diagnosis — we run one in 30 days, at a fixed price. What you get at the end is a road map, not a deck.
We run on the architecture we describe — our own AI assistant follows these four questions: an auditable trail behind every answer, data in the EU, and a hard boundary on what it may do by itself.
Book the AI Readiness Audit — 30 days →
This article was co-created with the assistance of AI. Facts and recommendations were verified by the author — precisely in the spirit of Art. 50 of the AI Act discussed above.