Why inference egress and retrieval overhead are becoming the hidden cost centre in enterprise AI

ROI & Cost Optimisation

24 April 2026 | By Ashley Marshall

Most AI budgets still focus on token prices. In production, the nastier surprise is everything wrapped around the model call: retrieval, reranking, data movement, observability and network egress.

The AI invoice most teams watch is no longer the one that matters most

When leaders say an AI assistant costs a few pence per interaction, they are usually talking about the visible model call. That number matters, but it is increasingly incomplete. In a production enterprise workflow, the answer a user sees is often the end of a much longer chain: a query is rewritten, several indexes are searched, chunks are embedded or refreshed, results are reranked, extra context is injected, guardrails inspect the output, logs are written, and the final response is delivered across one or more cloud boundaries. The model bill is only one line in that chain.

This is why finance teams are starting to feel an odd disconnect. Usage looks stable, yet infrastructure, networking and platform bills drift upward anyway. The hidden cost centre is not usually one catastrophic fee. It is the accumulation of small, repeated overheads that sit outside the headline prompt pricing. Retrieval-augmented generation was adopted because it looked cheaper and safer than retraining. In many cases that is still true. But cheaper than retraining does not mean cheap in production, especially when thousands of daily queries repeatedly pull context from large, distributed knowledge stores.

The UK government’s March 2026 AI Insights: RAG Systems note makes the trade-off plain: RAG reduces the need to retrain models whenever information changes, which makes systems more adaptable and cost effective. In the same paper, the implementation steps listed around chunking, indexing, query embedding and post-processing also reveal why running costs can spread well beyond one model endpoint. RAG adds operational layers by design. That is exactly where hidden spend creeps in.

What this means in practice is simple. If your dashboard tracks token usage but not retrieval calls, vector database latency, reranker frequency, cache hit rate, cross-region transfer and average context payload size, you do not yet understand your AI unit economics. You only understand the most obvious fraction of them.

Inference is scaling faster than many cost models assumed

There is another reason this issue is becoming more painful now. Modern systems are finding ways to buy better outcomes by spending more inference budget at runtime. That changes the economics. In March 2026, the UK AI Security Institute and Irregular reported that recent frontier models in cyber evaluations could productively use 10 to 50 times larger token budgets than typical evaluation settings, and that roughly 8 per cent of AISI tasks were only solved when the budget increased from 10 million to 50 million tokens. That is a research setting, not a customer support bot, but the signal matters. Performance is no longer always plateauing early. Sometimes the answer really does get better when you let the system think longer, search more and try again.

For enterprise buyers, that creates a subtle trap. Teams see a proof of value, then improve quality by adding multi-step retrieval, larger context windows, more agent turns, stronger reranking and better monitoring. Each change is rational in isolation. Together they produce a production architecture that is much more expensive than the pilot. The hidden cost centre emerges because the quality win is credited to the model, while the cost sits across several services and invoices.

AISI also noted that success rates scaled roughly with the log of total tokens used per attempt, and that average cost per run at the 50 million token limit was about 10 US dollars, with maximum cost below 60 US dollars. Again, those are evaluation numbers, not direct enterprise benchmarks, but they are useful because they demonstrate the broader point: once performance improves with more runtime budget, inference stops being a neat per-call commodity and starts behaving like an optimisation surface. Cost depends on how much search, reasoning and context assembly you allow.

The common misconception is that inference is now cheap because training is expensive. In reality, for most enterprise adopters, they are not training frontier models at all. Their real exposure is repeated runtime spend. The expensive thing is the thing you do every day, at scale, under service expectations. That is inference.

Retrieval overhead is not just vector search - it is the whole context supply chain

When people discuss retrieval costs, they often picture a vector database query and stop there. In production, retrieval is a supply chain. Documents need cleaning, chunking, embedding, indexing, versioning and periodic re-embedding. Queries may be rewritten. Search often combines keyword, semantic and metadata filters. Results are reranked. Access controls are checked. Relevant passages are packed into a prompt. Low confidence answers may trigger a second search or a fallback workflow. Every one of those steps has a cost in compute, storage, latency and engineering effort.

The UK government’s January 2026 guidance on making datasets ready for AI is useful here because it frames the issue as data product discipline, not model magic. It explicitly asks whether an AI capability has the right data, whether it meets the business need, whether it is cost effective and whether it is easy to maintain and compliant. That is the right lens. Retrieval pipelines are expensive when data is fragmented, poorly documented or stored far away from the inference environment. The bill rises because the system has to work harder to compensate for weak information architecture.

What this means in practice is that two organisations can run the same foundation model and get wildly different economics. One keeps authoritative data well-structured, access-controlled and close to the application. The other pulls from old file shares, SaaS silos and duplicated document stores across regions. The second organisation pays more for every answer even if the model is identical. This is why AI cost optimisation is often a data architecture problem wearing a model budget label.

There is a second misconception here. Many teams assume retrieval automatically lowers cost because it avoids fine-tuning. That is directionally true, but incomplete. Retrieval lowers some costs while introducing others. If your system retrieves too much context, refreshes embeddings too often, or fails to cache stable answers, the savings from avoiding retraining can be eaten away by constant runtime overhead.

Egress is the quiet multiplier that turns a good architecture into an expensive one

Network egress is one of the least discussed AI costs because it rarely appears in model vendor demos. Yet it is often the fee that punishes architectural sprawl. If your embeddings live in one cloud, your source documents in another, your application layer in a third platform and your model API somewhere else again, every retrieval-heavy interaction can move data repeatedly across paid boundaries. None of those transfers looks dramatic on its own. At enterprise query volumes, they compound.

This matters even more in the UK market because regulatory and sovereignty decisions often push firms towards hybrid or multi-environment designs. That can be the right choice. In April 2026, the UK government announced its £500 million Sovereign AI initiative, with support including up to 1 million GPU hours per startup and backing for companies such as Callosum and Doubleword. The strategic message is clear: where AI runs, and under whose control, matters. For enterprise buyers, though, sovereignty is not free. Data locality, private hosting and secure interconnects can be worth paying for, but they change the cost base. If you ignore egress and replication patterns while designing for control, the surprise arrives later on the infrastructure invoice.

That does not mean sovereign or hybrid approaches are wrong. It means finance and architecture must stay joined up. A chatbot answering policy questions from one tightly controlled UK environment may cost less overall than a supposedly cheap design that bounces documents, embeddings and prompts between several services. The lowest model price is not the same thing as the lowest operating cost.

The practical move is to map the full journey of data for one representative query. Where does the source document sit? Where are embeddings generated? Where is vector search executed? Where is reranking executed? Where is the model hosted? Where are logs written? Any paid boundary crossed repeatedly is a candidate hidden cost centre.

Compliance and accuracy requirements make cheap architectures look less cheap

There is a reason regulated organisations end up with more expensive AI stacks. They are not being slow. They are paying for accuracy, traceability and control. The ICO’s consultation on generative AI accuracy is a good reminder that the relevant question is not whether outputs are perfectly accurate in the abstract. It is whether the level of accuracy is appropriate for the purpose and whether personal data is processed lawfully and corrected where necessary. In practice, that pushes organisations towards stronger retrieval provenance, better source selection, tighter evaluation and more logging. All of that costs money.

This is where simplistic ROI narratives break down. A procurement team may compare two assistants and ask which one has the lowest token rate. A compliance lead asks different questions: can we show which source passages informed the answer, can we prevent old or inaccurate records being surfaced, can we honour access permissions, can we audit corrections, and can we keep data in the right jurisdiction? Those are not bolt-on extras. For many sectors, they define whether the system is usable at all.

What this means in practice is that enterprise AI programmes should price for assurance, not just generation. If you operate in financial services, healthcare, legal services or the public sector, the cheap prototype often becomes the expensive rebuild because it omitted the controls needed for production. Better to design those controls into the cost model from day one. That includes retrieval evaluation, source freshness rules, role-based access, redaction where needed, and clear records of which systems exchanged data during each workflow.

The counterargument is that adding all this governance kills the ROI. Sometimes it will. That is not a reason to ignore it. It is a reason to stop pretending every AI use case deserves production deployment. Some workflows are too low value to justify the retrieval, security and evidence burden they create. A disciplined no is often a better commercial decision than a superficially cheap yes.

How to regain control of the cost base before it becomes next year’s surprise

The good news is that this cost problem is manageable once you measure the right things. Start by abandoning any dashboard that reports only model spend. Your baseline should include cost per successful task, not cost per thousand tokens. Then break that task cost into components: retrieval calls, average context injected, embedding refresh volume, reranking frequency, cache hit rate, storage, logging and every paid network transfer. If a supplier cannot help you expose those numbers, you are buying a black box with a delayed invoice attached.

Next, redesign for locality. Keep the data needed for common answers close to the inference path. Reduce cross-region movement. Be ruthless about stale duplication. Many RAG stacks over-retrieve because nobody owns chunk quality or prompt packing. Better curation often beats bigger context. Likewise, caching stable answers, precomputing common retrieval paths and setting confidence thresholds can cut waste without reducing quality. Teams should also test smaller models with better retrieval before defaulting to a more expensive frontier model. A leaner architecture often wins twice: lower direct model spend and lower retrieval overhead because prompts stay tighter.

There is also a portfolio decision to make. Not every use case needs live retrieval. Some need batch generation. Some need strict templates with minimal context. Some need search, not generation. The organisations that keep AI cost under control are usually the ones willing to route work to the simplest tool that can do the job. They do not let every internal request turn into a full agentic workflow.

If I were advising a UK leadership team this quarter, I would ask for one thing above all: a single-page AI unit economics view for the top five workflows in production or pilot. Show the all-in cost per task, the hidden infrastructure components, the data movement path and the business value created. Once that exists, conversations about ROI become calmer, sharper and much more honest. Without it, inference egress and retrieval overhead will keep behaving like a hidden cost centre because, organisationally, you have allowed them to stay hidden.

Frequently Asked Questions

Why is inference becoming a bigger cost issue than training for many enterprises?

Most enterprises are not training frontier models from scratch. They pay repeatedly for runtime activity instead: prompts, retrieval, reranking, monitoring, security controls and data movement across live workflows.

Is retrieval-augmented generation still cheaper than fine-tuning?

Often yes, especially when knowledge changes frequently. But that does not make it cheap. Poor chunking, excessive context, constant re-embedding and cross-cloud transfers can erode the savings quickly.

What exactly counts as egress in an enterprise AI stack?

Egress is paid data leaving one service, region or cloud boundary for another. In AI systems that can include document transfer, embedding movement, prompt payloads, logs, cached responses and replicated datasets.

How can we spot hidden retrieval overhead early?

Measure retrieval calls per answer, average context size, reranker usage, cache hit rate, embedding refresh frequency, vector database latency and any network charges tied to the workflow. If you only measure tokens, you are late.

Does better accuracy always require a more expensive architecture?

Not always. Better data curation, tighter access control, smarter caching and smaller well-routed models can improve accuracy while reducing cost. What increases cost is uncontrolled complexity, not discipline.

Which UK compliance issues make these costs more significant?

UK GDPR duties around lawful processing, data minimisation and accuracy, plus ICO expectations on appropriate accuracy and rectification, often require stronger provenance, logging and access controls that add operational cost.

Should UK organisations avoid multi-cloud AI because of egress?

No. Multi-cloud or sovereign designs can be strategically sensible. The point is to cost them honestly and minimise unnecessary transfers, not to assume the cheapest model endpoint gives the cheapest overall architecture.