8× latency reduction for an institutional analytics platform
A data and analytics platform used by central and investment banks.
The platform answers research questions across a vast document estate, the kind of question that would otherwise cost an analyst a day of reading. In a product like that, latency is the product.
The search tool was suffocating under its own weight. Bankers running complex queries waited over two minutes for a generated response, and in this sector that is not an annoyance, it is an adoption killer. Behind each answer sat a search across the whole corpus, a chain of tool calls running one after another, and far more data than any single question needed. The internal AI team had hit a wall, and had concluded that the only remaining fix was to throw considerably more expensive compute at it.
The brief was to halve the retrieval time. The client expected running costs to rise to pay for it, and had accepted that.
- 01Re-architected the system from a monolith into a parallelised service framework, because the limits were built into the design and no amount of tuning would remove them.
- 02Removed the autonomous agents, which were slow and prone to hallucination, and replaced them with deterministic code-based tool calls that query the database directly.
- 03Introduced an abstraction layer over the models, so a task can be moved between them without the pipeline being rewritten around it.
- 04Evaluated the models against a fixed question set and routed each task to the cheapest one that answered it well, instead of sending everything to the slowest and most expensive.
- 05Added caching wherever the system was repeating work it had already done, and stripped the prompt bloat that was inflating every call.
- 06Deployed observability and tracing over latency and token spend, so cost and performance are measured continuously rather than discovered through a complaint.
They asked for two times. They got eight, and paid less to run it.
Measured on the client's own question set, before and after the rebuild.
For the same client we also engineered the OCR extraction pipelines and the cloud infrastructure underneath them.
The client came to us expecting to buy speed with compute. Speed, accuracy and cost are all decided by the architecture, and when a product feels slow to the people paying for it, the answer is almost always in how the work is structured rather than in how much hardware sits behind it.
Retrieval, extraction and the pipelines underneath
This engagement was document retrieval at scale, which is one of the systems we build for banks. The sector page sets out the rest.