· Manfredi Miraula · Case Study · 2 min read
From hours to minutes: near real-time data pipeline for a leading pharma company
Operational data was arriving hours late due to slow batch processes, limiting timely decision-making. We rebuilt the pipeline for near real-time synchronisation — including binary files up to 4GB.
Data that arrives late isn’t data — it’s history.
At a leading pharmaceutical company, operational decisions were being made on information that was already hours old. The teams knew the numbers, but the numbers weren’t current. The culprit: a batch architecture that had worked once but couldn’t keep pace with the business anymore.
The problem
The operational teams needed timely access to data to steer decisions. But the existing pipeline processed data in slow, monolithic batches — refreshing only periodically and creating a lag that made real-time analysis impossible.
The challenge was compounded by a technical constraint: the system also needed to handle large binary files up to 4GB, not just tabular records. Standard streaming solutions weren’t designed for this.
Every hour of delay meant decisions made on yesterday’s reality.
The solution
We redesigned the data pipeline from the ground up with a near real-time architecture on Azure Service Bus:
- Event-driven ingestion — data flows trigger automatically on source changes, not on a fixed schedule
- Incremental processing — only changed records are processed, reducing compute load and latency dramatically
- Binary file synchronisation — a custom layer handles large binary payloads (up to 4GB) with near real-time sync
- Full observability — monitoring and alerting on pipeline health, latency and data quality at every stage
We built on the client’s existing infrastructure, avoiding disruption to downstream consumers during migration.
The results
- Data available in minutes, not hours
- Near real-time synchronisation of binary files up to 4GB
- Improved decision-making across operational teams with current, reliable data
- Zero disruption to existing downstream consumers during migration
The principle that applies everywhere
Batch processing is often the right default — until the business outgrows it. When your teams are making decisions on stale data, the problem isn’t the people — it’s the architecture.
A targeted redesign can unlock capabilities that were simply impossible before.
Is slow data holding back your team? Get in touch — let’s explore whether a similar approach makes sense for you.