For a major insurance organization, data modernization was a strategic enabler: creating the trusted analytical capability needed to compete in an increasingly data led market, reducing technology risk, and strengthening governance.
The program created a scalable platform for faster decision making, resilient regulatory and operational reporting, and AI-enabled products and services.
We established a governed Databricks lakehouse that migrated approximately 30TB of critical underwriting data from 16 source systems.
- 01
The challenge
The insurer needed to modernize its analytical estate without compromising security, governance, or data quality. Complex source systems, inconsistent formats, and evolving requirements made bespoke pipeline development too slow and costly.
Without a more scalable approach, the organization risked prolonging reliance on legacy technology and limiting trusted access to critical underwriting data.
- 02
The approach
Kubrick created a governed, reusable Databricks framework that accelerated migration, reduced engineering effort, and supported analytics and AI readiness.
- Align the target architecture. Assessed the source estate and designed a Databricks medallion architecture that aligned the insurer with its wider data strategy and internal controls.
- Create a reusable migration capability. Built a metadata-driven framework with shared notebooks, infrastructure as code, and YAML configuration, replacing bespoke development with repeatable source onboarding.
- Embed governance by design. Automated type casting, deduplication, quarantine, quality checks, and reconciliation to surface integrity issues earlier and strengthen confidence in regulated data.
- Migrate and optimize at scale. Moved data through bronze, silver, and gold layers while reengineering high volume processing and parallelizing ingestion to improve performance.
- Transfer capability to the client. Worked alongside client teams, and maintained continuity after delivery through ongoing technical support and knowledge transfer.
- 03
The impact
The lakehouse delivered:
- Greater access and control: For the largest source, pipeline processing fell from more than 80 hours to 5 to 7 hours, giving the analytics team faster access to critical underwriting data in a modern, governed Databricks environment.
- End-to-end data integrity: Automated reconciliation and quality controls strengthen data confidence throughout ingestion and processing.
- More effective model development: The new lakehouse provides a scalable foundation for continued analytics and model development.
The lakehouse provides the scalability required for advanced analytics, machine learning, and AI-enabled initiatives as the organization continues to expand the value it creates from data.
With trusted data more accessible across the organization, analytics teams are better equipped to support pricing, claims, reporting, and future innovation.
Unlock more value from your Databricks investment
