Unlocking value from trusted underwriting data

The insurer’s underwriting division relied on legacy Actian servers to store and process critical analytical data. The platform was costly to maintain, limited in flexibility, and approaching a key renewal deadline.

The organization needed to migrate 30TB of underwriting data across 16 source systems into Azure Databricks while maintaining strict standards for governance, security, and data quality within a regulated insurance environment.

Complex source structures, varied ingestion patterns, inconsistent data formats, and evolving business requirements made traditional pipeline-by-pipeline migration approaches too slow and expensive. Without a scalable solution, the organization risked extending its dependence on legacy technology and limiting access to trusted underwriting data.

Kubrick implemented a governed Azure Databricks Lakehouse using a metadata-driven delivery framework that accelerated migration, reduced engineering effort, and established repeatable patterns for future source onboarding. 

1. Align the target architecture.
   We assessed the existing data estate and designed a Databricks Lakehouse using a bronze, silver, and gold medallion architecture. This standardized the progression of data from raw ingestion through to curated, business-ready datasets while supporting governance and auditability requirements.
2. Create a reusable migration capability.
   We leveraged Kubrick’s Databricks ETL-in-a-Box framework, built around shared notebooks, Databricks Declarative Automation Bundles, YAML-driven configuration, and infrastructure-as-code. New source systems could be onboarded through metadata and configuration rather than bespoke development, reducing delivery effort and improving scalability.
3. Embed governance by design.
   We implemented automated type casting, reconciliation, data quality controls, quarantine processes, and fail-fast validation checks to identify integrity issues earlier in the data lifecycle. Embedding governance controls throughout ingestion and transformation processes supported regulated underwriting workloads.

4. Migrate and optimize at scale.
   We migrated 30TB of data across 16 source systems into Databricks, supporting multiple ingestion patterns from a single codebase. The team re-engineered high-volume processing workloads and parallelized ingestion to improve performance and operational efficiency.
5. Transfer capability to the client.
   The team worked as an embedded delivery team alongside client stakeholders, supporting migration, validation, and knowledge transfer while ensuring continuity beyond the initial implementation.

With trusted underwriting data more accessible across the organization, analytics teams are better equipped to support pricing, claims, reporting, and future AI initiatives.

• Faster access to trusted underwriting data: processing time reduced from 80 hours to 5 hours for the largest source system
• Retired legacy infrastructure and transition to Azure Databricks in four months.
• Confidence in critical underwriting datasets.Improved engineering efficiency to accelerate future developments.

The business can control how it uses data to make decisions. Enabling effective analytical models and machine learning in a governed and scalable platform was the critical step to realise the value of their AI investments.