Cloud Migration of siloed datasets and on-prem warehouse in the Aviation industry with Snowflake
Migration and integration activities completed
Use cases migrated and available through Access Layer
Kubrick consultant FTE conversion during migration project
KubrickSnowflake
The challenge
The client was struggling with a fragmented system landscape, using a combination of on-premise and cloud-based solutions, which led to poor integration across the organisation and its wider group. This disjointed setup resulted in an overly complex data-sharing process and made it impossible to generate dynamic reports across the group.
To resolve these challenges, the client decided to migrate to a unified cloud-based platform that would enable easier access to data for both the business and the group. Additionally, they aimed to achieve process efficiencies by completely redesigning their data model using the Data Vault modelling technique.
The solution
Kubrick integrated with the client’s technology stack and worked alongside an additional external consultancy, TCS, to address key challenges. We helped resolve operational inefficiencies by implementing an improved Scrum operating framework and refining the team structure. To overcome the lack of an integrated loading tool, we developed an ingestion solution using AWS, with daily orchestration managed through Airflow.
We collaborated with IAGL to design and implement a Data Vault remodelling process, mapping attributes across seven use cases to ensure effective data loading. Our team also handled ETL transformation via Matillion, where the data was cleansed, de-duplicated, and validated through three distinct layers: landing, staging, and Data Vault (Snowflake).
We facilitated the migration of historical data through a one-time load into Snowflake, with data organised in tables defined by the Data Vault model and an attribute mapping exercise. Ongoing delta loads were scheduled to run daily, managed by Matillion pipelines, with Airflow workflows orchestrating direct acrylic graphs (DAGs) to streamline the process.
The results
As part of our solution, we integrated seamlessly with the client's key migration steps, driving operational efficiencies across both processes and people. This collaborative approach led to four out of seven migration use cases being successfully moved to production, ensuring a smooth transition to the new platform. Additionally, we created an Access Layer from scratch by working closely with end-user business stakeholders, which enhanced data accessibility and usability across the organisation. These outcomes significantly improved the client’s operational capabilities and supported their long-term data strategy.
Our seamless integration and expertise in data transformation enabled the client to unify their fragmented systems, achieving operational efficiencies and bringing 4 out of 7 critical use cases into production. Our collaboration with Snowflake ensured a smooth transition to a cloud-based platform, enhancing data accessibility and empowering long-term success.
