As government policy increasingly promoted secure data sharing between public-sector bodies, the agency needed stronger governance controls, lineage and access management before it could safely expand the use of its data assets.
The critical nature of the service meant the migration needed to protect production pipelines.
Kubrick designed the Unity Catalog architecture, built a migration playbook and automation tooling; and migrated active pipelines.
The agency gained table- and column-level controls, data lineage and improved discovery, creating a governed foundation for secure data sharing and future analytics.
- 01
The challenge
The UK Government’s Data Sharing Governance Framework identifies the need for cross-government data sharing, breaking down organisational silos, common governance controls, and sharing data securely while maintaining compliance requirements.
As part of its statutory reporting, the agency is also required to report on its governance arrangements.Furthermore, the agency wanted to leverage advanced Databricks features which all required that Unity Catalog (UC) be enabled.
To meet these expectations, the agency was expanding its Databricks Lakehouse to deliver data products for external partners and support greater collaboration between government departments. The existing Hive Metastore offered no way to restrict access at the required level, and duplicated tables to mask data for different users. The estate covered four workspaces and 12 existing pipelines producing approximately 1,000 tables, with Azure Data Factory orchestrating batch processing.
Moving production code and data to Unity Catalog required specialist migration knowledge, coordination across teams and careful sequencing.The agency needed stronger ownership, maintainability and lineage without disrupting pipelines that supported live operations.
- 02
The approach
Kubrick combined architecture design, automation and phased migration to introduce Unity Catalog while protecting production continuity and build capability.
- Assessed the estate
Kubrick used UCX assessments and design workshops to document the four workspaces, pipeline dependencies, users and groups. This established the migration scope and the target catalog, schema, and permissions design. - Designed governance and permissions
Kubrick defined the high-level Unity Catalog architecture and a permissions model covering individual users and directory groups. The design introduced fine-grained access controls and clarified how different personas would interact with governed data. - Built reusable migration tooling
Kubrick created mapping, object-creation and notebook-migration tooling. SQL and Python scripts converted storage paths and table references, created Unity Catalog objects and updated notebooks to the three-level namespace, reducing manual migration work. - Migrated and validated pipelines
Kubrick moved five active pipelines through development, UAT and production, including the associated Azure Data Factory and configuration changes. The team tested access controls and supported production cutover with documentation and hypercare. - Transferred the method
Kubrick documented the migration in a step-by-step playbook and role-based user guidance, then ran demonstrations and knowledge-transfer sessions so the agency could operate and extend the governed environment.
- Assessed the estate
- 03
The impact
The agency can now control access at table and column level for defined user personas, use lineage to understand how data moves through the platform, and discover data more easily.
The migration also created reusable documentation and automation for later pipelines beyond the 5 migrated during the project.
Unlock more value from your Databricks investment
