Migrate & Modernize

Why Unity Catalog is the game-changer for data governance

Posted by

Bhairab Ghosh

Data is the lifeblood of modern enterprises. As data grows in volume and complexity, data governance and security have also become complex. Without proper governance, data is disorganized, fragmented, and less reliable.

Additionally, enterprises are facing a host of data governance challenges, including the following:

Data silos prevent a single unified view of data across the enterprise.
Lack of data auditing – or identifying who is making data modifications, when they are changing data, and its purpose.

How does poor data governance impact organizations? Let’s look at some recent statistics:

Enterprises spend 20-40% of their IT budgets on fixing data governance issues.
60% of business leaders prioritize effective data governance.
80% of data security practitioners are prioritizing data security and governance ahead of AI initiatives.

With Databricks’ Unity Catalog solution and integration capabilities with platforms like GCP, organizations can now eliminate data silos and streamline their data governance. Let’s discuss why Databricks’ Unity Catalog solution is a game changer for data governance.

How Unity Catalog is transforming data governance

Centralized metastore
With traditional data governance tools, each workspace had its metastore that stored its metadata. This metadata was only accessible to users within that workspace. To access this data, enterprises required external tables to be created in the local metastore – and also know the exact path to the data stored in the data lake.

For instance, the marketing and supply chain function in the organization had their respective workspaces along with separate workspace objects. Using data pipelines, the marketing function can pull and cleanse data from its source. However, the supply chain function does not have real-time access to the marketing data lake.

With Unity Catalog, enterprises can now create a centralized metastore where all connected workspaces can store and access objects. With this capability, Unity Catalog transforms data governance for all attached workspaces in the metastore, and provides user access to the workspace (without any administrative overhead).
Databricks Lakehouse Federation
Designed by Databricks, Lakehouse Federation in Unity Catalog allows users to run database queries across multiple data sources without migrating the data to a unified platform. Besides Databricks, Lakehouse Federation enables database connections to various sources, including:

MySQL
PostgreSQL
Amazon Redshift
MS SQL Server
Google BigQuery

As an enterprise-wide data governance solution, Unity Catalog’s Lakehouse Federation provides a unified solution for both data and AI. This effectively makes it possible for enterprises to govern data across multiple platforms, without copying or moving the data to the Databricks platform.

User and group provision
With traditional governance tools, users and groups were assigned and managed within the workspace. Both workspace management and user assignment were highly federated in this approach. This made it challenging to know the following:

Which users or groups could access any particular workspace?
How to assign new users and groups to workspaces as they increased in number?

With Unity Catalog’s administrative console, data administrators can assign users and groups to workspaces from a central platform. All they need to do is add users/ groups in the console. Unity Catalog workspaces are then assigned properly to the appropriate user/ group existing in the administrative console.

Additionally, when workspaces are converted from other platforms to Unity Catalog, the existing users/ groups continue to exist in the workspace.

Cloud-powered data lake
With Databricks’ Unity Catalog, enterprises can now share live (or real-time) data securely across on-premises and cloud environments. Business users can now share and consume data using their preferred cloud platform, thus reducing the time-to-value.

When integrated with data intelligence platforms, Unity Catalog enables data scientists to leverage metadata in machine learning projects. For AI and machine learning applications, Unity Catalog accelerates data preparation and enables users to discover data by searching datasets.
Data access from external locations
With traditional tools, organizations found it challenging to create a mountpoint that could access the data lake. Essentially, mountpoints are used to perform read, write, and transform operations using an ETL pipeline. As each mountpoint is accessible at the workspace level, it could not be securely used by users/ groups from external workspaces.

This approach led to the creation of “siloed” workspaces for each user or group. With Unity Catalog, enterprises can overcome this limitation by using external locations. External locations are similar to mountpoints – except they integrate the cloud storage path with the storage credential. Using external locations, data access is now controllable using roles assigned to user groups.

How Onix can help you leverage Unity Catalog for data governance

With its Google Cloud Platform (GCP) integration, Databricks users can now work across Data+AI services on GCP, BigQuery,and Google Cloud’s AI platform. As a Databricks partner, Onix enables seamless data migration from existing warehouses to the Databricks cloud platform without any downtime. With our Birds suite of modernization tools, you can facilitate cloud migration to modernize your business operations.

Our team of Unity Catalog specialists can assist you to enable its data governance features and empower its users. Do you want to learn more? Contact us today.