Creating Environments (Dev and Prod) in Azure Databricks Unity Catalog

The Unity Catalog enables centralized access control, auditing, lineage, and data discovery functionalities across Azure Databricks workspaces. This guide will walk you through the process of setting up the Unity Catalog in Azure Databricks.

Working with Azure Databricks Unity Catalog

📌 Challange before with sharing Metastore

One of our prerequisites is that we can only create a single metastore per region. However, when sharing the metastore with full access to the catalogs, each workspace can view all the catalogs. This can lead to confusion, especially when we have separate prod, test, and dev workspaces, as the prod workspace may have visibility into dev data and vice versa.

Solution

The introduction of Binding Unity Catalog to specific workspaces simplifies the process of segregating environments and assigning catalogs to their respective workspaces. For instance, we can now have a dev_catalog and prod_catalog within a single Unity Catalog metastore. The dev_workspace can utilize the dev_catalog for the development environment, while the prod_workspace can utilize the prod_catalog for the production environment.

3 steps process to assign catalogs to workspaces.

In the end, when logging into a specific workspace, if the catalog is not mapped to that workspace, we will encounter a “no access” message.

Hope this helps!