Setting up an Azure Databricks Unity Catalog within a Vnet, Part 3 of 3

This blog—Part 3—concludes the series by covering the process for configuring the metastore.
 
3 min read
Kalyana Murugan Manicka Vadivel

Author

Kalyana Murugan Manicka Vadivel
Senior Solutions Architect
3 min read
Share

This three-part blog series provides a comprehensive guide to setting up the Azure Unity Catalog within a VNet environment. The series covers aspects from the Unity Catalog's introduction, infrastructure and networking configurations to security, metastore setup and workspace integration.

  • In , we provided an overview of the Unity Catalog and how to set it up.
  • In , we covered the network setup process.
  • This blog—Part 3—concludes the series by covering the process for configuring the metastore.

Let's configure that metastore.

Configure metastore

Step 1: Create metastore

A metastore is a metadata repository for the Unity Catalog that manages catalogs, schemas, tables and permissions.

  • Go to Databricks UI → click on Manage accounts (or go here: https://accounts.azuredatabricks.net)
  • Click on Admin SettingsCreate Metastore
  • Enter details:
    • Metastore name: (e.g., my_unity_metastore)
    • Region: Must match the workspace and Access Connector (one metastore per region)
    • Storage root path: e.g., abfss://<directory_name>@<storageaccount>. dfs.core.windows.net/
      • As a best practice, it is NOT recommended to store any application data in the root blob (DBFS) storage
      • It is highly recommended to store the application data on an external ADLS Gen2 Storage
    • Access connector: Select the one created (databricks-access-connector)
  • Click Create
  • Assign permissions to the metastore

Step 2: Attach workspace to metastore

  • Go to Databricks Admin Settings → Workspaces
  • Select the Databricks Workspace
  • Click Assign Metastore and select the metastore created earlier (my_unity_metastore)
  • Click Confirm

Step 3: Enable catalog features

Enable serverless compute

Serverless compute needs to be enabled for any model to access the custom function (SQL/Python).

  • In the account console, click Settings
  • Click the Feature enablement tab
  • Enable the Serverless compute for workflows, notebooks and DLT setting

Step 4: Validate setup

  • Deploy a virtual machine with the VNet if the front-end access is disabled
  • Go to the Azure Databricks workspace, launch and create a cluster
  • Create a notebook and attach it to the cluster and test
  • Go to SQL Editor, launch a serverless warehouse and run a DDL/DML to verify storage access
  • Create a SQL/Python function and go to the playground to access it using a model to verify serverless compute

Conclusion: Wrapping up Part 3

Setting up Azure databricks Unity Catalog within a Vnet environment ensures secure, scalable and governed access to structured and unstructured data. Configuring networking, security, metastore and workspace integrations can establish a centralized data governance model while maintaining fine-grained access control through Azure AD, Unity Catalog and Databricks Access Connectors.

With proper identity management, role-based access control and storage security policies, enterprises can collaborate seamlessly across multiple workspaces while ensuring compliance and security in the data architecture. By following this guide, we can successfully implement Unity Catalog in a Vnet setup, empowering organizations with efficient, secure and well-governed data management.

If you missed the first two blogs in the series, you can find them here:

and

References

What is Unity Catalog? - Azure Databricks | Microsoft Learn

Networking - Azure Databricks | Microsoft Learn

Classic compute plane networking - Azure Databricks | Microsoft Learn

Enable Azure Private Link as a standard deployment - Azure Databricks | Microsoft Learn

How to protect Data Exfiltration with Azure Databricks to help ensure Cloud Security | Databricks Blog

Share On