Introduction

Key features of Unity Catalog

1. Define once, secure everywhere
Unity Catalog offers a single place to administer data access policies that apply across all workspaces.

2. Standards-compliant security model
Unity Catalog’s security model is based on standard ANSI SQL and allows administrators to grant permissions in their existing data lake using familiar syntax, at the level of catalogs, databases (also called schemas), tables, and views.

3. Data discovery
Unity Catalog lets you tag and document data assets, and provides a search interface to help data consumers find data.

4. Built-in auditing and lineage
Unity Catalog automatically captures user-level audit logs that record access to your data. Unity Catalog also captures lineage data that tracks how data assets are created and used across all languages.

5. System tables (Public Preview)
Unity Catalog lets you easily access and query your account’s operational data, including audit logs, billable usage, and lineage.

Unity Catalog Object Model

1. Metastore: The top-level container for metadata. Each metastore exposes a three-level namespace (catalog.schema.table) that organises your data. It registers metadata about data and AI assets and the permissions that govern access to them.

2. Catalog: The first layer of the object hierarchy, used to organise your data assets. Users can see all catalogs on which they have been assigned the USE CATALOG permission.

3. Schema: A schema, also known as databases, are the second layer of the object hierarchy and contain tables and views. Users can see all schemas on which they have been assigned the USE SCHEMA permission, along with the USE CATALOG permission on the schema’s parent catalog.

4. Tables, views, and volumes: At the lowest level in the data object hierarchy are tables, views, and volumes. Volumes provide governance for non-tabular data.

How to Setup Unity Catalog in Databricks

Step 1: Confirm that your workspace is enabled for Unity Catalog

1. As a Databricks account admin, log into the account console.

2. Click Workspaces.

3. Find your workspace and check the Metastore column. If a metastore name is present, your workspace is attached to a Unity Catalog metastore and therefore enabled for Unity Catalog.

• If your account already has a Unity Catalog metastore defined for your workspace region, you can simply attach your workspace to the existing metastore. Go to Enable your workspace for Unity Catalog

• If there is no Unity Catalog metastore defined for your workspace’s region, you must create a metastore and then attach the workspace. Go to Create a Unity Catalog metastore.

Step 2: Add users and assign the workspace admin role

Step 3: Create clusters or SQL warehouses that users can use to run queries and create objects

Step 4: Grant privileges to users

GRANT CREATE SCHEMA ON demo_instance TO `data-enthusiasts`;

GRANT CREATE SCHEMA ON <workspace-catalog> TO `data-enthusiasts`;

Step 5: Create new catalogs and schemas

CREATE CATALOG IF NOT EXISTS mycatalog
  MANAGED LOCATION 's3://dept/cloud-data';

GRANT SELECT ON mycatalog TO `data-enthusiasts`;

Use Cases of Unity Catalog

1. Data Lineage
Unity Catalog supports end-to-end visibility of data flow through the Lakehouse visualised with data lineage. These graphs are access-control-aware and restrict access to lineage graphs based on the user’s access. One can visualise the lineage via UI or make API calls to integrate it with other catalogs. Databricks scan the underlying Spark code written in Notebooks to automatically document lineage which makes it quite easy for the industries to use this.



Capturing Data Lineage for your pipelines is also very convenient and useful, using Databricks

2. Data Governance
The Databricks Unity Catalog is designed to provide a search and discovery experience enabled by a central repository of all data assets, such as files, tables, views, dashboards, etc. This, coupled with a data governance framework and an extensive audit log of all the actions performed on the data stored in a Databricks account, makes Unity Catalog very attractive for businesses

Challenges and Limitations

1. R Workloads Limitations: Workloads in R don't support dynamic views for row-level or column-level security.

2. Shallow Clones: Shallow clones to create Unity Catalog managed tables are supported from Databricks Runtime 13.1 and above.

3. Bucketing Limitations: Bucketing isn't supported for Unity Catalog tables and attempts to create them result in exceptions.

4. Cross-Region Writing: Writing to the same path or Delta Lake table from workspaces in multiple regions can lead to unreliable performance.

5. Partition Scheme Limitations: Custom partition schemes created using commands like ALTER TABLE ADD PARTITION aren't supported for Unity Catalog tables.

6. Overwrite Mode for DataFrame Write Operations: Overwrite mode for DataFrame write operations into Unity Catalog is supported only for Delta tables.

7. Workspace-Level Groups Limitation: Groups created at the workspace level can't be used in Unity Catalog GRANT statements. Use account-level groups instead.

8. Thread Pool Usage: Standard Scala thread pools aren't supported; use special thread pools in org.apache.spark.util.ThreadUtils

• Object names cannot exceed 255 characters.

• The following special characters are not allowed:
• Period (.)
• Space ( )
• Forward slash (/)
• All ASCII control characters (00-1F hex)
• The DELETE character (7F hex)

• Unity Catalog stores all object names as lowercase.

• When referencing UC names in SQL, you must use backticks to escape names that contain special characters such as hyphens (-)