Microsoft announces the general availability of Cross-Cloud Data Governance with Azure Databricks, enabling seamless access and governance of AWS S3 data directly from Azure Databricks via Unity Catalog. This innovation simplifies security, compliance, and data management across cloud platforms without data migration. Unique :
Cross-Cloud Data Governance Now Live on Azure Databricks
Microsoft just rolled out a game-changer for data pros working across clouds. Azure Databricks now supports cross-cloud data governance with AWS S3 via Unity Catalog. This means you can securely access and govern AWS S3 data directly from Azure Databricks—no more tedious data migration or duplication.
What’s New?
Previously, accessing AWS S3 data from Azure Databricks meant costly ETL processes to move data into Azure Data Lake Storage (ADLS). Now, Unity Catalog lets you set up external S3 locations directly in Azure Databricks. This update simplifies your data workflows and slashes operational overhead.
“This release allows teams to directly configure and query AWS S3 data from Azure Databricks without the need to migrate or duplicate datasets.”
Major Updates and Benefits
- Unified Governance: Manage access policies and compliance across both Azure and AWS clouds from a single platform.
- Frictionless Data Access: Securely discover and analyze data across clouds in one workspace, reducing complexity.
- Enhanced Security: Gain centralized visibility with tagging, lineage, classification, and auditing for all cloud storage.
In short, this update tackles the headache of fragmented governance in hybrid cloud environments. Consistency and security are now easier to achieve.
How It Works
Set up is straightforward. Create a read-only AWS IAM role, configure storage credentials in Azure Databricks’ Catalog Explorer, and define external S3 locations. Then, apply consistent permissions across ADLS and S3 data. Finally, start querying your S3 data directly—no data moves required.
“Unity Catalog on Azure Databricks addresses challenges by providing a unified and open governance solution for all data and AI assets.”
Supported Features Include:
- AWS IAM role storage credentials
- S3 external locations and tables
- S3 external volumes
- S3 dbutils.fs access
- Delta sharing of S3 data from Unity Catalog
Why This Matters for Tech Teams
Hybrid and multicloud setups are the norm, but governance often lags behind. This release helps teams enforce consistent security policies and audit trails across platforms. It also reduces duplicated efforts and operational costs.
For anyone managing data across Azure and AWS, this is a must-watch update. It streamlines governance while boosting security and compliance.
Get Started and Join the Conversation
Ready to try it out? Follow Microsoft’s step-by-step guide to configure cross-cloud governance in Azure Databricks. Plus, don’t miss the Databricks blog for more details.
Also, mark your calendar for the Data + AI Summit 2025 in San Francisco this June. Microsoft will be there showcasing Azure Databricks innovations. It’s a perfect chance to connect with experts and level up your data game.
Azure continues to be a top choice for Databricks workloads, especially with these powerful new governance features.
From the New blog articles in Microsoft Community Hub
