Advancing Spark - Managing Files with Unity Catalog Volumes

In the time before Unity Catalog, we mounted our lakes to a workspace and had nice aliased folder paths to refer to incoming data files, sandbox data, experiments and any other types of lake file. Unity Catalog brings a huge amount of governance, security and management functionality, but we felt a huge gap when it came to accessing actual files! Unity Catalog Volumes fills this gap, providing a slick, easy way of bringing your file-based data into the catalog.

In this video, Simon walks through setting up a Unity Catalog volume, before showing how it can then be viewed, queried and even hooked up to Autoloader for efficient ETL loading.

If you need help rolling out Unity Catalog and revamping your lakehouse to take full advantage, get in touch with Advancing Analytics

[vincentdelbaen]

Thank you sir!
I'll try it out right away and probably include it to our ways of working.
I feel it can reduce the burden and avoid creating external locations for each data analysts projects.

[datawithabe]

Great video, ! as always, best place to learn new Databricks features :)

[coleb]

Great video. One unrelated question: how do you guys manage deployments with databricks? I come from an airflow +Jenkins background as an engineer. Would you recommend Jenkins for databricks deployments?

[AshleyBetts-ht]

Love your work Simon. Do you know if it is possible to have a credential that is not associated with same cloud provider as the Unity Catalogue instance? I have Databricks environment deployed on Azure but one of the ingestions is via an S3 bucket. I would love to be able to set this up as an external volume.

[atulbansal]

How can I get the access of data ricks environment for learning. I know there is a community edition available but somehow I am not able to load my raw files into that

[ErikParmann]

So with mounts we can have the dev workspace mount the dev containers, and the prod environment mount the prod containers, and they both get mounted to the same path. So the notebook don't have to 'know' if its running in dev or prod. How will that work in this new world? I noticed that the path contains "dev". Does each notebook have to figure out what environment it is in, and then read/write from the right paths and catalogs based on some string manipulation?

[MariusS-hp]

Does this also replace DBFS access in general?