17. Databricks & Pyspark: Azure Data Lake Storage Integration with Databricks

How to integrate Azure Data Lake Storage with Databricks?

There are several ways to integrate ADLS with Databricks such as using service principal, Azure Active Directory credentials etc. In this demo, two methods are demonstrated i.e directly accessing using access key and creating mount point.

What is mount point?
The mount point is a pointer to azure data lake storage. Once mount point is created databricks can access the files in ADLS as if local file system

This video covers end to end process to integrate ADLS with databricks. This demo exercise covers these three areas
1. Create Azure Data Lake Storage in Azure Portal
2. Create Mount point using ADLS Access Key
3. Read files in ADLS through Databricks using mount point

#DatabricksIntegrateADLS #SparkADLSIntegration #DatabricksMount #ADLSMount #SparkMount #DatabricksReadADLSFiles #DatabricksReadADLS #DatabricksReadCSVfromADLS #Unmount #DatabriksUnmount #Pyspark-unmount #DatabricksDButility #DbutilsMount #DbutilsList #Sparkmountunmount #DatabricksRealTimeproject #DatabricksRealTimeExercise #Sparkrealtimeproject #Pysparkrealtimeproject #DatabricksTutorial, #AzureDatabricks #Databricks #Pyspark #Spark #AzureDatabricks #AzureADF #Databricks #LearnPyspark #LearnDataBRicks #DataBricksTutorial databricks spark tutorial databricks tutorial databricks azure databricks notebook tutorial databricks delta lake databricks azure tutorial, Databricks Tutorial for beginners, azure Databricks tutorial databricks tutorial, databricks community edition, databricks community edition cluster creation, databricks community edition tutorial databricks community edition pyspark databricks community edition cluster databricks pyspark tutorial databricks community edition tutorial databricks spark certification databricks cli databricks tutorial for beginners databricks interview questions databricks azure

[xjlqngj]

Enjoying the PySpark tutorials! Can you make a video on setting up Azure and navigating the portal? It would be super helpful. Thanks for the great content!

[shivanisaini]

this video is worth watching, my concepts related to access the file in databricks are clear now thank you sir

[naveenkumarsingh]

hey you are using location as wasbs:// which is nothing but azure blob storage location, and sometimes you are taking abfss:// which is path to azure data lake gen2 location.. Since I am still learning, I am getting really confused now.. And your video says adls connection with databricks..then it should be abfss:// right for a file path?

[dhivakarb-dsmi]

I am getting this error
Operation failed: "This request is not authorized to perform this operation using this permission."

[sujitunim]

Really very helpful... could you please create video for on premise Kafka integration with databricks

[Ramakrishna]

Great knowledge. How can we apply access polices on mounted containers?
For ex, 50 users have acess for databricks so, 50 users can see the all files under mounted container but i want to give read acess for few users only? How can we?

[DivyenduJ]

Hello All, I am new to this and getting below error, many thanks if anyone could help for step 1:

Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key

[alexfernandodossantossilva]

If we have a Vnet on the storage account? How can we access?

[kartechindustries]

Sir does azure data lake comes under community groups or free services

[lucaslira]

with this option, is possible writing in data lake? Or only read?

[rambevara]

Don't we need to app registration for data lake?

[rajivkashyap]

Hi sir,
Any git link is dere so that we can copy and paste the code

[lucaslira]

How would I do if the container had more files instead of just 1?

[subbareddybhavanam]

Hi Raj, Can you please add data files too. like CSV and Json ...