Master Databricks and Apache Spark Step by Step: Lesson 30 - PySpark: Reading from a SQL Database

This video shows you how to read data from a SQL Database from Databricks/Spark using Python. SQL databases are the most popular platform for storing data so you are going to need this in your toolbelt.

Join my Patreon Community

Twitter: @BryanCafferky

Notebook with Code at:

Video on Creating an Azure SQL Database

Video on Creating an Azure Key Vault

[selimberntsen]

Great video Bryan! I did not even know this was possible with Spark. I will take this info practice very soon. The best series yet on Databricks and Spark ❤️

[muhammadzulqarnainbutt]

Hi Bryan. Please also talk about issuing concurrent queries to SQL DBS from the spark. i.e. reading data in parallel as the case you discussed brings entire data onto the driver node and also the optimization techniques.

[Peter-xnbk]

Hi Bryan, I curious that Should I need paging when query for prevent overwhelm databrick cluster memory ?

[gaelbosc]

Hi, do you know how to create a pushdown_query but with a "with statement" at the begining, is it at least possible ? Thx