Azure Databricks using Python with PySpark

Learn how to use Python on Spark with the PySpark module in the Azure Databricks environment. Basic concepts are covered followed by an extensive demonstrations in a Databricks notebook. Bring your popcorn!

[shaibalbose]

amazing presentation is only possible by people who has deep understanding and clarity - with the bonus of excellent communication skill ... thank you !!

[thehumbleone]

Even though its been 5 years since he published this, it is still jam packed with knowledge and how to work with PySpark, Thank you Bryan!

[SaurabRao]

Was really glad when you said ' highly recommend you don't restrict yourself to python' in a video which deep dives into Python with PySpark! A real good video.

[digwijoymandal]

When I noted this video, never knew that I would be watching it till the end. But I took time and watched it till the end and it took me 2 days as I practiced all along. Its totally worth it. Keep sharing your knowledge.
Cheers!

[umuttekakca]

Could not have been showcased more nicely and concisely.

[SuperGnarley]

Sir Cafferky! Thanks to your generous brilliance and my YouTube search skills, my day is made! Thank you so much for the information.

[christianlira]

Two excellent Azure Databricks videos Bryan, and thank you for taking the time for sharing your knowledge.

[suhasreddybondugula]

Really helped me to understand PySpark as a beginner. Hoping to see videos on real-time and streaming data. Thanks and keep sharing your wonderful knowledge Bryan.

[SIVERITOO]

I really had to log in just to like and subscribe. Your explanations are awesomely straight to the point and not time wasted, really excellent.

[mithileshsanam]

26:36 minor correction in the code, df.selectExpr() takes column names as sql would do. So, if we have spaces in col names, it wouldn't take the actual values. instead, use df.withColumnRenamed()

[SurenderSingh-rntp]

Really Great Explanation. Totally worth spending 2-3 hours to watch the video and understand all the concepts in detail. Thanks @Bryan Cafferky

[vajikaakbar]

Really helped me thank you so much. Keep sharing your knowledge.

[techsteering]

Thanks for this amazing video. Exactly, what I was looking for.

[saavipihu]

Nice tutorial, very well explained, thanks Bryan !!

[stateside_story]

Really great tutorial ... Thank you Bryan !

[saurinpatel]

Very good video, it would be awesome if you can create similar video just for the ML.

[nabilaabraham]

Hi Brian! Great tutorial!

At 26:44 when you rename the columns like 'blood pressure' to 'bloodpressure', the actual data doesn't get copied over. It looks like that new column 'bloodpressure' is just populated with 'bloodpressure' over and over again. That's not supposed to happen, right? The same thing happened when I used your syntax to copy columns with the SQL statements. Could you please advise on how to actually copy data over?

[chicagobeast]

Question - If I have a script written using pandas for transformations in a Databricks notebook... would I need to convert all the code to pyspark to realize the benefits or would it be okay if I only converted the 'inefficient blocks' and used pandas for some of the more simpler munging tasks?

[JimRohn-uc]

Since SQL is native to Spark is there any benefit of using PySpark over Spark SQL?

[whharding]

Brian,
do you mind a random question?
when in Databricks notebooks and writing base Python on a local pandas dataframe, is that technically still PySpark?

not sure why that question matters to me but it kind of bothers my brain not knowing for certain 🙃

if it is PySpark does that mean even pandas dataframes get passed to the optimiser, or is that restricted to distributed dataframes?

loving the videos, thank you.

also really love your sign off, thanks for pulling for us, great person!