Master Databricks and Apache Spark Step by Step: Lesson 23 - Using PySpark Dataframe Methods

In this video, you learn how to use PySpark dataframes methods on Databricks to perform data analysis and engineering at scale. This is the core of using Python on Spark and you need to learn the power but also the nuances involved.

Video demo notebook at:

Apache Spark Zeppelin Notebook link will be posted later.

For information on how to upload files to Databricks see:

[andywendycox]

Bryan - thanks so much for this series. You've made Databricks ( and Spark for that matter ) very easy to digest. These videos have been a lifesaver...

[BryanCafferky]

It was pointed out in a comment, that seems to have been deleted, that you should use spark context instead of sqlContext as the spark context is the newer unified way to connect to the Spark session. Where you see code like sqlContext.read.format(....), just replace sqlContext with spark and you should be all set.

[marina]

Brian, thank you for great presentation. Your gift to explain complicated things as simple concept is amazing

[dangustafsson]

Reaching the end of your series, very enlightening and friendly format. These end lectures are really interesting.
Now here, I’m looking to understand how to efficiently load data, based on different data sources (rmdb:s, hdfs, mongo).
And avoiding ‘shuffles’ or at least understand the cluster bottlenecks…. also on my to-do list…

[ravitutika]

waiting for it from last few weeks, Thanks Bryan

[arpitarora]

Hey man thanx for the whole series i just started working on databricks and was completely oblivious to how it works but your helped me quite alot so ... thanx for that )

[vibhaskashyap]

Thanks for sharing videos, great content and you make complex topics easy to understand 👍

[anandmahadevanFromTrivandrum]

Wow, that was a lot to take in, but well presented. Thanks again

[hmishra]

Made my weekend thanks again brayan keep up the good work .
BR,
Hardik 🙏😀

[amarnadhgunakala]

Looking forward more pyspark vids
Thanks

[neostar]

Thank you very Much Sir.. You made my life easy

[ranjeevtiwari]

Thanks for this series Bryan. The notebook you shared in github is of .dbc extension, can you update your git with current class notebook?

[Raaj_ML]

What is the use of caching ? If you do not do caching, anyways the data frame will remain in memory...right ?

[RandyL]

I'm confused on when to use "sqlContext.<somefunction>" versus "spark.<somefunction>" How do we know when to use what?

For instance, to query you use "spark.sql" but I see from documentation you can also do, "sqlContext.sql"...is there a difference?

[eugenezhelezniak]

Bryan - how do we know whether the dataframe is local or lives on the cluster? Is it as simple as pandas = local, spark = distributed? And follow up to that, if you have a large local pandas df, how do you work around degraded performance?

[Rickantonais]

Thank you for the material, yet there is a background high tone sound which makes the video horrible to listen

[juanpabloguerra]

Hey Bryan this is awesome content. I'm tying to open the file after cloning your GH repo but it seems like it downloads as a DBC file that can't be open on VS code using Jupyter notebooks for example. Is there anything I'm missing? Thanks a lot for the great content

[felixscarbrough]

Hi Bryan, I've been watching your series for a little while now and finding it very helpful. Unfortunately this video seems to have some really high pitched tone in lots of it and it makes it quite unpleasant to listen to. Is there any way you'd be able to remove this?