PySpark Tutorial: Spark SQL & DataFrame Basics

Subscribe if you enjoyed the video!

Best Courses for Analytics:
---------------------------------------------------------------------------------------------------------

Best Courses for Programming:
---------------------------------------------------------------------------------------------------------

Best Courses for Machine Learning:
---------------------------------------------------------------------------------------------------------

Best Courses for Statistics:
---------------------------------------------------------------------------------------------------------

Best Courses for Big Data:
---------------------------------------------------------------------------------------------------------

More Courses:
---------------------------------------------------------------------------------------------------------

[coemgeincraobhach]

Thanks so much Greg, great job!
Paying thousands for a masters at university, and people like you consistently pump out tutorials of way better quality. Its madness.

[jacobburt]

I appreciate you and your videos so much. In my data science classes we're expected to teach ourselves Pyspark, Dataframe, Pandas and a bunch of other technologies and you've made the experience much more manageable.

[TheALahiri]

Many thanks Greg for opening up a new frontier!
I had no idea Google Colab was so generous and allowed installation and practicing of Spark.
Your tutorial packs an astonishing amount of information, that too in an engaging way, in a very short timeframe.
You are now my Guru for Spark.

[andersborum]

These introductory videos are pure gold; thanks for sharing.

[ashutoshsingh]

Thank you so much. Pretty covers everything you to get started with pyspark. I wish you had included merging as well.

[barmalini]

In just 17 minutes I've learnt so much. Thanks!

[darrienjohnson]

Don’t know if you’ll see this but I got into data engineering thru my company. They provided me the opportunity to become a software engineer, I was previously a cable installer/field tech. Although they provided this opportunity, I’ve still had to do much of my learning on my own. Your channel is amazing. Videos like these make all the difference. I really appreciate you making content where you’re walking thru the code. Once I get this under my belt I plan on creating content as well. Thank you. 🙏🏾

[faizalshebli]

Great video. Simple yet effective to comprehend.

[saketsrivastava]

Amazing content...please prepare more like these.. 👍🏻

[gauravraichandani]

This was really amazing. Waiting for more uploads on pyspark.

[ramanantoaninaharintsoanan]

Thanks for sharing your knowledge. Great video.

[tkadado]

Excellent presentation and to the point !!!

[nishantbahikar]

Bro you have explained it so well.. keep going

[mohamedelkhaldi]

Thank you so much !!! Always great contents

[soumyadeeppattanaik]

hey.. Hogg while i am trying to extract sum of sales by grouping the states from the dataframe, its giving an unnesessary floating values. If the sum is 150.0 its giving like 150.856743 like this.can you explain this..

[AkshayKumar-vdwn]

Thank you for the video.
I have a problem -
When I convert a column from string to int and then run printSchema it shows String and not the int.
Is there a better way to convert string column to int in pyspark and make it a permanent change?
I use thr data uploaded locally, I.e from my computer.

Is this happens to only locally uploaded files? Will the conversation take place smoothly when operating on okne databases i.e through servers.

[noushinbehboudi]

Awesome. Please keep up the good work. Please make more videos in spark. Thank you

[gerardolamasrosales]

Hola, como creo una base de datos con pyspark?

[matattz]

I would like to hear your opinion on Ponder. Considering that you can now work with Ponder similarly to how you work with Spark, do you believe it is still necessary to learn PySpark? I'm interested in your perspective on this matter, and if you are aware of any downsides or differences between Ponder and Spark.

[limingcai]

I downloaded the train.csv file to my laptop's local hard drive, and tried to read it with titanic_df = Data\train.csv", header=True, inferSchema=True), but got an error message. Do you kbnow what I did wrong?