Apache Spark Transformation and Actions

#apachespark #datascience #dataengineering

In this video I am going to walk through various transformations and actions with Apache Spark. I will get into details of most commonly used functions and transformations with Apache Spark

[IsaiahShadE]

Probably the only person who tells you facts and reality in the data science community.

[HaridasJanjire]

Very well.. Very helpful to learn Apache spark with real business end to end case.

[KishoreKumar-yxnw]

Thanks Srinivasan for the wonderful explanation

[ayeshababar-flev]

Very elaborate and well-explained! Can you please share the code and notebook?

[designwithpicmaker]

Thank you bro thanks for this wonderful content video

[mukeshkesavan]

Thanks ton..! You made spark easy. Please make a video on how to optimize spark code and data skewness..

[ranjanirajamani]

Thank you, Sir my learning curve with regards to Spark has taken an exponential trend after watching your videos. It has been a rich learning experience. I have been trying to practice this parallely. I have a question regarding data frame in pyspark. When I tried to create the variable "bad_loan" using withColumn and when (for the various cases of loan_status), the variable doesnt get created in the table, though I can see it in the dataframe. When I try to access this column using a select statement, I get an error. Can you please throw some light on this?

[saurabhjain]

Thank you for the wonderful video...I have a question as you mentioned you should use sortWithinPartitions to avoid expensive transformations when you know that the particular data is in one partition, how will you know that?? I am assuming that is only possible when you partition the data based on the values of that particular column.

[rajeevrajeev]

Do you have this Databricks page somewhere in git?

[nagarajuch]

Videos are all very informative.
Is there anyway we can sort based on more than one attribute? eg: Country Ascending and Date Descending

[AkshayKumar-xosk]

@AIEngineering - Thanks a lot for your video. May I kindly check all your spark video codes are based on python? You don't use scala/java? Whatever we do in scala/java can also be done using python?

[deepakparamesh]

very nice explanation you please upload the code, sir?

[kketanbhaalerao]

Please
provide your GitHub link and also provide corona data and twitter data