How Sort and Filter Works in Spark | Spark Scenario Based Question | LearntoSpark

In this video, we will discuss about the commonly asked scenario based interview question on apache spark. We will also have answer to this question with demo on this scenario question using PySpark.

Blog link to learn more on Spark:

Linkedin profile:

FB page:

Github:

[SunilPandey-u]

Hi @azarudeen, you are really doing a very good job. The content you are providing here is more worth than any paid course. Thanks and keep upload such informational video to help like us. After going through your video i have learnt many things as i have started giving interview and i was lacking in with such question to answer.

[TeKnowledGeeK]

the optimization is taken care by spark and the filter pushed to file scan in option A and B, that is the main reason for the performance increase. Any thing else have I missed ?

[kk]

Nice video. Cache used in C and D option is not necessary and it only takes up some memory. If dataframe is used multiple times then caching is ideal.

[vijayvardhan]

Thanks for sharing quick knowledge sharing concepts...appreciate your effort...keep sharing

[balajiveerasingam]

Thanks for the video, my understanding is catalyst optimiser will prepare different plans and based on the cost it finisalise the dag. Then even we put filter then sort or sort then filter. It will optimised to good plan

[kunalk]

@azarudeen thanks a lot brother your efforts are highly appreciated. You are a gift to those around you. 🙏

[abdulslogin]

Nice Video Azar...Keep doing it! It would be nice to give the datasets link to practice/test along with you.

[gkr]

Hi Azarudeen, well explained .. I would like to learn Pyspark. Could you please help me by sharing the proper installation steps

[prateekaryan]

Nice Video :) . I tried it with other dataset using Spark Scala on Databricks . For me Option C is taking less time than the other options.

[uditsethi]

Why we reqruire cache() function if have already have persist() to do the same memory storage of dataframe along with other storage options?

[brogames]

Awesome Video Bro!!! keep it up one day you will have 10 million subscriber

[AshwinKarki-oe]

can you provide the source data link for these questions?

[avi]

Shouldn’t option A and option B run in same time?
Both the physical plans are same.
The lag might be due to some other process running in the PC while executing option B.

[rikuntri]

Hey if the size of data is high. Than the caching latency might be compensated. What do you say