Apache Spark | Spark Performance Tuning | Spark Optimization Techniques { Filter on Date }

Apache Spark | Spark Performance Tuning | Spark Optimization Techniques { Filter on Date }

In this video, we will learn a spark performance tuning technique. We will have a demo on PySpark and Spark Scala to understand the working of optimized code while using the date condition in filter logic.

Epoch Unix Converter:

Dataset Used:

Blog link to learn more on Spark:

Linkedin profile:

FB page:

#apachespark #spark #bigdata

[ravikirantuduru]

Make videos on the predicate push down

[shaikshavalikadapa]

I feel both are taking almost same.
Here it is taking time like to create unix_column and then for filter.
Where as above it was only taking time for filter.
Can some explain me these things please.?

[guptaashok]

good one. i am thinking, why catalyst optimizer is not doing it implicitly where ever date filter is applied.

[backdoorguy]

I don't think this is optimization, u have to consider the time taken to convert the time format to Unix timestamp as well...

[gothams]

Shahul, what is difference b/w
finance_df \
.repartition(2) \
.write \
.partitionBy("id") \
.mode("overwrite") \
.option("header", "true") \
.option("delimiter", "~") \
.csv("s3a://" + + "/fin")

and

finance_df \
.repartition(2, 'id') \
.write \

.mode("overwrite") \
.option("header", "true") \
.option("delimiter", "~") \
.csv("s3a://" + + "/fin")

[yogeshwarangovindarajan]

Shall you please post videos for how to install spark by its source code?

[srmanojInd]

Hi Azar.im shalini...continously following ur tutorials.it really awesome.keet it up...i have got responses for my queires which i raised...thanks for that. I have an another query. In pyspark i have 500 columns in a file, while i read or i wanted to take only 200 out it. I knew how we can limit rows...for columns i dont have an idea.can u pls help me for this when u get time...thanks

[sushantshekhar]

Hi Azarudeen, can you please help me with the ambiguity problem, I have pinged you on FB messenger also.