How to apply Filter in spark dataframe based on other dataframe column|Pyspark questions and answers

Welcome to DWBIADDA's Pyspark scenarios tutorial and interview questions and answers, as part of this lecture we will see,How to apply Filter in spark dataframe based on other dataframe columns

[Maceta]

Finally found a good explanation. Thanks so much!

[adeputri]

Hi thank's for the great explanation
I'm currently working on a very large datasets on my pyspark.
But i'm having trouble filtering the datasets using your first method. Can you tell me if its the problem with the memory or is it the problem with my enormous datasets itself?
The error I received was : java heap space

[antoniyc]

which is more optimal, use join or convert to list ?