Spark Scenario Based Question | Read from Multiple Directory with Demo| Using PySpark | LearntoSpark

In this video, we will see a spark interview question. This Scenario based question has now a days become common question in Spark interview. We will discuss on the Spark Optimized way to read many files from multiple directory using PySpark.

Linkedin profile:

FB page:

Sample Dataset and Code Snippet:

Blog link to learn more on Spark:

[bunnyvlogs]

First of all a great....namaskar....for ur are a life saver....My sincere request is to make end to end pyspark application with unit test cases...in databricks environment itself...so it will help us in real world senarios

[priyankas]

Very nicely explained. All your videos are good

[mohitupadhayay]

Is there a way to understand the reason behind why some records went into _corrupt_record column if the number of columns are very large?

[shilpasthavarmath]

Which notebook is use for scala programming?

[shilpasthavarmath]

Hi.. Please let me know if scala developers are using jupiter for spark coding... If no, what is used for scala programming?

[kaustubhjoshi]

Hello, how do u connect your Jupyter notebook to the hadoop node

[muddy]

Hi - can we use manifest file to list multiple directories and use manifest file in DF read API?

[AmitKumar-lcsm]

In how many ways we can load data from RDBMS to Hdfs? Please answer and all your videos are really helping me in interviews. Thanks

[surendrag]

I want to read multiple csv files in same directory when columns are dis order ? pls suggest me to read best way

[ravikirantuduru]

Can you do video on similar function foldleft which is there in scala in pyspark to apply on some columbs.

[reddymaestro]

I would like to learn spark from you how can I. Get it