Saving PySpark DataFrame AS One .CSV File | Big Data

In PySpark, the write method doesn't save the dataframe as a single csv. In this video I will talk about many ways of solving this problem and compare them in terms of memory and speed.

[kernelab]

you have a good tv voice. you should be commentating a science documentary or something

[theunwaveringkeynote]

very straightforward, thanks for sharing!

[hoangng]

This trick is a lifesaver :) Thanks a lot!!!

[kernelab]

first cat command in my opinion is not needed too, just do cat ./cleaned_data/*.csv > data.csv