Turning multiple CSV files into a single pandas data frame

[sloughpacman]

Your material has the habit of provoking thought, which often leads me off on exploratory tangents for a couple of hours. Thanks for wasting my day, Reuven 😀

[learner]

I never knew you could have a list of dataframes. Thank you!

[ManjeeriG]

Thank you so much for this tutorial. I have gone through multiple tutorials but this one is the easiest and (ofcourse) smartest way to combine multiple csv into the one. 😊😊 You got a new subscriber.

[matthewhsu]

Learn something new today. Thank you so much!

[Merqos]

Thank you so much for this example! This made me save so much time!

[samogo]

Excellent, worked fine for me also, thank you!

[silvajonatan]

Hi, fantastic infomation. Very didatic. Thanks a lot.

[mikej]

Nice!!! - I've seen several ways to solve this problem and this is the most efficient I've personally came across!
Question for you, In [13], there will be times I want to use this approach because I want to verify the number of rows in each file....how would you do it? TIA!

[traal]

Would this also work with a generator expression passed into pd.concat? That would look nicer and probably save some memory. 😊

[guocity]

thats really helpful, what about reading multiple csv file, read new csv file ignore repeated rows

[varunsharma]

Worked for me. Earlier I was using excel power query to join multiple csv's and then importing them to pandas, but it had a limitation of max 10million rows in excel. This tutorial is very helpful. Thanks and subscribed. One issue is that I am facing in this code is that I want to add my individual csv file names as a column in concatenate dataframe that is missing in this code.Any tips

[awaisraza]

I have a main folder and in that main folder there are 35 sub folders. and their names are like S01, S02... S35. And each folder has a dataset of same structure. How i can concatenate that data into one dataframe?

[regal]

What if the datasets doesnt haw anything in common, like one is geological data, one is survey data, one is market analysis and each of them has a massive number of null values . Also the unique ids are different for example, one table has SLT20284903 and some others just numbers . What do we do then ?

[tomtaetz]

How would you go about this when the csv files don’t all contain the same column names(some different some the same)?

[koosschutter]

Where can I download some of these large files for testing? I want to split or combine some CSV files but with my basic laptop can't load everything in memory and I need all the columns. Me and chatgtp came up with appending files which is slow. I'm not a data scientist, I just want to split and combine csv files and test my and my pc's capabilities.

[FletchersHeating]

Thanks for this video!
Is there a tutorial or information on how to do the same but for multiple data frames? ie. one csv = one dataframe

Many thanks

[nvduk]

If one of the csv files is blank inbetween then it's breaking for loop. How to avoid that?

[KingofDiamonds]

I can't get this to waork for me. I need help.