How can I split a large csv file (7GB) into smaller Csv files using Python| Stack overflow Question

code:

[jasonp]

You are my hero. Whenever I have a question and need it answered with solid examples: your channel always appears with a video answering my question. A treasure trove for any data practitioner. Thanks.

[nearflatline]

Thank you so much this was exactly what I need for slicing up an overly large .csv file. I am glad to have come across your video.

[prateeksarangi]

Thanks !! Helps a lot from Zipfile module !!

[naduarte]

I tried it for a XLSB file of around ~4GB after converting it with excel to CSV UTF-8, but it doesn't work. Don't know if Excel messed things converting, but seems like it has to much information that Pandas couldn't open, even trying to read it as Bytes or with readlines, or other libraries.

I tried all the possible pd.read_XXXX methods of Pandas after trying all kinds of formats and its specific engine as pandas argument. Tried XLSB, CSV, ODS, XLS, XLSX, etc and neither of them were able to be read by Pandas. Excel opened the file in around 5-10 minutes, and Pandas couldn't read it, even after 90 minutes (My laptop has i9 12gen + NVME1TB + 32Gb RAM).

[mohitbhagat]

Hey, Soumil thanks for this Awesome code.

[rishichaturvedi]

Will it work if the chunk remaining at last is less that the chunk size we defined?, And is the procedure same for xlsx file format?

[mrunal]

my large csv file is in Portuguese language, and i need to get them in English. can you tell me how? i have tried with Translator from googletrans module, that lead to errors in my case that too after a large span of waiting time.

[ohp]

Noice … I have a 600 gb file .. I will try this and post the time taken.

[zulfiqurhaider]

You are just splitting the first 100 data in one CSV. How can you shuffle and store 100 randomly selected data in one CSV?

[srikantaghosh]

Can you create a video on scrapping a web page and store the data into S3 to use it in athena?

[georgesmith]

Just open the file and use a generator for reading line by line

[neuromyoprosthetics]

Where to keep sample file who's is to be splitted into many

[ravitalaviya]

Hiii sir
I have 3GB size of json file how to convert in CSV

[sonnguyenthanh]

How could I avoid `the number order` at first column in splitted file?

[neuromyoprosthetics]

When make videos, always remember that every one not geneious as yourself to understand so fast rapid, go slow dear