Python Pandas Tutorial 15. Handle Large Datasets In Pandas | Memory Optimization Tips For Pandas

Often datasets that you load in pandas are very big and you may run out of memory. In this video we will cover some memory optimization tips in pandas.

#️⃣ Social Media #️⃣

❗❗ DISCLAIMER: All opinions expressed in this video are of my own and not that of my employers'.

[ranjitprakash]

Thanks for the video, I was struggling with the memory issue in pandas and was suggested by peers to use Dask. However I think the best approach is to optimize the dataset as well by defining the dtype in conjunction with other avenues such as Dask. I will implement it on my project!

[satishpatil]

Hi sir, work with hugh side of data on excel, and use the VBA macro to manipulate it, and it takes very much time and sometimes macro get corrped, can you suggest me if I can do the same operations using pandas ? And print that data into Excel

[xxotto]

Thank you very much! I'm a new sub

[raghavs]

Hi Dhaval, I have a problem statement to build a DL model. Need your suggestions

How can I find differences between two images?? The first image is a master, the second image looks like master but there will be differences. I have to point out the anomalies building a DL model. Please suggest me your views. I have a discussion tomorrow..please help

[preeti]

Hi sir
i have around 10 million data and 50 columns all columns are needed for work
pandas give memory error at the time of reading it from db

[hayathbasha]

In some case, We might not have columns in csv on that case

Assume i have 16columns i need to use 3 columns then how to read efficiently

[hayathbasha]

Can u pls make a video of comparing two different csv using pandas

[ravitalaviya]

Hii sir
I have 3GB size of json file. I want to convert in csv file.
Please, help me with that.

[curiousstreamer]

Please make the video on bitcoin
Please 🙏🙏🙏