Process HUGE Data Sets in Pandas

Today we learn how to process huge data sets in Pandas, by using chunks.

◾◾◾◾◾◾◾◾◾◾◾◾◾◾◾◾◾
📚 Programming Books & Merch 📚

🌐 Social Media & Contact 🌐

[Opento]

I can't always follow everything he says, cause he moves pretty quick and throws a lot at you, but he's always straight to the point, no fluff, and innovative.

I always glean more things to look up after hearing it from NeuralNine first.

[leythecg]

wie immer top content perfekt präsentiert!

[goku-npbk]

why would you use csv format instead of parquet or hdf5 for large datasets?

[lakshay]

Your explanation is very good can you do a video on the Python project that else the position of an eye

[Ngoc-KTVHCM]

In excel file, method "pd.read_excel" has no parameter "chunksize", how to handling the big data in many sheet in excel? Please help me!

[tcgvsocg]

i was litteraly watch a video when you post a new video...i like that!(8)

[thisoldproperty]

I like the simplicity. Wonder if a similar thing could be done with sql queries given they usually store incredibly large datasets.

[siddheshphaple]

How can I connect database in python, and how to optimise it if I have 60L+ records in it

[TomKnudsen]

Thank you.. Could you please make a tutorial on how you would stip out certain elements from a file that is not your typical "list", "csv" or "json".. Find this task to be the most confusing and difficult things you can do in Python. If needed, I can provide you with a text file which include information about airports such as runways, elevation, etc. Perhaps there are some way to clean such file up or even convert it to a json/excel/csv etc.

[maloukemallouke]

thanks but how you deal with depending row like times series data or observations like text where context correletead to row?

[wzqdhr]

The hard part is how to append the new feature back to the original dataset without loading them in one shot

[artabra]

OMG tnx im trying to open csv file with million data then my pc collapse so i find some i9 computer with 16gb ram to open it thanks now i can open big files using pandas.

[uzeyirktk]

how we can further work on it. Suppose if want to use groupby function on column [ 'A '].

[hynguyen]

i'm a simple man, i see vim, i press like

[FabioRBelotto]

Can we use each chunk to spawn a new process and do it in parallel?

[tauseefmemon]

Why was the RAM increasing? should not it stop increasing once the data is loaded?

[RidingWithGerdas]

Or with really huge datasets, use Koalas, interface is pretty much the same as pandas