Pandas 2.0 : Everything You Need to Know

In this video I give an overview of pandas 2.0 and the main changes related to the apache arrow backend.

Timeline:
00:00 Intro
01:04 Legacy Numpy
02:49 Arrow Backend
03:44 Missing Values
04:33 Speed
05:47 Interoperability
07:42 Arrow Data Types

Check out my other videos:

Links to my stuff:

::::::::::::::::::::
Music: Head Candy - William Rosati
::::::::::::::::::::

[BabakFiFoo]

Rob, you are making the best videos. I am always watching them and learn new stuff. I learned python and pandas myself, and your videos helped me improve them several times fold. thank you!

[irfanshaikh]

I'm not sure what'd i'd be without you and this YT channel Rob. Thanks for being and amazing teacher to a rookie like myself
lots of respect and love

[TheMacister]

Thanks Rob! Couple months seeing your videos and your content is on the spot ! 🎉

[edwardCYHsu]

I am working exactly on my programming assignment of the date manipulation and it is very confusing. With the PyArrow datatype support, it is a lot easier from now. Thank you for highlighting the significance to us. You are a saint.

[igordeoliveirabarrosfaluhe]

Thank you Rob! Your videos are always useful with a such nice flow!

[mrdbourke]

Epic video Rob!

I use pandas everyday and glad to hear it’s getting faster!

[chintansawla]

Thanks for sharing the update! Very well articulated

[gustavojuantorena]

Great video Rob! I think Pandas will continue to be very useful in the data science community.

[rayankhan]

I particularly liked the conversion between the Pandas with Arrow backend and polars.. maybe you should make a separate video on it

[farazahmed]

thank you so much for such a kind information.

[michaelsoldmann]

Hi Rob, thank you for the content. I'd be interested in seeing a video on using the old np.where() and np.select() in the new pandas 2.0.

[AFlockOfToasters]

Good job pronouncing Interoperability!

[vincentverdugo]

Hey, your DS videos are awesome! I was using ChatGPT to learn more about Apache Arrow, Polars, etc from all your videos. Can you do a coding livestream or video about bioinformatics data like biological sequence data or drug development data? Thank you!

[Micro-bit]

Hey Rob, Great stuff! .. I moved my data transformation app to PD2.0 and I have a problem with data conversion. When Im converting data from int[pyarrow] to string using astype(str) im loosing pyarrow and pandas convert it to object :/. I cant find the tool to do it properly. All best!

[incremental_failure]

I was about to switch to Polars but now with Pandas Arrow dtypes, I need to do more research. Speed is one thing but lazy processing and memory usage are very important.

[pietraderdetective]

ah yess the much needed Pandas improvement!
this is what I like in a competitive environment..it sharpens everyone involved.
I agree with what has been said in the comment section: now we can use Pandas to handle large dataset properly.

especially that 20x speed improvement when reading data using Pyarrow backend is so good!

[franky]

I would be interested to know how much time you lose when converting from pandas to polars and back to pandas?

[nikjs]

This int converting into float has been a long time PITA for me. This upgrade will be much welcome.

[x-axis]

Hey Rob!
That text editor looks awesome. Which one is it?

[KenJee_ds]

Awesome! Are there any drawbacks to using the pyarrow backend?