How to Optimize and Speed Up Pandas

3 Simple ways to optimize pandas

* Optimize datatypes of dataframe
* Use built-in functions
* Search for smart alternative
* Do tests

Bonus tips:

Use NumPy arrays/matrix
# Convert the frame to its Numpy-array representation. Deprecated since version 0.23.0

#Return a Numpy representation of the DataFrame.

# Convert the DataFrame to a NumPy array
Optimize data when you read it
Convert dates to Datetime
Loop Pandas data in smart way ( iterrows, itertuples, zip )

for r in zip(t['a'], t['b']): # 0.005645036697387695

---------------------------------------------------------------------------------------------------------------------------------------------------------------

Data set: Stack Over Flow 2018 insights

---------------------------------------------------------------------------------------------------------------------------------------------------------------
Topics

pandas expand list column
value_counts for list column for a dataframe

Bonus

combine head and tail
slicing iloc with range
value_count on all columns
sum per column
do a sum of several columns
sum all columns with iteration
be careful when you chain operations with pandas

---------------------------------------------------------------------------------------------------------------------------------------------------------------
Bonus resources

March 2019 Machine Learning Study Path

---------------------------------------------------------------------------------------------------------------------------------------------------------------
Code store

Socials

If you really find this channel useful and enjoy the content, you're welcome to support me and this channel with a small donation via PayPal.

[vrbaac]

thanks for this video i learned so much... i see ur sample has thousands of rows... while i am working on a daraset with only 500 rows but doing pandas profiling report causes the browser to have "not enough memory" error 😢