Merge, Join, Append, Concat - Pandas

“There should be one—and preferably only one—obvious way to do it,” — Zen of Python. I certainly wish that were the case with pandas. In reading the docs it feels like there are a thousand ways to do each operation. And it is hard to tell if they do the exact same thing or which one you should use. That's why I made An Opinionated Guide to pandas—to present you one consistent (and a bit opinionated) way of doing data science with pandas and cut out all the confusion and cruft.

I'll talk about which methods I use, why I use them and most importantly tell you the stuff that I've never touched in my years of data science practice. If this sounds helpful to you then please watch and provide feedback in your comments.

This series is beginner-friendly but aimed most directly at intermediate users.

“Opinionated Guide–Combining DataFrames” GitHub repo:

Helpful links:

[luisabriful]

Thank you so much for your video! I've just started learning python for data science as self thought and found your videos by chance . I've watched several python tutorials so far but yours are particularly good. Clear explanation, good quality video, easy to follow, not too long, not too short and you go staringht to the point. Very well done and thanks for sharing! 👏

[Moiez]

Budding data analyst here currently learning Panda on udemy(just finished up with Numpy) - Love your opionated take on Pandas! I was confused on what cases and why I should use join, merge, concat etc.

[arhataria]

Lecture notes - Merge, Join, Append, Concat
1. Merge(=join)
-df.reset_index() - before or after merging -> can turn it into useful format
-specify the on=['a', 'b'], then pandas will pick it up
-inner/outer
-indicator=True -> show you what type of merge was done
-columns with the same name -> suffixes=('_left', '_right")
2. Concat(=append)
-multiple dataframes / merge(stack) rows together
-keys=['from1', 'from2'] : add an extra index(I'd like to know where that data source come from)

[DisturbeD]

3:42 I dont understand what index means sometimes people refere to index kinda like to a primary key autoincrement in sql but sometimes on something completly different im not a native english speaker and that is what tripps me up, to my eye at 3:42 theres 3 columns sex, smoker and tip, and when you do the merge u specify right_index=True how am i supposed to know what that referes to, lets assume theres more columns than just 3 how would u join the tables in that case with merge, i just started learning about pandas btw

[TopicalAuthority]

Skip the first 2 minutes before watching.

[divyanshrana]

please clarify that if we use merge for he same category and keys ...will the values assigned to them will also add up or not.

[aaronbaldwin]

Thanks for your video and examples. Very clear.

[annainsf]

Would like to see your video about "combine_first, merge_ordered, merge_asof".

[raml]

hey..! thanks for the video.i am just trying append a newsheet to excel using pandas.it should create a sheet by running the code and old sheets data should be there.can you guide?

[haneulkim]

Great video! do u know reason pandas created concat and append if they are the same thing?

[omerfarukelma]

hello, how can i add df coloumn to geodataframe?

[akzork]

I think I am gonna jump on the merging bandwagon.

[niteshmaruthi]

Great video - thank you. You can maybe add 3 way joins as well?

[李爽-sh]

is it possible to fully replace concat with merge？

[wesleymutalekapolyo]

Thank you very much, you have saved me. I have a question, How do you merge two data frames with different date format?