Concatenating and Appending dataframes - p.5 Data Analysis with Python and Pandas Tutorial

Welcome to Part 5 of our Data Analysis with Python and Pandas tutorial series. In this tutorial, we're going to be covering how to combine dataframes in a variety of ways.

In our case with real estate investing, we're hoping to take the 50 dataframes with housing data and then just combine them all into one dataframe. We do this for multiple reasons. First, it is easier and just makes sense to combine these, but also it will result in less memory being used. Every dataframe has a date and value column. This date column is repeated across all the dataframes, but really they should all just share the one, effectively nearly halving our total column count.

When combining dataframes, you might have quite a few goals in mind. For example, you may want to "append" to them, where you may be adding to the end, basically adding more rows. Or maybe you want to add more columns, like in our case. There are four major ways of combining dataframes, which we'll begin covering now. The four major ways are: Concatenation, joining, merging, and appending. We'll begin with Concatenation.

[EvilSpeculator]

I haven't had this much fun following a tutorial in a long time. <thumbs up>

[Angel_Fontalvo]

so there I am googling a newbie pandas related question and I got so excited when I saw that the top result was for one of my fav youtubers!

[claudiospera]

Hi, I am just starting the series of videos on python and pandas. You are maying teacher and you're SO didactic!! All the best

[SerenityAZ]

Thanks a ton for posting these tutorial videos on Python.

[Dutchtraordinary_Living]

Absolutely amazing. Clear and straight to the point:)

[vikaskundra]

One of the great tutorial series out there. Thanks man!!

[ananthchief]

thanks a ton. these were extremely helpful. Keep adding more videos to the list, more on data analysis, statistics and Pandas :) Thanks again, love your tutorials :)

[laurafosci]

I type ''how to add a row to a dataframe in python'' and you come up. So glad to see you again

[matheusc]

Thank you for your work on this series!

Wonderful tutorial! Great job and thanks a lot for sharing!

[RaunakJoshi]

Lot of stuff to learn from you man. Such great tutorials. Keep up the good work. God bless. \m/ Peace.

[baijuthomas]

I was looking for the database video .on how to import a daily multiple excel files, perform analysis, create a report and then export all the data into a database that contains year to date data

[gaslight]

Keep the videos comin Harrison! Do you know what the next financial topics you will cover are yet?

[veggiet]

These videos are really cool, but this one I don't get, it appears that concat and append do the same thing, but in a slightly different manner, concat creating a new dataframe out of N number of dataframes and append simply adding one to another one. The questions I'm left with are: When would you use this? When would you prefer one over the other? Is there a collective preference among the pandas community (similar to the 'import pandas as pd' phrase) ?

[jsx]

Can you show how to make a new DF in which you have a single index (1, 2, 3?) and you populate with certain columns from other DFs? For instance, what if we wanted a DF with 2001-2008 but wanted a standard index starting at zero, then populate with columns from different dataframes? I understand the 'easy' examples. But when I have to do something like this? Lost.

[Videoomanny]

Hi, sentdex, thanks for your videos, you are a great teacher!

I have a question about pandas. What if I want to add or substract from ane column the same number. For example, I have a dataset with values [8, 3, 16, 29] and I want to add to all of these numbers, for example 2. How can I get dataset [10, 5, 18, 31]?

Thanks in advance!

[tingfengxian]

I use the statement ""df4 = df1.append(df3)"", which is normal

[alsherbin]

Anytime I append/concatenate with adding columns (like df3 to df1), I get the error message RuntimeWarning: invalid value encountered in greater
(abs_vals > 0)).any()". I am using Anaconda3 in Windows10. Please explain!

[sdoken]

Thanks a lot sentdex. Question:
What you are doing at 6:01 seems a bit weird to me. You are appending a Series to a Dataframe. Yes it works and everything so obviously it is not wrong but most people think of pandas dataframe as tables and Series as a Column. Am I Wrong? So treating this Series s as a new *row* to a dataframe is kind of counterintuitive to me and makes me feel weird inside lol.
I had never seen series used that way but I guess they can be. Is it okay if I think of this as a special gimmick in python and continue to think if Series as columns?
Thanks for all your work sentdex.

[kashishshah]

what is the difference between concat and append? if they do the same thing