Python Tutorial: Introduction to Exploratory Data Analysis

---
Yogi Berra said, "You can observe a lot by watching." The same is true with data. If you can appropriately display your data, you can already start to draw conclusions from it.

I'll go even further. Exploring your data is a crucial step in your analysis.

When I say exploring your data, I mean organizing and plotting your data, and maybe computing a few numerical summaries about them.

This idea is known as exploratory data analysis, or EDA, and was developed by one of the greatest statisticians of all time, John Tukey. He wrote a book entitled Exploratory Data Analysis in 1977 where he laid out the principles.

In that book, he said, "Exploratory data analysis can never be the whole story, but nothing else can serve as the foundation stone." I wholeheartedly agree with this, so we will begin our study of statistical thinking with EDA.

Let's consider an example.

Here, we have a data set I acquired from data dot gov containing the election results of 2008 at the county level in each of the three major swing states of Pennsylvania, Ohio, and Florida. Those are the ones that largely decide recent elections in the US. This is how they look when I open the file with my text editor. They are a little prettier if we look at them with in a Pandas DataFrame,

in this case only looking at the columns of immediate interest, the state, county, and share of the vote that went to Democrat Barack Obama.

We could stare the these numbers, but I think you'll agree that it is pretty hopeless to gain any sort of understanding from doing this. Alternatively, we could charge in headlong and start defining and computing parameters and their confidence intervals, and do hypothesis tests. You will learn how to do all of these things in this course and its sequel. But a good field commander does not just charge into battle without first getting a feel for the terrain and sizing up the opposing army. So, like the field commander, we should explore the data first.

In this chapter, we will discuss graphical exploratory data analysis. This involves taking data from tabular form, like we have here in the DataFrame,

and representing it graphically. You are presenting the same information, but it is in a more human-interpretable form.

For example, we take the Democratic share of the vote in the counties of all of the three swing states and plot them as a histogram. The height of each bar is the number of counties that had the given level of support for Obama. For example, the tallest bar is the number of counties that had between 40% and 50% of its votes cast for Obama.

Right away, because there is more area in the histogram to the left of 50%, we can see that more counties voted for Obama's opponent, John McCain, than voted for Obama.

Look at that. Just by making one plot, we could already draw a conclusion about the data, which would have been extraordinarily tedious by hand counting in the DataFrame.

Now let's review some of the basic ideas behind EDA with a couple exercises.

#PythonTutorial #DataCamp #Statistical #Thinking #Python #Exploratory #Data #Analysis