Python Tutorial : Outliers in Credit Data

---

Now that we've performed some basic exploration of the data, let's take a closer look at some of the columns and begin preparing it for modeling.

As with any machine learning problem, data preparation is the first step. But why? When our data is properly prepared we reduce the training time of our machine learning models.

Also, prepared data can also have a positive impact on the performance of our model. This is important because we want our models to predict defaults correctly as often as possible.

Consider this ROC chart. This shows the accuracy of three different models on the same data throughout different stages of processing. The light blue line represents a model trained on tidy and prepared data, while the orange line's model trained on raw data. The light blue line represents the most accurate model, because the curve is closest to the top left corner. We will see more graphs like this later when we check the accuracy of our models.

The first type of preparation we will look at is outlier detection and removal.
Unfortunately, data entry systems producing bad data is fairly common. If the data entry specialist was tired or distracted, they can enter incorrect values into our system.

It's also possible for data ingestion tools to create erroneous values in our data as a result of technical problems or system failures.

With outliers in our training data, our predictive models will have a difficult time estimating parameters like coefficients. This can cause our models to not predict as many defaults. Think of the coefficients as how much each column or feature is weighted to determine the loan status. Notice the coefficient differences in this example. It's possible that outliers in interest rate can cause that column to be weighted much more than normal. This will affect predictions.

One way we can detect outliers, is to use cross tables with aggregate functions like those from the previous video.

Here, we call crosstab on our credit loan data just like before to find the average interest rate.

For this example, we might expect to see the values on the left with our normal data. However, there could be some extreme outliers in the data which would result in the data on the right. This would cause problems with modeling. Imagine having an interest rate of 59,000 percent!

Another way to detect outliers is to use visuals. For this we can easily use plots like histograms and scatter plots, which we saw in the previous video.
Here, we can see that a couple records have a person's employment length set at well over 100. This would suggest that two loan applicants are over 136 years old! This, for now at least, is not possible.

So, we know outliers are a problem and want to remove them, but how? We can easily use the drop method within the pandas package to remove rows from our data.

In this example, we first use basic python subsetting to find rows with a person's employment length greater than 60. What this returns is the index position of that row in our data frame. From there, we call the drop method on our data frame to have it remove the rows in the data frame which match the index positions found earlier.

Now, we can see visually that the outliers have been removed according to our criteria, and the data looks much more realistic.

So, we've learned how to find and remove outliers from our data. Let's move on to some programming exercises to practice these skills!

#DataCamp #PythonTutorial #CreditRiskModelinginPython