Python Tutorial: Introduction to model validation

---
Hello, my name is Kasey Jones - And welcome to this course on model validation. Let's get started!

So what is model validation?

Well, model validation consists of various steps and processes that ensure your model performs as expected on new data. The most common way to do this is to test your model's accuracy on data it has never seen before (called a holdout set). If your model's accuracy is similar for the data, it was trained on, and the holdout data, you can claim that your model is validated. However, model validation can also consist of choosing the right model, the best parameters, and even the best accuracy metric.

The ultimate goal of model validation is to end up with the best performing model possible, that achieves high accuracy on new data. Before we begin exploring model validation, let's review some basic modeling steps using scikit-learn.

Modeling in Python follows a simple procedure, regardless of the type of model you are constructing. Whether you are a seasoned scikit-learn veteran or new to building models with this module, let's take a quick look at these steps.

First, we create a model by specifying the model type and its parameters. In this case, we are creating a random forest regression model with RandomForestRegressor().

Second, we fit the model using the .fit() method. This method has two main arguments. X, an array of data used in the model as training data, and y, an array of response values matching the size of the X array. When .fit() is used, the model parameters will be printed in the console.

To assess model accuracy, we generate predictions for data using the .predict() method.

And lastly, we look at the accuracy metrics. Here we are comparing the model's predictions (the variable predictions) and the actual responses, y_test.

Future lessons and exercises will be devoted to accuracy metrics, as they are a vital component to model validation. For this current example though, we are looking at the mean absolute error. This function takes two arrays as arguments. The true values, y_true, and the predicted values, y_pred, and returns the mean absolute error between them.

This process of generating a model, fitting, predicting, and then reviewing model accuracy will be repeated throughout this course. If you are unfamiliar with these steps, you should consider taking the prerequisite courses. They will go into more detail about using Python and performing these modeling steps.

Throughout this course, we will use fivethirtyeight's ultimate Halloween candy power ranking dataset several times. This dataset contains 85 different candies, data on their various characteristics, and a column specifying how often that candy was selected in a head-to-head match-up with other candies. This column is a win-percentage and contains values between 0 and 100.

Model validation's main goal is to ensure that a predictive model will perform as expected on new data. Obtaining predictions for training data (or seen data) and testing data (or unseen data) is coded in the same way and uses the predict() method.

Generally, models perform a lot better on data they have seen before, as unseen data may have features or characteristics that were not exposed in the model. If your training and testing errors are vastly different, it may be a sign that your model is overfitted. We will use model validation to make sure we get the best testing error possible.

Let's see why model validation is so important by looking at an example of training and testing accuracies.

#PythonTutorial #DataCamp #Model #Validation #Python