Scikit Learn Incorporation - p.16 Data Analysis with Python and Pandas Tutorial

In this Data Analysis with Pandas and Python tutorial series, we're going to show how quickly we can take our Pandas dataset in the dataframe and convert it to, for example, a numpy array, which can then be fed through a variety of other data analysis Python modules. The example that we're going to use here is Scikit-Learn, or SKlearn. In order to do this, you will need to install it:

pip install sklearn
From here, we're almost already done. For machine learning to take place, at least in the supervised form, we need only a couple things. First, we need "features." In our case, features are things like current HPI, maybe the GDP, and so on. Then you have "labels." Labels are assigned to the feature "sets," where a feature set is the collective GDP, HPI, and so on for any given "label." Our label, in this case, is either a 1 or a 0, where 1 means the HPI increased in the future, and a 0 means it did not.

[기바랜드]

Really appreciate all the effort you gave to us.
Not just for this series, I'm giving so much help from your channel.
Hope your channel spread much wider so that more people learn from the perfect teacher.

[ozandundar]

FYI: cross_validation is model_selection now.
Great tutorial, thanks a lot

[gaslight]

For anybody else watching, the benchmark accuracy (selecting all increase or all decrease in future HPI) is 45% or 55% depending on what you choose so this is a good improvement!

FYI harrison, on your website you rename the HPI Benchmark to "US_HPI" from "United States" but in the code posted for this video you still refer to it as "United States".

[EvilSpeculator]

It literally took me weeks to write a sparse list parser in C# that was feeding into libSVM. Python does it in 20 minutes - awesome.

[EranM]

This is how you quickly investigate your results:

1) I would suggest to train until a certain point in time and test after that point. After all, this is a sequential data and not random observations.

I've tested the score and its looks like this:

clf.score: 0.703125 (need to read implementation to understand this score)
AUC: 0.758748674443
Precision: 0.55


2) I would use an AUC metric to evaluate how good this model learns.
3) After using a precision (tp / (tp + fp)) and got such a low score of 0.55 I decided to print a confusion_matrix.
4) confusion_matrix results:
0 1
0 | 23 18
1 | 1 22

Which means i can predict 0 with precision of 95% and predict 1 with precision of 55%.
So this model is REALLY GOOD for predicting FALLS, and close to RANDOM in predicting RISES.

Conclusion:
Model is good to predict falls of HPI.

All in all, GREAT SERIES sentdex !!!

[jasdnbsad]

What an amazing series, thanks.
A great introductory course to get started with Pandas.

Looking forward to more great series, such as Data Visualisation 👏

[sukumarhonkote]

finished the pandas tutorial. it was awesome. thanks

[devonsdistendedgut]

Amazing tutorial series man! Just finished it :)

[fabhlc]

Love the "WHELP!" attitude at the end haha

[jhillyt]

Just got through the set of lessons; great stuff. Thanks.

[pranavarora]

Thank You Sentdex. It was a really wonderful tutorial.

[eastwoodsamuel]

Awesome tutorial series....Thank you..

[gauravsinsinwar]

brilliantly explained ...loved the pace

[shashankgupta]

Awesome tutorial. Thanks for all the help :)

[gianlucapontoni]

Thank you for the tutorials! love you man

[tayebsaadi]

Thank you for your videos and your teaching

[eastwoodsamuel]

X = np.array(housing_data.drop(columns=['US_HPI_future', 'label']))
It works now

[WilliamEllison]

God your smart. Going to have to go over this again

[ConnerDKnox]

Hi Harrison! Big fan of your videos, I hadn't had time to work through all the code on my own yet but was wondering what sort of accuracy you get if you simply always predict increases as the HPI has somewhat steadily increased over time? As best I can tell the scoring is the percentage of predictions that the model correctly predicted, so might it even make sense to look at the models score as (% correct model prediction - % correct always increase prediction)/(1-% correct always increase prediction)? Thanks a ton for the work you put into these videos!

[karanbania]

Hey Sentdex!
amazing series, still relevant after all these years.
but all the databases you have used from quandl are ofcourse no longer available, If you could maybe share files of those data(IF you still have them), on your site maybe we could work along.
Thanks!