Lesson 4: Deep Learning 2019 - NLP; Tabular data; Collaborative filtering; Embeddings

The basic steps are:
1. Create (or, preferred, download a pre-trained) *language model* trained on a large corpus such as Wikipedia (a "language model" is any model that learns to predict the next word of a sentence)
2. Fine-tune this language model using your *target corpus* (in this case, IMDb movie reviews)
3. Remove the *encoder* in this fine tuned language model, and replace it with a *classifier*. Then fine-tune this model for the final classification task (in this case, sentiment analysis).

After our journey into NLP, we'll complete our practical applications for Practical Deep Learning for Coders by covering tabular data (such as spreadsheets and database tables), and collaborative filtering (recommendation systems).

Then we'll see how collaborative filtering models can be built using similar ideas to those for tabular data, but with some special tricks to get both higher accuracy and more informative model interpretation.

This brings us to the half-way point of the course, where we have looked at how to build and interpret models in each of these key application areas:
- Computer vision
- NLP
- Tabular
- Collaborative filtering

For the second half of the course, we'll learn about *how* these models really work, and how to create them ourselves from scratch. For this lesson, we'll put together some of the key pieces we've touched on so far:
- Activations
- Parameters
- Layers (affine and non-linear)
- Loss function.

We'll be coming back to each of these in lots more detail during the remaining lessons. We'll also learn about a type of layer that is important for NLP, collaborative filtering, and tabular models: the *embedding layer*. As we'll discover, an "embedding" is simply a computational shortcut for a particular type of matrix multiplication (a multiplication by a *one-hot encoded* matrix).

[TheEskimosPantry]

I clapped at the end of the lecture, sat at my desk... What a fantastic teacher!

[Carbon-XII]

I like your lessons more than Netflix shows.

[DrEhrfurchtgebietend]

He hints at this but does not say that there is not a good feature importance for nets. Just use permutation feature importance. Also partial dependence plots are great for understanding relationships to targets.

[englishtobinary]

Great lecture! Huge thanks to you and your team!

[anujadhar]

Hi, the courses are very helpful. Can I get the Lesson notes and Jupyter notebook?

[vladimirgetselevich]

Should not matrix have the same number of columns as vector rows for correct multiplication? Or does he assume the matrix to be transformed before doing the multiplication?

[kathrynabeyta]

1:00:00
Why is this video reuploaded?
Happy belated birthday, Jeremy!

[leynerbejarano]

I am having a problem with the tabular lesson, lr_find, also the train_loss and valid_loss appear nan

[stephengleason]

Why is this video reuploaded?
ablan inglis

[jonatani]

20:49
I think that you should never use the test set in any of your trainings. That is cheating and you won't have a genuine metric about the real world performance of the final model. (Your model's LM will perform better on the test set than on a never seen text in the wild.)
It's a gut feeling and would have to test this hypothesis.