Lesson 4: Deep Learning 2018

We complete our work from the previous lesson on tabular/structured, time-series data, and learn about how to avoid overfitting by using dropout regularization. We then introduce natural language processing with recurrent neural networks, and start work on a language model.

[zhuyixue]

The first part is about DL for structured (tabular) data, the second part starting at 1:23:35 is about DL for NLP

[toequantumspace]

Jeremy thanks for your tutorials and fast.ai. I submitted my results in kaggle histopatologic cancer detection competition and I am at 148 place out of 300! On my first ever neural network! Your approach works so well.

[geishachlen]

00:00:04 More cool guides & posts made by Fast.ai classmates
“Improving the way we work with learning rate”, “Cyclical Learning Rate technique”,
“Exploring Stochastic Gradient Descent with Restarts (SGDR)”, “Transfer Learning using differential learning rates”, “Getting Computers to see better than Humans”

00:03:04 Where we go from here: Lesson 3 -> 4 -> 5
Structured Data Deep Learning, Natural Language Processing (NLP), Recommendation Systems

00:05:04 Dropout discussion with “Dog_Breeds”,
looking at a sequential model’s layers with ‘learn’, Linear activation, ReLu, LogSoftmax

00:18:04 Question: “What kind of ‘p’ to use for Dropout as default”, overfitting, underfitting, ‘xtra_fc=’

00:23:45 Question: “Why monitor the Loss / LogLoss vs Accuracy”

00:25:04 Looking at Structured and Time Series data with Rossmann Kaggle competition, categorical & continuous variables, ‘.astype(‘category’)’

00:35:50 fastai library ‘proc_df()’, ‘yl = np.log(y)’, missing values, ‘train_ratio’, ‘val_idx’. “How (and why) to create a good validation set” post by Rachel

00:39:45 RMSPE: Root Mean Square Percentage Error,
create ModelData object, ‘md =

00:45:30 ‘md.get_learner(emb_szs, …)’, embeddings

00:50:40 Dealing with categorical variables
like ‘day-of-week’ (Rossmann cont.), embedding matrices, ‘cat_sz’, ‘emb_szs’, Pinterest, Instacart

01:07:10 Improving Date fields with ‘add_datepart’, and final results & questions on Rossmann, step-by-step summary of Jeremy’s approach

Pause

01:20:10 More discussion on using Fast.ai library for Structured Data.

01:23:30 Intro to Natural Language Processing (NLP)
notebook ‘lang_model-arxiv.ipynb’

01:31:15 Creating a Language Model with IMDB dataset
notebook ‘lesson4-imdb.ipynb’

01:31:34 Question: “So why don’t you think that doing just directly what you want to do doesn’t work better?” (referring to the pre-training of a language model before predicting whether a review is positive or negative)

01:33:09 Question: “Is this similar to the char-rnn20 by karpathy?”

01:39:30 Tokenize: splitting a sentence into an array of tokens

01:43:45 Build a vocabulary ‘TEXT.vocab’ with ‘dill/pickle’; ‘next(iter(md.trn_dl))’

The rest of the video covers the ins and outs of the notebook ‘lesson4-imdb’, don’t forget to use ‘J’ and ‘L’ for 10 sec backward/forward on YouTube videos.

02:11:30 Intro to Lesson 5: Collaborative Filtering with Movielens

[kelvinzhu]

structured (tabular) data: 25:05 - 01:23:35

[pku]

MOOC at its best. Thank you very much.

[yangyuchendg]

Regarding the random feature dropout part, can we delete data in every other row/column instead of randomly delete for dropout =0.5 ?

[naviarh]

Jupyter notebook для Google Colaboratory с комментариями на русском языке
Jupyter notebooks for Google Colaboratory with russian comments

[programtesting]

Links discussed in the video are:

[Fanaro]

Intuitively, if you're going to apply Dropout, you should start with the last layers, shouldn't you? Because the first layers are generally trying to classify simpler features, which are less likely to be overfitted.

[austinmw]

Did he impute with with median from entire df or does it impute Val with train median?

[Darleer]

Anyone knows if there's a lecture/forum post which talk more about text generation using fastai? I noticed Jeremy mentioned that he won't be going into it too much but I kinda want to learn more about that particular topic

[isbestlizard]

I'm lost at 1:52:00... you're presenting the network the first 75 tokenised words of a review, and training it to predict the 2nd word? How and why does that work? I could understand if you were predicting the 76th word, given the first 75, but not the 2nd. It sounds pretty easy to learn how to predict the 2nd word in a sequence, if you're told the first 75... you just repeat the 2nd word you were told, and ignore words 1 and 3 through 75?

[peterklemenc]

At 1:49:50 there's the matrix that we split. What I'm confused about is why is it 10 Million for the Y axis, times 64 ... and not 1 million. ... That gives us 640 million in total ... What am i missing?

[nalinm]

"but wait, there's more!" haha