Lesson 11: Cutting Edge Deep Learning for Coders

We’ve covered a lot of different architectures, training algorithms, and all kinds of other CNN tricks during this course—so you might be wondering: what should I be using, when? The good news is that other folks have wondered that too, and have provided some great analyses of the pros and cons of various techniques in practice. We’ll be taking a look at a few highlights of these papers today.

Then we’re going to learn to GPU accelerate algorithms by taking advantage of Pytorch, which provides an interface that’s so similar to numpy that often you can move your algorithm onto the GPU in just an hour or two. In particular, we’re going to try to create the first (that we know of) GPU implementation of mean-shift clustering, a really useful algorithm that deserves to be more widely known.

To close out the lesson we will implement the heavily publicized “Memory Networks” algorithm, and will answer the question: does it live up to the hype?

[ericpb]

Lesson 11 video timeline:

00:00:30 Tips on using notebooks and reading research papers

00:03:15 Follow-up on lesson 10 and more word-to-image searches

00:07:30 Linear algebra cheat sheet for deep learning (student's post on Medium)
& Zero-Shot Learning by Convex Combination of Semantinc Embeddings (arXiv)

00:10:00 Systematic evaluation of CNN advances on ImageNet (arXiv)
ELU better than RELU, learning rate annealing, different color transformations,
Max pooling vs Average pooling, learning rate & batch size, design patterns.

00:27:15 Data Science Bowl 2017 (Cancer Diagnosis) on Kaggle

00:36:30 DSB 2017: full preprocessing tutorial, + others.

00:48:30 A non-deep-learning approach to find lung nodules (research)

00:53:00 Clustering (and why Jeremy wasn't a fan before)

01:08:00 Using Pytorch with GPU for 'meanshift' (clustering cont.)

01:22:15 Candidate Generation and LUNA 16 (Kaggle)

01:26:30 Accelerating K-Means on GPU via CUDA (research)

01:27:15 ChatBots ! (long section)
Staring with "memory networks" at Facebook (research)

01:57:30 Recurrent Entity Networks: an exciting area of research in Memory Networks

01:58:45 Concept of "Attention" and "Attentional Models"