Lesson 13: Cutting Edge Deep Learning for Coders

One application of deep learning that has progressed perhaps more than any other in the last couple of years is Neural Machine Translation. In late 2016 it was implemented by Google in what the New York Times called The Great A.I. Awakening. There’s a lot of tricks needed to reach Google’s level of translation capability, so we’ll be doing a deep dive in this lesson to learn nearly all the tricks used by state of the art systems.

Next up, we’ll learn about Densenets, which in July 2017 were awarded the CVPR Best Paper award, and have been shown to provide state of the art results in computer vision, particularly with small datasets. They are very similar to resnets, but with one key difference: the branches in each section are combined through concatenation, rather than addition. This apparently minor change makes a big difference in how they learn. We’ll also be using this technique in the next lesson to create a state of the art system for image segmentation.

[ericpb]

Lesson 13 video timeline:

00:00:10 Fast.ai student accepted into Google Brain Residency program

00:06:30 Cyclical Learning Rates for Training Neural Networks (another student's paper)
& updates on Style Transfer, GAN, and Mean Shift Clustering research papers

00:13:45 Tiramisu: combining Mean Shitft Clustering and Approximate Nearest Neighbors

00:22:15 Facebook AI Similarity Search (FAISS)

00:28:15 The BiLSTM Hegemony

00:35:00 Implementing the BiLSTM, and Grammar as a Foreign Language (research)

00:45:30 Reminder on how RNN's work from Lesson #5 (Part 1)

00:47:20 Why Attentional Models use "such" a simple architecture
& "Tacotron: a Fully End-To-End Text-To-Speech Synthesis Model" (research)

00:50:15 Continuing on Spelling_bee_RNN notebook (Attention Model), from Lesson 12

00:58:40 Building the Attention Layer and the 'attention_wrapper.py' walk-through

01:15:40 Impressive student's experiment with different mathematical technique on Style Transfer

01:18:00 Translate English into French, with Pytorch

01:31:20 Translate English into French: using Keras to prepare the data
Note: Pytorch latest version now supports Broadcasting

01:38:50 Writing and running the 'Train & Test' code with Pytorch

01:44:00 NLP Programming Tutorial, by Graham Neubig (NAIST)

01:48:25 Question: "Could we translate Chinese to English with that technique ?"
& new technique: Neural Machine Translation of Rare Words with Subword Units (Research)

01:54:45 Leaving Translation aside and moving to Image Segmentation,
with the "The 100 layers Tiramisu: Fully Convolutional DenseNets" (research)
and "Densely Connected Convolutional Networks" (research)