Machine Learning Foundations - Deep Learning in Life Sciences Lecture 02 (Spring 2021)

6.874/6.802/20.390/20.490/HST.506 Spring 2021 Prof. Manolis Kellis
Deep Learning in the Life Sciences / Computational Systems Biology

0:00 Lecture overview
1:15 What is machine learning?
4:50 Machine learning notation and terminology
14:24 Types of machine learning
18:45 Objective functions
27:08 Optimizing the objective function
29:40 Training, validation, and test sets
35:24 Performance measures for classification: confusion matrix, ROC
39:50 Performance measures for regression: Pearson, Spearman
42:22 Significance tests
46:12 Multiple hypothesis
48:23 Correlation is not causation
52:30 Traditional neural networks
57:20 Non-linearity
1:02:07 Training a neural network: back-propagation, gradient-based learning
1:13:30 Controlling model complexity
1:14:57 Model capacity
1:15:35 Generalizability
1:19:30 Improving generalization
1:21:20 Conclusion, Questions, Goodbyes

[nunocalaim]

Imagine if Manoli's kid watches these lectures in the future and suddenly remembers the day when he/she burst into dad's office. That's time travelling

[emdadsami]

🎯 Key Takeaways for quick navigation:

00:02 🧠 *Introduction to Machine Learning Foundations*
- Defining machine learning as the process of converting experience into expertise.
- Exploring the broad definition of machine learning as computational methods using experience to improve performance.
- Mitchell's definition: Learning from experience with respect to some class of tasks, improving performance as experience increases.
04:52 🤖 *Training Set and Task Explanation*
- Describing the training set as a set of known inputs (x) with corresponding known outputs (y).
- Highlighting the difference between training set and test set.
- Emphasizing the importance of generalization power for evaluating model performance.
08:08 🎓 *Inputs, Outputs, and Model Function*
- Introducing terminology: input space (x), label space (y), and the function that translates x into y.
- Clarifying the role of features, labels, and predictions in machine learning.
- Discussing the use of parameters, weights, and biases in the transformation function.
11:37 🌐 *Types of Models in Machine Learning*
- Differentiating between discriminative and generative models.
- Explaining the concept of parameterized models and how they represent hypotheses.
- Discussing supervised, semi-supervised, and reinforcement learning.
15:37 🎯 *Diverse Machine Learning Tasks*
- Highlighting that machine learning is not limited to classification, discussing regression.
- Introducing the concept of unsupervised learning and its focus on finding patterns.
- Mentioning reinforcement learning and partial feedback scenarios.
18:54 🎯 *Objective Functions in Machine Learning*
- Defining the objective function as the function optimized during training.
- Explaining error, cost, and loss functions, with examples like zero-one loss and cross-entropy.
- Discussing loss functions for classification, emphasizing the importance of optimization.
23:32 📊 *Classification Loss Functions*
- Focusing on classification loss functions: zero-one loss and binary cross-entropy.
- Connecting logistic regression to binary cross-entropy and understanding its probabilistic nature.
- Highlighting the role of cross-entropy in capturing information about predictor values.
24:32 🧠 *Introduction to Cross Entropy and Softmax*
- Categorical cross-entropy loss and its information-based approach.
- Introduction to softmax, a function mapping space into a zero-one output.
- Probabilistic interpretation and likelihood based on category distribution.
27:06 🎓 *Model Training Overview*
- Structure of the problem: input data, weights, bias, and output.
- Initiating model parameters and building the optimizer.
- Stochastic Gradient Descent (SGD) explained with the use of mini-batches.
29:44 🏋️‍♂️ *Training and Evaluation*
- Importance of training and testing sets in model evaluation.
- Overfitting: when the model performs well on training but not on validation.
- Balancing training and validation to gauge true generalization power.
35:21 📊 *Metrics for Model Evaluation*
- Explanation of true positives, false positives, true negatives, and false negatives.
- Overview of metrics: accuracy, precision, specificity, recall, F1 score.
- Introduction to receiver operating characteristic (ROC) curves.
42:09 📈 *Correlation and Significance Tests*
- Understanding correlation in the context of machine learning.
- Explanation of significance tests using the t-distribution and p-values.
- Correction methods for multiple hypothesis testing.
50:44 🧘 *Brain Break and Polls*
- Instructor engages participants in a physical activity to boost attention.
- Polls conducted to assess participants' learning progress and pace satisfaction.
52:38 🧠 *Introduction to Neural Networks*
- Overview of neural networks, emphasizing multiple layers learning abstractions.
- Hierarchy in deep learning architectures: input layer, hidden layers, and output layer.
- Historical context, roots in the 60s, and evolution, including convolutional filters.
54:40 🧊 *Basics of Deep Learning Neural Networks*
- Inspiration from the human brain's neural structure.
- Explanation of neurons, dendrites, axons, and the computational abstraction of weights.
- Neural network basics: learning function, transforming pixels into output probabilities.
56:14 🤖 *Activation Functions*
- Introduction to activation functions: sigmoid, soft plus, soft max, and ReLU.
- Explanation of the sigmoid unit's firing concept.
- Emphasis on ReLU as a popular activation function in modern deep learning.
57:26 🚀 *Non-Linearity and Neural Network Power*
- Significance of non-linearity in expanding the space of learnable functions.
- Explanation of how non-linearities enable learning complex functions like XOR.
- Overview of the original sigmoid unit, soft plus, and the preferred choice of ReLU.
01:03:20 🎓 *Gradient-Based Learning*
- Foundation of gradient-based learning: adjusting weights using partial derivatives.
- Explanation of adjusting weights to minimize errors in neural networks.
- Introduction to stochastic gradient descent and its role in optimization.
01:06:29 🔄 *Back Propagation and Chain Rule*
- Detailed explanation of backpropagation and its role in adjusting weights.
- Application of the chain rule in computing gradients at each layer.
- Discussion on the importance of backpropagation in training neural networks.
01:10:40 🎚️ *Optimization Techniques*
- Additional techniques in gradient-based learning: learning rate, weight decay, and momentum.
- Explanation of learning rate's role in adjusting the step size during optimization.
- Introduction to weight decay and momentum as methods to prevent overfitting.
01:14:03 🛠️ *Model Capacity and Generalization*
- Definition and importance of model capacity and VC dimension.
- Explanation of non-parametric model capacity related to the training set size.
- Discussion on generalizability and the model's ability to perform on unseen inputs.
01:16:23 📈 *Model Capacity and Generalization*
- Model capacity is related to the effective number of parameters, VC dimension, or dimensions of the model.
- Increasing model capacity can improve fit to the training set but may lead to poor generalization.
- The trade-off involves adjusting model complexity and regularization techniques to avoid overfitting.
01:19:44 🔍 *Balancing Model Capacity and Regularization*
- Model capacity can be adjusted by reducing or increasing the number of neurons.
- Regularization methods include dropout, weight decay, and clamping to control overfitting.
- The balance between parameter regularization and model complexity regularization is crucial.
01:22:07 🤔 *Questions on Model Capacity and Design*
- Questions on how to calculate or estimate the capacity of a deep neural network.
- Exploring the decision-making process for adding more units versus adding new layers in a network.
- Emphasizes that these considerations are part of the art rather than having a strict theoretical framework.

Made with HARPA AI

[theworldsonfire.]

I’m only half way through the Meaning of Life Symposium. It’s wonderful! I’m looking forward to the rest of these lectures as well.

[khalidsaifullaah]

looking forward to the next one, hope it gets uploaded soon😍😊

[tricialobo]

Could we please have a copy of the problem sets?
Thank you for making this series available!

[aigerim]

I love the idea of a little stretch break haha

[MrLikon]

I guess instead of _expected risk (loss)_ it should read _expected loss (risk)_ because thats the defenition of risk.

[siddharthkotwal]

Thanks Manoli for putting these lectures online! I lost some acquaintances to COVID and as an ML Engineer, I’m determined to see how I can contribute to Healthcare and Computational Biology to bring better solutions to general public. Where can I find the Notebooks and Problem Assignments?

[mil]

Thank you for this, sir. May I ask if non MIT people can also have a glimpse of the practice set and the instructions in it?

[davidedavidedav]

Really good videos, but the background noises are annoying