Direct Preference Optimization (DPO) - How to fine-tune LLMs directly without reinforcement learning

Direct Preference Optimization (DPO) is a method used for training Large Language Models (LLMs). DPO is a direct way to train the LLM without the need for reinforcement learning, which makes it more effective and more efficient.
Learn about it in this simple video!

This is the third one in a series of 4 videos dedicated to the reinforcement learning methods used for training LLMs.

Video 3 (This one!): Deterministic Policy Optimization

00:00 Introduction
01:08 RLHF vs DPO
07:19 The Bradley-Terry Model
11:25 KL Divergence
16:32 The Loss Function
14:36 Conclusion

Get the Grokking Machine Learning book!
Discount code (40%): serranoyt
(Use the discount code on checkout)

[miklefeldman]

Thank you very much for the video!
Do I understand correctly that RLHF still has some advantages, namely that by using it we can gather a small amount of human preferences data, and then, after training a reward model using that data, it will itself evaluate many more new examples?
So by having trained the reward model, we have basically free human annotator, that can rate endless new examples.
In the case of DPO, however, we only have the initial human preferences data and that’s it.

[Cathiina]

Hi Mr. Serrano! I am doing your coursera course at the moment on linear algebra for machine learning and I am having so much fun! You are a brilliant teacher, and I just wanted to say thank you! Wish more teachers would bring theoretical mathematics down to a more practical level. Obviously loving the very expensive fruit examples :)

[guzh]

DPO main equation should be PPO main equation.

[IceMetalPunk]

I'm a little confused about one thing: the reward function, even in the Bradley-Terry model, is based on the human-given scores for individual context-prediction pairs, right? And πθ is the probability from the current iteration of the network, and πRef is the probability from the original, untuned network?

So then after that "mathematical manipulation", how does the human-given set of scores become represented by the network's predictions all of a sudden?

[frankl]

Really love the way you broke down the DPO loss, this direct way is more welcome by my brain :). Just one question on the video, I am wondering how important it is to choose the initial transformer carefully. I suspect that if it is very bad at the task, then we will have to change the initial response a lot, but because the loss function prevents from changing too much in one iteration, we will need to perform a lot tiny changes toward the good answer, making the training extremely long. Am I right ?

[subhamkundu]

Thanks for sharing. Is there any hands on resource to try DPO ?

[AravindUkrd]

Thanks for the simplified explanation. Awesome as always.
The book link in the description is not working.

[mekuzeeyo]

Great video as always. I have a question, in practice which one works best using DPO or RLHF?

[frankl]

Did anyone expect something different than Sofmax regarding the Bradley-Terry model as myself? 😅

[VerdonTrigance]

It's kinda hard to remember all of these formulas and it's demotivating me from further learning.