filmov
tv
Direct Preference Optimization (DPO) explained: Bradley-Terry model, log probabilities, math
Показать описание
In this video I will explain Direct Preference Optimization (DPO), an alignment technique for language models introduced in the paper "Direct Preference Optimization: Your Language Model is Secretly a Reward Model".
I start by introducing language models and how they are used for text generation. After briefly introducing the topic of AI alignment, I start by reviewing Reinforcement Learning (RL), a topic that is necessary to understand the reward model and its loss function.
I derive step by step the loss function of the reward model under the Bradley-Terry model of preferences, a derivation that is missing in the DPO paper.
Using the Bradley-Terry model, I build the loss of the DPO algorithm, not only explaining its math derivation, but also giving intuition on how it works.
In the last part, I describe how to use the loss practically, that is, how to calculate the log probabilities using a Transformer model, by showing how it is implemented in the Hugging Face library.
If you're interested in how to derive the optimal solution to the RL constrained optimization problem, I highly recommend the following paper (Appendinx A, equation 36):
Chapters
00:00:00 - Introduction
00:02:10 - Intro to Language Models
00:04:08 - AI Alignment
00:05:11 - Intro to RL
00:08:19 - RL for Language Models
00:10:44 - Reward model
00:13:07 - The Bradley-Terry model
00:21:34 - Optimization Objective
00:29:52 - DPO: deriving its loss
00:41:05 - Computing the log probabilities
00:47:27 - Conclusion
I start by introducing language models and how they are used for text generation. After briefly introducing the topic of AI alignment, I start by reviewing Reinforcement Learning (RL), a topic that is necessary to understand the reward model and its loss function.
I derive step by step the loss function of the reward model under the Bradley-Terry model of preferences, a derivation that is missing in the DPO paper.
Using the Bradley-Terry model, I build the loss of the DPO algorithm, not only explaining its math derivation, but also giving intuition on how it works.
In the last part, I describe how to use the loss practically, that is, how to calculate the log probabilities using a Transformer model, by showing how it is implemented in the Hugging Face library.
If you're interested in how to derive the optimal solution to the RL constrained optimization problem, I highly recommend the following paper (Appendinx A, equation 36):
Chapters
00:00:00 - Introduction
00:02:10 - Intro to Language Models
00:04:08 - AI Alignment
00:05:11 - Intro to RL
00:08:19 - RL for Language Models
00:10:44 - Reward model
00:13:07 - The Bradley-Terry model
00:21:34 - Optimization Objective
00:29:52 - DPO: deriving its loss
00:41:05 - Computing the log probabilities
00:47:27 - Conclusion
0:08:55
Direct Preference Optimization: Your Language Model is Secretly a Reward Model | DPO paper explained
0:48:46
Direct Preference Optimization (DPO) explained: Bradley-Terry model, log probabilities, math
0:19:38
Reinforcement Learning from Human Feedback (RLHF) & Direct Preference Optimization (DPO) Explain...
0:09:10
Direct Preference Optimization: Forget RLHF (PPO)
0:21:15
Direct Preference Optimization (DPO) - How to fine-tune LLMs directly without reinforcement learning
0:58:07
Aligning LLMs with Direct Preference Optimization
0:36:25
Direct Preference Optimization (DPO): Your Language Model is Secretly a Reward Model Explained
1:01:56
Direct Preference Optimization (DPO)
0:26:55
DPO Debate: Is RL needed for RLHF?
0:05:12
Direct Preference Optimization (DPO) in AI
0:42:49
Direct Preference Optimization (DPO)
0:08:00
Direct Preference Optimization (DPO): A low cost alternative to train LLM models
1:03:55
Towards Reliable Use of Large Language Models: Better Detection, Consistency, and Instruction-Tuning
0:14:14
Direct Preference Optimization
0:53:03
DPO - Part1 - Direct Preference Optimization Paper Explanation | DPO an alternative to RLHF??
0:47:55
DPO : Direct Preference Optimization
0:24:28
Direct Preference Optimization
0:33:26
ORPO: Monolithic Preference Optimization without Reference Model (Paper Explained)
0:45:21
How DPO Works and Why It's Better Than RLHF
0:05:27
Direct Preference Optimization or DPO is out and TR-DPO is in ? | New LLM Paper
0:37:53
Direct Preference Optimization (DPO) - math insight explained
0:37:12
PR-453: Direct Preference Optimization
0:00:23
Day 7 / 75 of 75HardResearch | Direct Preference Optimization (DPO)
0:05:08
LLM Alignment Methods - DPO vs IPO vs KTO vs PCL
Комментарии