Direct Preference Optimization (DPO): Your Language Model is Secretly a Reward Model Explained

preview_player
Показать описание
  1. Gabriel Mongaras
  2. Machine Learning
  3. Artificial Intelligence
  4. Attention
  5. Deep Learning
  6. Transformers
Рекомендации по теме
Direct Preference Optimization: 0:08:55

Direct Preference Optimization: Your Language Model is Secretly a Reward Model | DPO paper explained

Direct Preference Optimization 0:48:46

Direct Preference Optimization (DPO) explained: Bradley-Terry model, log probabilities, math

Direct Preference Optimization 0:21:15

Direct Preference Optimization (DPO) - How to fine-tune LLMs directly without reinforcement learning

Direct Preference Optimization 0:36:25

Direct Preference Optimization (DPO): Your Language Model is Secretly a Reward Model Explained

Direct Preference Optimization: 0:09:10

Direct Preference Optimization: Forget RLHF (PPO)

Reinforcement Learning from 0:19:38

Reinforcement Learning from Human Feedback (RLHF) & Direct Preference Optimization (DPO) Explain...

Aligning LLMs with 0:58:07

Aligning LLMs with Direct Preference Optimization

Direct Preference Optimization 1:01:56

Direct Preference Optimization (DPO)

LLM Alignment: Techniques 0:59:58

LLM Alignment: Techniques for Building Human-Aligned AI

Direct Preference Optimization 0:05:12

Direct Preference Optimization (DPO) in AI

Direct Preference Optimization 0:01:00

Direct Preference Optimization in One Minute

DPO Debate: Is 0:26:55

DPO Debate: Is RL needed for RLHF?

Direct Preference Optimization 0:14:14

Direct Preference Optimization

Direct Preference Optimization 0:42:49

Direct Preference Optimization (DPO)

Towards Reliable Use 1:03:55

Towards Reliable Use of Large Language Models: Better Detection, Consistency, and Instruction-Tuning

What is Direct 0:16:43

What is Direct Preference Optimization?

Direct Preference Optimization 0:11:35

Direct Preference Optimization (DPO): How It Works and How It Topped an LLM Eval Leaderboard

DPO : Direct 0:47:55

DPO : Direct Preference Optimization

Direct Preference Optimization 0:08:00

Direct Preference Optimization (DPO) of LLMs to Reduce Toxicity

DPO - Part1 0:53:03

DPO - Part1 - Direct Preference Optimization Paper Explanation | DPO an alternative to RLHF??

DPO - Part2 0:41:21

DPO - Part2 - Direct Preference Optimization Implementation using TRL | DPO an alternative to RLHF??

Direct Preference Optimization 0:15:11

Direct Preference Optimization Your Language Model is Secretly a Reward Model

LLM training process 0:07:51

LLM training process with Direct Preference Optimization (DPO) and bypass Reward Model (Part3)

Direct Preference Optimization: 1:22:20

Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Ko/En Subtitles)

Комментарии
Автор

Dude, you're a hero for making these videos! Definitely earned a subscription from me.

rw-kbqv
Автор

I'm really glad a channel likes yours exist.

FaultyTwo
Автор

Thank you so much! I truly enjoyed your video and the way you explain things. There are moments, however, where I find myself a bit lost when you don't delve deeper, and I wish you could expand on those points. I would appreciate it if you could recommend a video, article to help me become more familiar with the basic concepts related to such papers in this field. Understanding these basics would make it much easier for me to grasp the material. I feel some gaps in my knowledge that If I work on them then understanding these papers, their mathematical notation will be easier. I focusing on papers in this area of transformers and training models.. Any suggestions would be greatly appreciated. Btw you definitely earned my subscription as well.

unclecode
Автор

Nice one, thanks for sharing. The replacement of the reward model by the MLE term looks appealing when we have groundtruth (generated reply and reference reply). Still, the advantage of reward models is mainly in their potential to be used on new samples without groundtruth presented (i.e., no reference replies in a self-play training on new datasets), so how would the MLE loss work in such scenarios?

prof_shixo
Автор

I was reading Zephyr. It led to this DPO paper, which landed me on your channel. I am soooo happy. Keep it up!

bibiworm
Автор

Hey @Gabriel. Can you please clear my doubt, why @12:58 we cant just directly backpropagate the loss like we do in simple fine tuning? I am not understanding it? Please share any relevant resource?

YashVerma-iilx