.png)
Post fine-tuning LLMs with Direct Preference Optimization
Thanh Long Phan
Since the publication of our previous blog on Reinforcement Learning from Human Feedback (RLHF), an alternative algorithm has been introduced, that does not require the use of a reward model to fine-tune Large Language Models (LLMs) based on human preferences. This method is called Direct Preference Optimization (DPO) and was introduced in the paper Direct Preference Optimization: Your Language Model is Secretly a Reward Model , which was one of the best papers at NeurlPS 2023. Well-known open-source models such as Mixtral 8x7B have been optimized using DPO. At the time of writing this blog, Meta also released their new Llama 3 models, which also utilize DPO for fine-tuning. And one week later, Microsoft has unveiled Phi-3, also leveraging DPO for its optimization processes.
.jpg)
