What is DPO (Direct Preference Optimization)?
TL;DR
A method for directly optimizing AI models from human preference data. Simpler and more efficient than RLHF.
DPO (Direct Preference Optimization): Definition & Explanation
DPO (Direct Preference Optimization) is a technique for aligning AI models with human preferences. Traditional RLHF (Reinforcement Learning from Human Feedback) requires a two-stage process of training a reward model followed by reinforcement learning, but DPO can directly optimize the model from human preference data (e.g., 'response B is better than response A') without an intermediate reward model. This improves training stability and reduces computational costs. Proposed by a Stanford University research team in 2023, DPO has been adopted for tuning many open-source models including LLaMA 2 and Zephyr.