== Direct Preference Optimization
http://arxiv.org/abs/2305.18290

=== Abstract

Existing methods for LLM steerability involve the following steps:
1. Make humans score model generations and obtain relative scores.
2. Fine tune the unsupervised LM to align with these preferences.

The second step is often performed using RLHF. This is done as follows:
1. Fit a reward model which reflects human preferences.
2. Fine tune the unsupervised LM using RL to maximize rewards.

This paper introduces a new parameterization of the reward model, that enables
extraction of the optimal policy without having to train a reward model. This
enables alignment of the model with a simple classification loss. The resulting
algorithm, called DPO, is stable, performant and lightweight, eliminating the
need for sampling from the LM during fine tuning.

The paper also claims that DPO compares or exceeds PPO based RLHF in several
tasks.

=== Introduction

LLMs are trained on very large datasets with a wide variety of skillsets.
However, some of these skills may not be desirable to imitate. Our goal is to
make the model aware of common mistakes, but we want to steer the model to only
generate the best responses.

Existing methods steer language models using a curated set of _human preferences_
which are representative of the type of behaviours that humans find safe and
helpful. This stage follows a large-scale unsupervised pre-training on a large
dataset. The most straightforward appraoch for preference learning is supervised
fine tuning on human demonstrations.

However, the approach which is most successful is RLHF. As described before,
this approach fits a reward model to a dataset of human preferences and then
uses RL to optimize a language model policy to produce responses which
correspond to high rewards. This pipeline is considerably more complex than
supervised learning, and involves sampling from a language model during the
training loop. This leads to significant computational costs.

This paper describes how to directly optimize a language model to adhere to
human preferences without explicit reward modeling or RL. The authors show that
their algorithm, DPO, is at least as effective as existing methods, including
RLHF, in tasks such as sentiment modulation, summarization and dialoge, using
language models with upto 6B parameters.

=== Related Work

While LLMs are better trained using a dataset of expert demonstrations
(instruction tuning), relative human judgements of model responses are often
easier to collect. Therefore, subsequent works have fine-tuned LLMs with
datasets of human preferences. These methods optimize a reward function for
compatibility with the preference dataset under a preference model, such as the
Bradley-Terrey model. The next step is to fine-tune a language model using RL
algorithms such as REINFORCE, PPO, etc. A related line of work uses fine-tuned
LLMs to generate additional preference data for targeted attributes such as
safety or harmlessness.

=== Preliminaries

The RLHF pipeline @ziegler2020finetuning includes the following steps:
1. Supervised Fine Tuning
2. Preference Sampling
3. Reward Learning
4. RL Optimization

After being pretrained in an unsupervised manner on a large dataset, the model
undergoes supervised fine tuning on a rather small but high quality dataset to
obtain $pi^"SFT"$.

This supervised model is then sampled to collect multiple answers $(y_1, y_2) tilde pi^"SFT" (y | x)$ for
a given prompt $x$. These answers are then presented to human labelers who
express their preferences over these answers. These preferences are assumed to
be generated by some latent reward model $r^*(y, x)$ which we do not have access
to.

There are a number of methods used to model preferences, the Bradley-Terry model
being a popular choice. This model states that the human preference distribution
can be written as:

$ p^*(y_1 gt.curly y_2 | x) = exp(r^*(x, y_1)) / (exp(r^*(x, y_1)) + exp(r^*(x, y_2))) $

