Hello from the Qualiteg Research Team!

Today we explain Direct Preference Optimization (DPO)—proposed in 2023 by the research team of Rafael Rafailov, Archit Sharma, and colleagues—from the fundamentals through to its applications.

The method was introduced in the paper "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" and has had a major impact on how AI models are trained. The paper proposes DPO as a new technique for aligning the behavior of language models (LMs) with human preferences, but its applications have recently expanded beyond language models to VLMs and other model types.

What's more, the theory is relatively simple, which is why its popularity has been steadily growing.

The Background Behind DPO

Because language models are pre-trained on massive datasets, they acquire broad knowledge and capabilities—but controlling their behavior has been difficult.

For that reason, conventional language model training used reinforcement learning (RL) to reflect human preferences, but this came with challenges: complex implementation and high computational cost.

Reinforcement Learning from Human Feedback (RLHF) is useful, but it is complex, computationally expensive, and burdensome.

To solve these challenges, the team developed DPO as a more direct and simpler approach. The method attracted attention immediately after publication and has been adopted by many leading AI research organizations, including OpenAI, Google, and DeepMind.

The Core Principles of DPO

Conventional RLHF trains a reward model and then tunes the policy to maximize it.
DPO, by contrast, is a new method that tunes the language model directly on human preference data, without explicitly training a reward model.

Mathematically, it is expressed as follows:

L(θ) = E[log(1 + exp(β(r_w(x_w) - r_w(x_l))))]

where:

• θ: the model parameters
• β: the temperature parameter
• r_w(x): the preference score
• x_w, x_l: the "preferred" output and the "rejected" output, respectively

By minimizing this loss function—a simple classification loss—the model learns to produce outputs aligned with human preferences, which is why the method is both stable and computationally efficient.

As a result, DPO has been shown to match or exceed conventional RLHF methods (such as PPO) on tasks like sentiment control, summarization, and dialogue. It is also easy to implement, with no need for elaborate hyperparameter tuning.

What about applications beyond language?

Applications in Image Generation AI

Let's consider the image generation context.

Models that generate images from text prompts (such as DALL·E and Stable Diffusion) can struggle to reflect a user's specific tastes. In fact, the more precisely you try to specify what you want, the harder it gets.

With DPO, human feedback comparing "preferred images" against "non-preferred images" is used to instill those preferences directly into the model.

For example, the style and quality of generated images—say, "realistic landscapes" or "anime-style characters"—can be tuned to match specific preferences.

Example: Style Optimization

preferred_style = get_human_preference(image_A, image_B)
loss = dpo_loss(model_params, preferred_style)
model_params = optimize(loss)

As a concrete example, in anime-style character generation, features such as eye size and line weight can be adjusted based on human preferences.

Example: Quality Improvement
DPO is also effective for improving image quality:

quality_score = Σ(wi * quality_metric_i)

Here, wi represents the weight of each quality metric, learned from human preference data.

A DPO Implementation Example for Image Generation

Now let's look at implementing DPO for image generation.

The Overall Picture of a DPO Implementation

The most important question in implementing Direct Preference Optimization (DPO) is how to get the model to learn human preferences. The PyTorch implementation presented here uses an image generation task to illustrate the basic structure of preference learning.

How the DPOTrainer Class Works

At the heart of DPO is the DPOTrainer class.

This class manages the entire training process. At initialization, it sets the temperature parameter β and the learning rate.

β controls the strength of the preference signal: the larger the value, the more the gap between preferred and rejected outputs is emphasized. The optimizer is Adam.

The loss function implementation is especially important and characteristic. The compute_dpo_loss method takes pairs of preferred and rejected outputs and assigns a score to each. It then computes a loss expressed as log(1 + exp(β(rejected - preferred))). With this loss function, training pushes preferred outputs toward higher scores and rejected outputs toward lower scores.

import torch
import torch.nn as nn
import torch.optim as optim

class DPOTrainer:
    def __init__(self, model, beta=0.1, learning_rate=1e-5):
        self.model = model
        self.beta = beta
        self.optimizer = optim.Adam(model.parameters(), lr=learning_rate)
    
    def compute_dpo_loss(self, preferred_outputs, rejected_outputs):
        # Compute preference scores
        preferred_scores = self.model(preferred_outputs)
        rejected_scores = self.model(rejected_outputs)
        
        # Implementation of the DPO loss function
        loss = torch.log(1 + torch.exp(self.beta * (rejected_scores - preferred_scores)))
        return loss.mean()
    
    def train_step(self, preferred_batch, rejected_batch):
        self.optimizer.zero_grad()
        loss = self.compute_dpo_loss(preferred_batch, rejected_batch)
        loss.backward()
        self.optimizer.step()
        return loss.item()

A Neural Network for Evaluating Images

For image evaluation we use a convolutional neural network implemented as the ImageDPO class. It is a perfectly ordinary CNN. To walk through it briefly: the network takes an RGB image as input and extracts features through a series of convolutional layers. The first convolutional layer expands the 3-channel input to 64 channels; the extracted features are then flattened into one dimension and converted by a fully connected layer into a single preference score. That final preference score is the distinctly DPO-flavored part.

With this structure, the network effectively captures the visual features of an image and quantifies how desirable the image is. This part is standard practice: ReLU activations add non-linearity, enabling the network to learn complex features.

# Example application to image generation
class ImageDPO(nn.Module):
    def __init__(self):
        super().__init__()
        self.backbone = nn.Sequential(
            nn.Conv2d(3, 64, 3, padding=1),
            nn.ReLU(),
            nn.Conv2d(64, 64, 3, padding=1),
            nn.ReLU(),
            nn.Flatten(),
            nn.Linear(64 * 32 * 32, 1)  # adjust according to image size
        )
    
    def forward(self, x):
        return self.backbone(x)

# Usage example
def train_image_dpo():
    model = ImageDPO()
    trainer = DPOTrainer(model)
    
    # Training loop
    for epoch in range(num_epochs):
        for preferred_batch, rejected_batch in data_loader:
            loss = trainer.train_step(preferred_batch, rejected_batch)
            print(f"Epoch {epoch}, Loss: {loss:.4f}")

Implementing the Training Process

The actual training is controlled by the train_image_dpo function. It creates instances of the ImageDPO model and the DPOTrainer, and for each epoch it iterates through pairs of preferred and rejected images from the dataset. For each batch, the pair of preferred and rejected images is fed into the model to compute their respective scores; the loss is then computed from these scores, and the model parameters are updated to minimize it.

There are DPO-specific touches here and there, but at its core this is the familiar CNN-style image network.

In the End, What Matters Is the Preference Data

As the implementation walkthrough shows, DPO itself is very simple. To reap the benefits of this method, it ultimately comes down to the input data. DPO does nothing on its own—it is only as good as its inputs. In other words, you need to properly prepare a dataset of image pairs, which involves collecting preference judgments from human evaluators. In addition, by monitoring the training process and evaluating the model appropriately, you can verify its performance and make adjustments as needed. Combining these elements properly is what makes it possible to build an effective DPO training system.

You may be thinking, "So it all comes down to data preparation after all"—but anyone who has been through RLHF will agree that this is a far happier place to be.

Applications to Video Generation

Next, let's look at applications to video generation.

Our own video generation AI actually incorporates DPO as well.

What matters most in video generation is accounting for the time axis.

frame_consistency_score = Σt(similarity(frame_t, frame_t+1))
motion_naturalness = evaluate_motion_smoothness(frames)
total_score = α * frame_consistency_score + β * motion_naturalness

To improve the naturalness of human motion, human evaluators select the "more natural movement" among the multiple still frames that make up a video. The data a human judges as "natural" becomes the preference data, which is then used for DPO training. This achieves a level of naturalness one step above conventional interpolation techniques.

A DPO Implementation Example for Video Generation

Implementing DPO for video generation is somewhat more complex than for image generation.

The key point is maintaining consistency along the time axis.

A video generation AI does not produce a video in one shot; the video is formed as a collection of still frames (called a sequence).

The individual still frames are stitched together like a flip-book to appear as video. For each independent frame to look natural, the temporal ordering between frames becomes critical—in other words, "maintaining consistency along the time axis" is essential.

(As an aside, the most important element of the AI Human we are working toward is also "maintaining consistency." Whether in imagery or in personality, consistency is what makes a human seem human—a genuinely fascinating theme.)

Here we take a detailed look at the implementation for video generation, centered on the VideoDPO class.

Feature Extraction with 3D Convolutions

At the core of the VideoDPO class is a frame_encoder built with 3D convolutional layers (Conv3D). These layers extract not only spatial features but temporal features as well. The input data is handled as a 5-dimensional tensor: (batch size, channels, frames, height, width).

The network itself is standard fare, but to walk through it: the first convolutional layer expands the 3-channel input to 64 channels, a MaxPooling layer then reduces the size of the feature maps, and the next convolutional layer expands from 64 to 128 channels for a richer feature representation. Each convolutional layer is followed by the familiar ReLU.

Scoring Temporal Features

The extracted features are evaluated by the temporal_scorer. This module first flattens the multi-dimensional feature maps, passes them through an intermediate layer with 512 units, andfinally converts them into a single score. This score serves as a holistic measure of video quality.

class VideoDPO(nn.Module):
    def __init__(self, num_frames=16):
        super().__init__()
        self.frame_encoder = nn.Sequential(
            nn.Conv3d(3, 64, kernel_size=(3, 3, 3), padding=(1, 1, 1)),
            nn.ReLU(),
            nn.MaxPool3d(kernel_size=(2, 2, 2)),
            nn.Conv3d(64, 128, kernel_size=(3, 3, 3), padding=(1, 1, 1)),
            nn.ReLU(),
            nn.MaxPool3d(kernel_size=(2, 2, 2))
        )
        self.temporal_scorer = nn.Sequential(
            nn.Linear(128 * 4 * 4 * 4, 512),
            nn.ReLU(),
            nn.Linear(512, 1)
        )
    
    def forward(self, x):
        # x shape: (batch_size, channels, frames, height, width)
        features = self.frame_encoder(x)
        features = features.view(features.size(0), -1)
        score = self.temporal_scorer(features)
        return score

Evaluating Inter-Frame Consistency

Now let's look at the inter-frame consistency mentioned earlier.

The compute_temporal_consistency function computes the differences between consecutive frames using mean squared error (MSE).

(If you need a refresher on MSE, please see our Qiita blog, in Japanese.)

The smaller this value, the smoother and more natural the video's motion. The final return value is negative, by design, so that higher consistency yields a smaller loss.

def compute_temporal_consistency(frames):
    """Compute the inter-frame consistency score"""
    consistency = 0
    for i in range(len(frames)-1):
        consistency += torch.nn.functional.mse_loss(frames[i], frames[i+1])
    return -consistency  # smaller differences mean higher consistency

# Example training loop for video generation
def train_video_dpo(model, train_loader, num_epochs=10):
    trainer = DPOTrainer(model)
    
    for epoch in range(num_epochs):
        for preferred_videos, rejected_videos in train_loader:
            # Basic DPO loss
            dpo_loss = trainer.train_step(preferred_videos, rejected_videos)
            
            # Additional loss accounting for temporal consistency
            temporal_loss = compute_temporal_consistency(preferred_videos)
            
            # Combined loss
            total_loss = dpo_loss + 0.5 * temporal_loss
            
            print(f"Epoch {epoch}, DPO Loss: {dpo_loss:.4f}, "
                  f"Temporal Loss: {temporal_loss:.4f}")

Implementing the Training Process

The train_video_dpo function controls the actual training process. In each epoch, the DPO loss is computed using pairs of preferred and rejected videos. In addition, temporal consistency is evaluated for the preferred videos, and the combined loss function is minimized. By weighting the temporal consistency loss with a coefficient of 0.5 and combining it with the DPO loss, the implementation balances preference learning against preserving video quality. Both loss values are printed every epoch, allowing detailed monitoring of training progress.

Practical Considerations

We have shown a simple implementation, but once again the input data is what matters. You need pairs of "good videos" and "bad videos" as input, and collecting them is not easy—it also demands clear evaluation criteria. Here too, building an appropriate dataset determines the model's performance.

DPO's Impact and Outlook

Since DPO's debut, many AI research organizations have adopted the method, and a variety of improved versions and applied studies have been published. Since 2024 in particular, applied research has accelerated in image and video generation—a wave that we at Qualiteg have joined as well.

In video processing, thanks in part to its good fit with mobile cameras, further research is under way into more efficient training methods and real-time applications.

Summary

Introduced in 2023, DPO—thanks to its mathematical simplicity and ease of implementation—is being rapidly adopted across many fields, from language models to image and video generation. Its ability to learn human preferences efficiently is a major advantage, especially for generating visual media where subjective quality judgments matter.

We plan to leverage DPO in our own services to release offerings of even higher quality and efficiency—stay tuned!

At Qualiteg, we bring the full range of the latest AI science and AI engineering to bear on services and solutions that deepen human creativity. We are a small, elite team taking on the challenge of transforming the world with cutting-edge AI science and engineering. If you are confident in your skills, we would be delighted if you considered a career with us!