Hello!

Today I'd like to explain EMA*, which plays an important role in image generation, video generation, and other models.

It also serves us well in MotionVox™, our AI avatar video generation service.

That said, EMA is not a technique exclusive to image generation. It has become a highly valued tool across everyday deep learning practice—both training and inference (including generative use)—so let's walk through it from the fundamentals all the way to implementation.

* As for how to pronounce EMA: I say "Emma," while others spell it out as "E-M-A." Either is fine.

EMA Fundamentals

Put simply, EMA (Exponential Moving Average) is a technique that averages a model's weights.

It is actually a long-standing concept, used in fields such as stock price analysis, but in deep learning its importance was recognized relatively recently—a classic case of "hey, this actually works pretty well."
(And not just with EMA—it is by now common knowledge in the deep learning world that many advances come not from stacking up theory but from "we tried it, and it worked.")

Why Does EMA Matter?

As you know, deep learning involves two phases: training and inference (that is, actually using the model).

Each phase has its own concern:

• Training: "the model needs to adapt quickly to new data"
• Inference: "producing stable results is what matters"

In human terms:

• Training: "absorbing new information and constantly changing"
• Inference: "calmly and reliably applying what has been learned"

That's roughly the idea.

EMA is the technique for achieving this stable inference.

In other words, it is a technique for inference that just behaves well.

How EMA Works

The Basic Idea

EMA averages past data using exponentially decaying weights.

$$
\text{EMA}t = \beta \cdot \text{EMA}{t-1} + (1-\beta) \cdot \theta_t
$$

Expanding this formula with a summation gives the following.

$$
\text{EMA}t = (1-\beta) \sum_{i=0}^{t} \beta^i \theta_{t-i} + \beta^t \text{EMA}_0
$$

$$ \begin{array}{l}
・\text{EMA}_t \text{ is the exponential moving average at time } t\\
・\beta \text{ is the smoothing factor } (0 \leq \beta < 1)\\
・\theta_t \text{ is the observed value at time } t\\
・\text{EMA}_0 \text{ is the initial value}
\end{array} $$

Let's look at the first term of the equation above.

$$ \begin{array}{l}
(1-\beta) \sum_{i=0}^{t} \beta^i \theta_{t-i} \text{ represents the weighted sum of past data}\\
\text{Here, the factor } \beta^i \text{ gives older data exponentially smaller weights} \\
(1-\beta) \text{ acts as a normalization term.}
\end{array} $$

Next, the second term:

$$ \begin{array}{l}
\beta^t \text{EMA}_0 \text{ represents the influence of the initial value.}\\
\text{As } t \text{ grows, this term approaches } 0\\
\text{In other words, once enough time has passed, the influence of the initial value becomes negligible}\\
\end{array} $$

If the formulas alone don't quite click, that's perfectly understandable—so this time we've prepared an EMA simulator that lets you experience the effect firsthand. Let's play with it!

Playing with the EMA Simulator

First, move the slider all the way to the left (β = 0). In this state, the green line (EMA) moves almost identically to the blue line (raw data). This is the state where new data is adopted essentially as is.

Next, move the slider gradually to the right. Around β = 0.3, the green line becomes progressively smoother. In this range, it still responds quickly to changes in the data while shaving off a little of the noise.

Now try moving it to around β = 0.7. Here the green line becomes considerably smoother, filtering out short-term noise more effectively. At the same time, you can see it beginning to lag slightly behind sharp changes in the blue line.

Finally, move the slider near the right end (β = 1). The green line becomes extremely smooth and barely moves at all. This is the state where past data dominates and new data has almost no influence.

How was that? Hopefully the meaning of the formula now feels tangible.

Now That You've Felt It: EMA's Properties in Brief

As the simulator makes intuitive, the weighting in EMA has these properties:

• The newest data has the greatest influence
• The influence of older data decays exponentially
• But it never disappears completely (history is preserved)

There is one more benefit: memory efficiency. That is,

• Unlike a simple moving average, there is no need to keep the entire history
• You only need to remember the previous EMA value

That is all it takes.

The Role of the Decay Rate β

As you likely felt in the simulator, EMA's behavior changes dramatically depending on the value of β:

• β = 0.9: new values have a strong influence (sensitive to rapid changes)*
• β = 0.999: old values have a strong influence (more stable)
  

(* In the EMA simulator above, even β = 0.9 already feels like the influence of new values is fairly weak. In deep learning, however, the contest is played out in the number of nines—0.999, 0.9999, and so on—so from that vantage point, β = 0.9 is, relatively speaking, "sensitive to rapid changes.")

For example, when β = 0.99:

influence of the new value = 1%, influence of past values = 99%
which means:

1. Temporary noise is suppressed
2. Sustained changes are gradually reflected
3. Sharp fluctuations are smoothed out

These are its characteristic behaviors.

EMA in Deep Learning

Now that we understand what the EMA technique does, let's look at how it helps in deep learning.

Stabilizing Parameters

First, EMA's main role in deep learning is as a "weight-averaging technique for building an inference model." A common point of confusion: it is not about stabilizing the training itself.

In a normal training process, weights are updated using optimizers such as SGD or Adam. As training proceeds, the weights move toward the objective, but because of the stochastic nature of mini-batches, they oscillate as they converge.

You know the graph—the one where the oscillations gradually shrink.

This is where EMA comes in.

During training, a moving average of the model's weights is computed and maintained in the background. Concretely, an exponentially decaying average is taken over the weights at each iteration. This yields values in which the weight oscillations of the later stages of training have been smoothed out.

The payoff is a more stable inference model that is not swayed by the weight oscillations of the final stages of training. In other words, the training process itself proceeds as usual, while you obtain inference weights with better generalization performance.

Memory Efficiency

From a memory standpoint, EMA is, as you can see, a remarkably simple and efficient technique to implement. Since only the current parameter values and the previous moving-average values need to be kept, the additional memory footprint is minimal. Computationally, it requires nothing more than a simple weighted average, so the cost is extremely small. This is especially welcome when training large neural networks.

Asynchronous by Nature—A Good Fit for Batch Processing

EMA adapts flexibly to a variety of training setups, including batch processing and online learning.

In batch training, it smooths the parameter updates after each batch, appropriately moderating batch-to-batch variation. In sequential settings such as online learning, it can likewise incorporate past information appropriately according to the characteristics of the data stream. In particular, by choosing the β value well, you can tune the balance between new and past information to handle a wide range of training scenarios.

EMA also pairs well with learning-rate scheduling, enabling more stable parameter updates in the later stages of training. This improves the model's convergence and ultimately leads to better final performance.

Implementing EMA

Basic Implementation

Let's look at a basic implementation of EMA in PyTorch.

import copy
import torch
from typing import Optional

class EMAModel:
    """Wrapper class for an exponential moving average (EMA) model
    
    Computes and maintains an exponential moving average of the model's parameters.
    This can improve the model's stability and performance.
    
    Args:
        model (torch.nn.Module): the source model to apply EMA to
        decay (float, optional): EMA decay rate—the "β" in the EMA formula. Defaults to 0.999
        device (Optional[torch.device], optional): device to place the EMA model on. Defaults to None
    """
    def __init__(
        self, 
        model: torch.nn.Module, 
        decay: float = 0.999, # the "β" in the EMA formula
        device: Optional[torch.device] = None
    ):
        self.decay = decay
        
        # Use the specified device if given; otherwise use the same device as the model
        self.device = device if device is not None else next(model.parameters()).device
        
        # Create a deep copy of the model and move it to the target device
        self.ema_model = copy.deepcopy(model).to(self.device)
        
        # Set to evaluation mode (turn off training-specific behavior)
        self.ema_model.eval()
        
        # Mark parameters as not requiring gradient computation
        for param in self.ema_model.parameters():
            param.requires_grad_(False)
            
    def update(self, model: torch.nn.Module) -> None:
        """Update the EMA model using the current model's parameters
        
        Args:
            model (torch.nn.Module): the current model used to update the EMA
        """
        with torch.no_grad():
            for ema_param, model_param in zip(
                self.ema_model.parameters(), 
                model.parameters()
            ):
                # Move the model parameter to the same device as the EMA model
                model_param_on_device = model_param.to(self.device)
                
                # EMA update rule: new value = decay * old value + (1 - decay) * current value
                ema_param.data.mul_(self.decay)
                ema_param.data.add_(
                    model_param_on_device.data * (1.0 - self.decay)
                )
    
    def get_model(self) -> torch.nn.Module:
        """Get the current EMA model
        
        Returns:
            torch.nn.Module: the EMA model in its updated state
        """
        return self.ema_model

Let's walk through the core parts of the code above (the EMAModel class).

class EMAModel:
    def __init__(self, model, decay=0.999):
        # Create the EMA model here
        self.ema_model = copy.deepcopy(model)  # ← make a copy of the regular model
        self.decay = decay
        
        # The EMA model does not train, so set it to evaluation mode
        self.ema_model.eval()
        # Gradient computation is also unnecessary, so turn it off
        for param in self.ema_model.parameters():
            param.requires_grad_(False)
    
    def update(self, model):
        # Update the EMA model's parameters here
        with torch.no_grad():
            for ema_param, model_param in zip(
                self.ema_model.parameters(), 
                model.parameters()
            ):
                # EMA update rule
                ema_param.data.mul_(self.decay)
                ema_param.data.add_(
                    model_param.data * (1.0 - self.decay)
                )

The code's main functionality consists of two parts.

First, the __init__ method creates the EMA model. Here, copy.deepcopy(model) is used to make a complete copy of the regular training model. This copied model becomes the EMA model. Because the EMA model does not perform training, it is set to eval() mode, and gradient computation for its parameters is turned off since it is not needed.

Next, the update method updates the EMA model. It uses the regular training model's current parameters to update the EMA model's parameters. The update computes a weighted average using the decay value* (e.g., 0.999). As a result, the EMA model's parameters become a moving average of the regular model's parameters.

In short, this code implements one continuous process: create a copy of the regular training model, then gradually update that copy's parameters using the regular model's parameters.

(* decay corresponds to β in the original formula.)

The Training Loop

Now let's look at the code on the side that uses EMAModel.

import torch
from torch import nn
import torch.optim as optim
from your_model import YourModel  # your original model
from ema_model import EMAModel   # the EMAModel implemented above

# 1. Prepare the models
model = YourModel()  # regular training model
ema = EMAModel(model, decay=0.999)  # create the EMA model

# 2. Training setup
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# 3. Training loop
num_epochs = 100
for epoch in range(num_epochs):
    model.train()  # set to training mode
    
    for batch_idx, (data, target) in enumerate(train_loader):
        # Regular training step
        optimizer.zero_grad()  # reset gradients
        output = model(data)   # forward pass
        loss = criterion(output, target)  # compute the loss
        loss.backward()    # compute gradients
        optimizer.step()   # update the model

        # Update the EMA model (important: do this after the optimizer updates the model)
        ema.update(model)

    # Evaluation at the end of the epoch
    model.eval()  # set to evaluation mode
    ema_model = ema.get_model()  # get the EMA model
    
    # Evaluate with the regular model
    with torch.no_grad():
        # Compute accuracy on the validation set
        normal_acc = evaluate(model, val_loader)
    
    # Evaluate with the EMA model
    with torch.no_grad():
        # Compute accuracy on the validation set
        ema_acc = evaluate(ema_model, val_loader)
    
    print(f'Epoch {epoch}:')
    print(f'  Normal Model Accuracy: {normal_acc:.4f}')
    print(f'  EMA Model Accuracy: {ema_acc:.4f}')

In this code, we first create the regular training model and use it to initialize the EMA model. The loss function, optimizer, and other training settings are configured exactly as in ordinary training—nothing special is required for EMA.

Inside the training loop, the regular model is trained in the usual way.

Concretely, that means the usual sequence of computing the loss, computing gradients, and updating parameters. After training on each batch, call ema.update(model) to update the EMA model. This update must always be performed after optimizer.step() has updated the model's parameters.

At the end of each epoch, evaluate both the regular model and the EMA model and compare their performance. You can obtain the EMA model with ema.get_model(). At evaluation time, the EMA model is already in eval() mode, so there is no need to set it again here.

For the decay parameter (β in the original formula), values such as 0.999 or 0.9999 are commonly used. The larger the value, the more stable the model becomes, but the slower it updates. One approach is to use a smaller value such as 0.99 early in training and increase it later.

Implemented this way, the model can make more stable predictions than the regular model. Using the EMA model at test time or for inference in production yields especially stable results. When saving your model, save both the regular model and the EMA model.

Tracking How the Weights Are Updated

import torch
from torch import nn
import torch.optim as optim
from your_model import YourModel
from ema_model import EMAModel

def track_parameter_changes(model, ema_model, param_index=0):
    """Track and display changes in the model's parameters"""
    for name, param in model.named_parameters():
        for ema_name, ema_param in ema_model.ema_model.named_parameters():
            if ema_name == name:
                track_param = param.data.flatten()[param_index]
                ema_val = ema_param.data.flatten()[param_index]
                
                print(f"{name}:")
                print(f"  Current:  {track_param.item():.6f}")
                print(f"  EMA:      {ema_val.item():.6f}")
                break

# Prepare the models
model = YourModel()
ema = EMAModel(model, decay=0.999)

# Training setup
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Training loop
num_epochs = 100
for epoch in range(num_epochs):
    model.train()
    
    for batch_idx, (data, target) in enumerate(train_loader):
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
        
        # Update the EMA model
        ema.update(model)
        
        # Display weight changes every 100 batches
        if batch_idx % 100 == 0:
            print(f"\nEpoch {epoch}, Batch {batch_idx}")
            print(f"Loss: {loss.item():.4f}")
            track_parameter_changes(model, ema)
    
    # Evaluation at the end of the epoch
    model.eval()
    ema_model = ema.get_model()
    
    with torch.no_grad():
        normal_acc = evaluate(model, val_loader)
        ema_acc = evaluate(ema_model, val_loader)
    
    print(f'\nEpoch {epoch} Summary:')
    print(f'  Normal Model Accuracy: {normal_acc:.4f}')
    print(f'  EMA Model Accuracy: {ema_acc:.4f}')
    
    # Display the parameter state at the end of the epoch
    print("\nParameter state at epoch end:")
    track_parameter_changes(model, ema)

If you add the above to the earlier training code:

• It prints the current loss and the state of the weights every 100 batches
• You can also check the state of the weights at the end of each epoch
• track_parameter_changes compares the parameter at the same position in each model and displays the difference between the regular model's and the EMA model's values

Example Output

Epoch 0, Batch 0
Loss: 2.3045
conv1.weight:
  Current:  0.023456
  EMA:      0.023433

Epoch 0, Batch 100
Loss: 1.9876
conv1.weight:
  Current:  0.025678
  EMA:      0.024123
...

In this example output, we can first see the training progressing: the loss decreases from 2.3045 to 1.9876, showing that the model is indeed learning.

As for the weight changes, taking the conv1.weight parameter as an example, the regular model's value (Current) changes relatively sharply from 0.023456 to 0.025678, while the EMA model's value moves more gently, from 0.023433 to 0.024123.

This illustrates EMA's characteristic effect of damping abrupt parameter changes. While the regular model's parameters fluctuate significantly, the numbers confirm that the EMA model's parameters are updated more smoothly and gently. Thanks to these stable updates, the EMA model is less affected by noise and can make more stable predictions.

Real-World Use Cases

class StableDiffusionTrainer:
    def __init__(self):
        self.model = DiffusionModel()
        self.ema_model = EMAModel(self.model)
    
    def train_step(self, prompt, image):
        # Regular training
        self.optimizer.zero_grad()
        loss = self.model.train_step(prompt, image)
        loss.backward()
        self.optimizer.step()
        
        # Update the EMA model
        self.ema_model.update(self.model)
    
    def generate(self, prompt):
        # Use the EMA model for inference
        return self.ema_model.ema_model.generate(prompt)

EMA in Image and Video Generation Models

In image generation models—particularly diffusion models such as Stable Diffusion—EMA plays an important role. Here is an example implementation with commentary.

class StableDiffusionTrainer:
    def __init__(self):
        self.model = DiffusionModel()
        self.ema_model = EMAModel(self.model)
    
    def train_step(self, prompt, image):
        # Regular training
        self.optimizer.zero_grad()
        loss = self.model.train_step(prompt, image)
        loss.backward()
        self.optimizer.step()
        
        # Update the EMA model
        self.ema_model.update(self.model)
    
    def generate(self, prompt):
        # Use the EMA model for inference
        return self.ema_model.ema_model.generate(prompt)

Using EMA in Image Generation Models

In image generation models—particularly diffusion models such as Stable Diffusion—EMA plays an important role. Here is an example implementation with commentary.

class StableDiffusionTrainer:
    def __init__(self):
        self.model = DiffusionModel()
        self.ema_model = EMAModel(self.model)
    
    def train_step(self, prompt, image):
        # Regular training
        self.optimizer.zero_grad()
        loss = self.model.train_step(prompt, image)
        loss.backward()
        self.optimizer.step()
        
        # Update the EMA model
        self.ema_model.update(self.model)
    
    def generate(self, prompt):
        # Use the EMA model for inference
        return self.ema_model.ema_model.generate(prompt)

This code is a simplified implementation of the Stable Diffusion training process. In the class initializer, two models are created: the main DiffusionModel that actually trains, and an EMAModel for applying EMA.

In the training step, ordinary gradient-descent training runs first: optimizer.zero_grad resets the gradients, the model's training step computes the loss, and then backward and the optimizer step are executed. After this regular training, the EMA model's weights are updated. This update happens after every training step, gradually forming a stable set of weights.

At generation (inference) time, as explained above, the EMA model is used rather than the regular model. That's because the EMA model has had the noise of training smoothed away, enabling more stable generation.

This kind of implementation matters because, especially when training image generation models, the quality of the generated images often fluctuates substantially during training.

It is genuinely common for a model that is producing good-looking images at one point to see quality drop after the next update. The cause is abrupt changes in the weights during training.

Using EMA smooths the weight updates and stabilizes the quality of the generated images. The stability of the output improves, and the influence of noise in the training data and batch-to-batch variation is reduced.

A Practical Example

In real Stable Diffusion training, the implementation often looks like this:

class DiffusionTrainer:
    def __init__(self):
        # Initialize the model
        self.model = UNet()
        # Set up the EMA model (with a relatively high decay rate)
        self.ema_model = EMAModel(self.model, decay=0.9999)
        
    def training_loop(self, dataloader):
        for epoch in range(num_epochs):
            for batch in dataloader:
                # Regular training step
                loss = self.train_step(batch)
                
                # Update the EMA
                self.ema_model.update(self.model)
                
                if self.steps % 1000 == 0:
                    # Periodically check generated samples
                    with torch.no_grad():
                        # Generate images using the EMA model
                        samples = self.sample_images(
                            self.ema_model.get_model()
                        )
                
    def sample_images(self, model):
        """Image generation
        Uses the EMA model instead of the regular model"""
        return model.generate(...)

When using EMA in image generation, a few important points deserve attention. For image generation, very high decay values such as 0.9999 are common (at that level, the simulator above would barely show any visible difference), which makes the weight updates even gentler. It is also important to periodically generate images with the EMA model to check quality.

In this way, EMA has become a key technique for improving the quality and stability of image generation models, and for large models that require long training runs in particular, its use is established as standard practice.

Conclusion

This has grown a little long, so let's wrap things up.

First, a quick review in Q&A form.

Q: What is EMA, and why is it used in deep learning?

EMA (Exponential Moving Average) is a technique for averaging a model's weights. In deep learning, it is used primarily to improve stability at inference time.

Q: How does it relate to SGD? Is it a recent technique?

When it comes to stabilization, a question we often hear is: "What is the relationship between EMA and SGD?"

The short answer: they belong to different contexts.

SGD (stochastic gradient descent) is a method for updating weights during training—an optimizer. It has many variants, such as Adam and SGD with momentum.

EMA, meanwhile, is a technique for averaging weights that have already been learned. (This is the part people tend to find confusing.)

If you've followed the article this far, you probably have this down already, but to restate:

• SGD is used to update the weights during training
• EMA is used to smooth the weights after training updates

That's the division of labor.

They are used together in the pattern "train with SGD" → "apply EMA to the result" → (repeat).

Q: Is it only used for inference?

Yes—
EMA is used mainly at inference time.

This is because of the trade-off that

- training requires rapid adaptation to new information
- inference requires stable output

.

In Closing: Why EMA Matters, and Where It Is Applied

In deep learning, EMA is an important—and elegant—technique for reconciling two conflicting demands: flexibility during training and stability during inference.

Its hallmark is delivering two benefits at once: fast learning with the regular model and stable inference with the EMA model.

The technique has produced major results particularly in generative domains such as image and video generation. In generative models like Stable Diffusion and GANs, output stability is critical, and adopting EMA has made consistent, high-quality generation possible.

EMA is also put to work in self-supervised learning, enabling more stable learning of feature representations.

EMA's great advantages are that it is relatively simple to implement, computationally cheap, and memory-efficient. Its versatility—it can be applied across a wide range of deep learning methods—also deserves special mention.

Looking ahead, we can expect its use in ever-larger models, combinations with new training algorithms, and research into adaptive decay-rate scheduling. Simple as the mechanism is, its impact is enormous, and it is firmly established as a standard technique in modern deep learning. Our video generation AI, MotionVox™, also builds an EMA model during training, smoothing parameter updates to achieve a high degree of stability!

Thank you for reading to the end!

See you next time!