The Evolution of Image Generation Technology: From GANs to Diffusion Models

Hello! This is the Qualiteg Research Team!

Today we would like to talk about image generation technology.

As many of you know, AI-based image generation technology has evolved rapidly over the past decade.

It is now 2024—exactly ten years since GANs were introduced in 2014.

While text-based generative AI is getting most of the attention these days, the computer vision field has also made great strides, from image generation AI and super-resolution AI to video generation AI.

In this article, we look back at the historical development of image generation technology, focusing mainly on Generative Adversarial Networks (GANs) and diffusion models. In particular, we take a closer look at how these technologies have been applied in the fields of image generation and super-resolution.

Before the main article—we have a digest video!

We cover the highlights of this article in the video below, so please feel free to check it out as well.

GANs and Diffusion Models

What Is a GAN?

A GAN (Generative Adversarial Network) is a deep learning model proposed in 2014 by Ian Goodfellow and his colleagues. After working at Google Brain, he reportedly joined Apple's machine learning research team in March 2019.

A GAN consists of the following two main components:

1. Generator
   Generates realistic-looking data (e.g., images) from random noise.
2. Discriminator
   Determines whether given data is real or fake (i.e., produced by the Generator).

The GAN training process proceeds as the Generator and Discriminator compete with each other:

• The Generator tries to produce data that fools the Discriminator.
• The Discriminator tries to distinguish real data from data produced by the Generator.

Through this adversarial training process, the Generator gradually learns to produce high-quality data that closely resembles the real thing.

Key Characteristics of GANs

• Capable of generating high-quality, diverse data
• Functions as a form of unsupervised learning
• Applicable to a wide range of tasks (image generation, style transfer, super-resolution, and more)

Next, let's look at diffusion models.

Diffusion Models

The theoretical foundations of diffusion models were proposed in 2015, and they began attracting attention in the field of image generation around 2020.

In fact, the theory itself was conceived around the same time as GANs.

1. Reverse diffusion process
   Learns the process of gradually generating an image from noise.
2. Score function
   Directly models the gradient of the image probability density (the score function).

The Training Process of Diffusion Models

• Simulates the process of gradually adding noise to an image (the forward diffusion process).
• Models the reverse of that process (the reverse diffusion process) and learns how to remove the noise.

Rather than directly modeling the probability density function p(x) over image space, diffusion models learn the score function s(x) := ∇x ln p(x). This approach offers the following advantages:

1. No normalization required
   C·p(x) can be used in place of p(x), and the specific value of the constant C has no effect on the computation.
2. Local comparison
   Makes it easy to compare probabilities with neighboring images.

The score function is learned through a noising-denoising process, in which Gaussian noise is added to an image and the model learns to remove it.

Major Variants of Diffusion Models

1. Classifier guidance
   • Enables conditional image generation
   • Derives p(x|y) ∝ p(y|x)p(x) using Bayes' theorem
2. Introducing temperature
   • Controls the quality of generated images using an inverse temperature β
   • The larger β is, the closer the generated image is to the maximum likelihood estimate
3. Classifier-free guidance (CFG)
   • Achieves conditional generation without an external classifier
   • Extracts the classifier from the model itself

Comparing GANs and Diffusion Models

GANs and diffusion models are both groundbreaking technologies in the field of image generation, but their approaches and characteristics differ significantly.

GANs are characterized by "adversarial training," in which two networks—a Generator and a Discriminator—learn by competing with each other. This scheme allows GANs to generate images quickly and produce sharp results. However, training tends to be unstable, and GANs suffer from "mode collapse," a problem that limits the diversity of the generated images.

Diffusion models, on the other hand, use a single denoising network and learn the process of gradually transforming noise into an image. This approach makes training more stable and makes it easier to generate highly diverse images. Fine-grained control is also possible at each stage of the generation process, which makes diffusion models well suited to more complex tasks such as text-to-image generation. The trade-off is that the multi-step generation process makes image generation slower than with GANs and requires more computational resources.

The Evolution of GANs and Diffusion Models

Now let's look at the major milestones for GANs and diffusion models in chronological order.

The evolution of GANs and diffusion models vividly illustrates the progress of image generation technology.

GANs were proposed by Ian Goodfellow and colleagues in 2014 and revolutionized the field of image generation. DCGAN arrived the following year, delivering more stable training and higher-quality image generation. In 2017, CycleGAN enabled image-to-image translation, Wasserstein GAN improved training stability, and SRGAN brought GANs to super-resolution, greatly expanding the range of GAN applications. In 2018, ProGAN made high-resolution image generation possible, and StyleGAN in 2019 enabled the generation of images with diverse styles and features.

Diffusion models, meanwhile, had their theoretical foundations—drawing on non-equilibrium thermodynamics—proposed in 2015, but it took some time before they attracted attention in the field of image generation. DDPM (Denoising Diffusion Probabilistic Models) was proposed in 2019 and further improved by Jonathan Ho and colleagues in 2020. In 2021, OpenAI developed GLIDE, dramatically improving text-to-image generation capabilities; the super-resolution model SR3 was also proposed that same year. In 2022, the release of Stable Diffusion by Stability AI made high-quality image generation available to the general public.

Over this period, GANs achieved high-quality image generation early on and found a wide variety of applications. Diffusion models, though later arrivals, developed rapidly on the strength of stable training and high controllability, and in recent years they have been becoming the mainstream of image generation. Both technologies continue to take on more complex tasks such as text-to-image generation and super-resolution, and further evolution is expected.

Super-Resolution: The Technology and Its Evolution

Super-resolution is a technology that generates high-resolution images from low-resolution ones.
Both GANs and diffusion models play important roles in this field as well, so let's take a look.

Why Super-Resolution Matters

There are several reasons why super-resolution technology is considered important.

1. Improving the quality of existing low-resolution images
   Old photos and videos, as well as low-quality digital images, can be upscaled to restore fine detail.
2. Overcoming hardware limitations
   Low-resolution images resulting from the physical limitations of cameras and sensors can be upscaled in software.
3. More efficient data transfer and storage
   By transferring and storing images at low resolution and converting them to high resolution only when needed, the load on networks and storage can be reduced.
4. Scaling up the output of machine learning models
   Many image generation models, such as those based on VGG, are trained on relatively small images (e.g., 224x224 pixels) for computational efficiency. When these models are used to generate images for real videos or large displays, the generated images must be enlarged, and super-resolution technology plays a key role in that enlargement process.

Temporal Super-Resolution for Video

In recent years, with the growing demand for video content, development of temporal super-resolution technology for video has accelerated. Rather than simply upscaling each frame independently, this technology aims to produce high-quality video by taking the temporal continuity between frames into account.

1. Maintaining inter-frame consistency
   In temporal super-resolution, it is important to upscale each frame while preserving consistency between adjacent frames. This produces smooth video free of flicker and artifacts.
2. Motion compensation and estimation:
   By accurately estimating object motion and integrating information across frames accordingly, finer and more natural movement can be reproduced.
3. Spatiotemporal feature extraction
   Using 3D convolutional neural networks and recurrent networks, features are extracted along the time axis as well, taking advantage of richer spatiotemporal information.
4. The challenge of real-time processing
   To support video streaming and live broadcasting, fast temporal super-resolution algorithms that run in real time are under development.
5. Accounting for long-range temporal dependencies
   To maintain consistency over longer time scales, researchers are also studying super-resolution methods that take a wider range of frames into account.

Recent Research Trends

1. TecoGAN (Temporally Coherent GAN)
   A GAN-based model that achieves high-quality video super-resolution while maintaining temporal consistency, balancing smooth transitions between frames with the preservation of fine detail.
2. EDVR (Enhanced Deformable Convolutional Networks)
   Achieves high-quality video super-resolution by processing multiple frames simultaneously and performing inter-frame alignment.
3. BasicVSR++
   Combines bidirectional propagation with deformable convolutions to enable video super-resolution that accounts for long-range temporal dependencies.
4. RSTT (Recurrent Structure-Texture Learning)
   Learns structural and textural information separately, achieving video super-resolution that is sharp down to fine details while maintaining temporal consistency.
5. Applying diffusion models: Recently, research has begun on applying diffusion models—which have been so successful in image generation—to video super-resolution. Their potential to produce high-quality, diverse results while maintaining temporal consistency is drawing attention.

These technological advances are expected to find applications not only in improving video quality but also in a wide range of fields such as VR and AR, medical imaging, and surveillance video analysis. Looking ahead to the spread of high-speed communication technologies such as 5G and IOWN, they also suggest the possibility of new video delivery models in which video is transferred efficiently at low resolution and upscaled on the receiving side.

Temporal super-resolution is more than just a way to improve image quality—it is a key technology that opens up new possibilities for video content, and we can expect it to continue evolving.

Super-Resolution with GANs

Super-Resolution with Diffusion Models

Comparing GANs and Diffusion Models

Looking Ahead

Image generation technology continues to evolve rapidly, and we can expect further improvements in quality and diversity, as well as integration with other AI technologies, in areas such as the following.

1. Multimodal generation
   Comprehensive generation combining text, audio, and video
   As we also discuss in the video below, we expect end-to-end (E2E) training to eventually become the mainstream approach.

1. Interactive generation
   Generating and editing images in real time in response to user input
2. Ethical considerations: preventing misuse such as deepfakes and addressing copyright issues
3. Improving computational efficiency: developing techniques that enable high-quality generation with fewer resources
4. Deepening theoretical understanding: further clarifying the mathematical foundations of diffusion models
5. Further evolution of super-resolution: achieving both higher upscaling factors and higher quality

Conclusion

From GANs to diffusion models, image generation technology has made astonishing progress over the past decade.
Diffusion models in particular, with their solid mathematical foundations and flexible controllability, are becoming the new standard for image generation and super-resolution.

These technologies are bringing innovation to a wide range of fields, including art, entertainment, and scientific research.

In super-resolution, GAN-based models were long the mainstream, but approaches using diffusion models have recently been attracting attention as well. The two approaches have different characteristics and are chosen according to the use case.

While we look forward to continued technical evolution and the opening of new fields of application, we must also pay attention to the ethical challenges involved. Image generation technology is a powerful tool that extends human creativity and opens up new possibilities for expression, and we intend to continue putting it to active use!