Why Diffusion Won: GANs, VAEs, and the Diffusion Advantage

The first lesson in this course mentioned that diffusion wasn't the only approach AI researchers tried for generating images, it's simply the one that won out. This final lesson closes the loop by looking at the two major alternatives, GANs and VAEs, and explaining at a conceptual level why the noise-and-denoise approach you've spent this whole course learning about ended up on top.

You don't need to become an expert on these older methods. The goal is context: understanding what came before helps you appreciate why diffusion's specific design choices, several of which probably seemed a little unusual when you first read about them, turned out to matter so much.

What You'll Learn

• What a GAN is and the basic idea behind how it generates images
• What a VAE is and how it relates to the "latent space" idea from earlier
• The specific practical problems that limited both older approaches
• Why diffusion's step-by-step design avoided those problems

GANs: Two AI Systems Competing Against Each Other

A GAN, short for Generative Adversarial Network, takes a fundamentally different approach than diffusion. Instead of one model learning to denoise, a GAN pits two models against each other in a competition:

• A generator tries to produce fake images.
• A discriminator tries to tell the difference between the generator's fake images and real photos.

The two are trained together: every time the discriminator correctly catches a fake, the generator adjusts to get more convincing. Every time the generator fools the discriminator, the discriminator adjusts to get better at spotting fakes. Run over enough rounds, the idea is that the generator becomes good enough to produce images a human (or the discriminator) can't distinguish from real ones. It's sometimes compared to a forger and an art detective locked in an ongoing rivalry, each one getting better because of the other.

For several years in the late 2010s, GANs produced the most impressive AI-generated images available, and they're still used today in some specialized applications. But this adversarial setup, one model trying to fool another, turned out to be notoriously unstable to train. Common failure modes included:

• Mode collapse: the generator discovers a small handful of images that reliably fool the discriminator and starts producing only slight variations of those, losing diversity entirely.
• Training instability: because the two models are locked in a constantly shifting back-and-forth competition, training can oscillate or fall apart entirely rather than steadily improving, which made GANs difficult and expensive to train reliably at large scale.

VAEs: An Earlier Version of "Compressed Space"

A VAE, or Variational Autoencoder, will sound more familiar, because you already learned its core idea in the latent space lesson: an encoder compresses an image into a compact representation, and a decoder expands it back out. That compress-and-reconstruct structure is, in fact, exactly where the "encoder" and "decoder" terminology used in modern latent diffusion came from.

VAEs generate new images by taking a random point in that compact latent space and decoding it directly into an image, no denoising loop involved. The practical problem was output quality: VAE-generated images had a well-known tendency to look noticeably blurry and lack sharp, fine detail, compared to both GANs and, later, diffusion models. The compression was effective, but decoding straight from a random point to a finished image in a single step, without any equivalent of the gradual refinement you learned about in this course, tended to average out detail rather than sharpen it.

Why Diffusion Avoided Both Problems

With those two approaches in mind, you can now see how several design choices from earlier lessons, which might have seemed arbitrary at the time, are direct solutions to GAN and VAE weaknesses:

• No adversarial competition, so no instability. Diffusion training (lesson 2) is a single model learning a well-defined, stable prediction task, "given this noisy image, predict the noise", against a known correct answer. There's no second model to destabilize the process, which is a major reason diffusion models train far more reliably than GANs.
• Gradual refinement, so no blurriness. Rather than decoding a random point directly into a finished image in one shot like a VAE, diffusion spends dozens of steps (lessons 3 and 6) progressively sharpening the result, coarse structure first, fine detail last. That gradual process is a big part of why diffusion output looks sharper and more detailed than typical VAE output.
• Genuine diversity, so no mode collapse. Because the model learns a general denoising skill rather than converging on a small set of images that fool a competitor (lesson 2), different random starting noise reliably produces different, varied results, without the repetitive-output problem GANs were prone to.

It's worth noting that modern latent diffusion, the specific approach this course has focused on, actually borrows the best idea from VAEs (compress into an efficient latent space) and combines it with the best idea from diffusion's own denoising approach (gradual, stable refinement) rather than discarding VAE thinking entirely. That combination is a big part of why it's the dominant approach today: it inherited genuine strengths from what came before it, while fixing their most limiting weaknesses.

Where This Leaves You

Across this course, you've gone from "type a sentence, get an image" to a complete, no-math picture of the actual mechanism: random noise, a trained denoising skill, a compressed latent space to make it fast, a prompt that guides every step, a planned schedule that calibrates each step, and a design that learned from the strengths and weaknesses of what came before it. That's genuinely the core of how essentially every major image generator available today actually works under the hood.

If you want to put this understanding into practice, the tool-focused courses on this platform, AI Image Generation for Beginners and Midjourney & DALL-E Mastery, pick up exactly where this one leaves off, now with the added context of why their prompting techniques work the way they do. And if you're curious about the other direction of visual AI, how a model reads and understands an image that already exists, rather than generating one, Computer Vision Basics: How AI Sees covers that side in the same no-code style as this course.

Key Takeaways

• GANs generate images through two competing models (a generator and a discriminator), which produced impressive results but were unstable to train and prone to mode collapse.
• VAEs use the same encoder-decoder compression idea behind latent space, but decoding directly from a random point in one step tends to produce blurry results.
• Diffusion avoids GAN-style instability because it's a single model learning a stable, well-defined prediction task with no adversarial competition.
• Diffusion avoids VAE-style blurriness because it refines an image gradually over dozens of steps instead of decoding it in one shot.
• Modern latent diffusion combines the efficient compression idea from VAEs with diffusion's own stable, gradual refinement, which is a major reason it became the dominant approach.