Starting From Noise: The Trick Behind Diffusion

Here's the part that surprises most people the first time they hear it: every image a diffusion model generates starts as pure static. Not a rough sketch. Not a blurry outline. Random, meaningless visual noise, the kind of fuzzy snow you'd see on an old TV with no signal.

That seems like a strange place to start if your goal is a crisp picture of a golden retriever in a birthday hat. This lesson explains why starting from noise is actually the key that makes the whole system work, and how a model learns to see a hidden image inside what looks like nothing at all.

What You'll Learn

• Why diffusion models generate images by removing noise rather than adding detail
• How a model is trained to "undo" noise, using millions of examples
• Why starting from randomness produces originality, not garbage
• The difference between training a diffusion model and using one

Two Ways to Think About Generating an Image

Imagine two different ways an AI could try to draw a golden retriever:

Approach A: Build up from nothing. Start with a blank canvas and add pixels one at a time, deciding what color each one should be, in order, left to right. This is roughly how older text-generation AI writes a sentence, one word after another. It's slow, and small early mistakes compound as you go.

Approach B: Refine down from noise. Start with an entire canvas of random static, then look at the whole image at once and ask, "What would make this look slightly more like a photo, and slightly less like noise?" Nudge every pixel a little in that direction. Repeat that same question dozens of times, each time working from a slightly cleaner version of the previous answer.

Diffusion models use Approach B. Instead of committing to details early and hoping they hold up, the model works on the whole image simultaneously and gradually sharpens it. Rough shapes emerge first, and fine detail arrives last, closer to the way a photograph develops in a darkroom than the way you'd draw with a pencil.

Training: Teaching a Model to Undo Noise

To understand why this works, you have to look at how a diffusion model is trained, before it ever generates anything new.

Researchers start with a massive dataset of real photos and, separately, descriptions of what's in them. For each photo, they run a simple, repeatable process:

1. Take a clear photo of, say, a dog.
2. Add a small amount of random noise to it. The photo now looks almost the same, just slightly grainy.
3. Add more noise. Now it's noticeably degraded.
4. Keep adding noise in small increments, dozens of times, until the original photo is completely gone and all that's left is pure static.

This is the "forward process" you saw mentioned in the last lesson, and at training time, it's a known, mechanical procedure. At every single step, the researchers know exactly how much noise they added and exactly what the image looked like one step earlier, since they did it themselves.

That "one step earlier" answer is the training target. The model is shown a noisy image and asked: "Given this, predict what the slightly-less-noisy version one step back looked like." It makes a guess, the guess is compared against the real answer (which the researchers already know, because they created the noise on purpose), and the model's internal settings are adjusted to make its next guess a little better. Repeated across millions of images and billions of these noise-prediction guesses, the model gradually gets very good at one specific skill: given a noisy image, predict what noise was added, so it can be subtracted back out.

That's it. That's the entire trained skill. A diffusion model doesn't memorize a library of finished pictures. It learns a general, reusable ability to estimate "what noise is in this, and what would this look like with a bit less of it."

Why Randomness Produces Originality

This explains something that might otherwise seem magical: why a diffusion model can generate a picture that has never existed before, instead of just recombining pieces of photos it was trained on.

Because the model learned a general denoising skill rather than memorizing specific images, it can apply that skill to any starting noise pattern, including a completely new, randomly generated one that was never part of training. Change the random starting static even slightly, and you get a different, equally valid image on the other end. That's why regenerating the same prompt gives you a different picture each time: the starting noise is different, so the denoising process arrives somewhere new.

This also explains why the random noise isn't garbage waiting to be discarded, it's closer to a seed. It's the source of variation and originality in the system. Two people with the identical prompt get two different images because they're starting from two different patches of static.

From Training to Generating

Once a model has learned this denoising skill, generating a brand-new image works by running that same skill in reverse, starting from scratch:

1. Generate a canvas of pure random noise (no photo involved this time, just static).
2. Ask the trained model: "What noise do you predict is hiding a clearer image, based on everything you learned in training?"
3. Subtract out that predicted noise, revealing a slightly less noisy image.
4. Feed that result back in and repeat the question, dozens of times.

Each round, the image gets a little clearer. The next lesson walks through exactly what those intermediate steps look like and what the model is actually paying attention to along the way.

Key Takeaways

• Diffusion models generate images by starting from random noise and gradually removing it, rather than building an image up from a blank canvas.
• During training, researchers add noise to real photos in small steps and teach the model to predict and reverse each step.
• The model learns a general "predict and remove noise" skill, not a memorized library of images, which is why it can create genuinely new pictures.
• Starting from different random noise, even with the same prompt, produces a different image, that's the source of variety in AI-generated art.
• Generating an image reuses the same denoising skill learned in training, just run from pure static instead of a real photo.