From Image Diffusion to Video

AI video tools start from the same idea as AI image tools. They begin with random noise and clean it up, step by step, until a picture appears. The big difference is that a video model has to clean up many pictures at once and make them fit together in time.

This lesson shows how that one change, from one picture to a stack of pictures, shapes everything else about AI video. You do not need any code or math.

What You'll Learn

• A quick recap of how image diffusion works
• Why a video is harder than many separate images
• How a video model cleans up a whole clip at once
• What latent space means for video, and why it matters
• How video models learn from clips and captions

A quick recap: how image diffusion works

Most AI image tools use a diffusion model. During training, the model sees real images with more and more noise added. It learns to predict the noise so it can remove it. To make a new image, it starts from pure noise and removes a little at a time. Your text prompt steers each step, so the picture moves toward what you asked for.

That is the whole idea in one paragraph. If you want the full picture, with noise schedules and the denoising loop, take How Diffusion Models & Image Generation Actually Work. This course builds on it and focuses on what changes when the output has to move.

A video is a stack of frames

A video is a series of still images called frames, shown fast enough that your eye sees motion. Many videos use about 24 to 30 frames for every second.

So a 5 second clip is not one picture. It is well over 100 pictures. And they cannot be any pictures. Each frame has to look almost the same as the one before it, with small, sensible changes:

• The same person, with the same face and clothes
• The same room, with objects in the same places
• Movement that flows smoothly, without jumps

This link between frames is called temporal consistency. "Temporal" means "related to time." It is the main new problem in AI video, and the next lesson is all about it.

Why not make one frame at a time?

The obvious idea is to generate frame 1, then frame 2, then frame 3, each one with an image model. This does not work well. Each frame would be a fresh guess. The person's shirt might change color. The background might shift. The clip would flicker like a slideshow of slightly different photos.

Early AI video looked a lot like this. Modern video models take a different path.

Denoising a whole clip at once

Most modern video models treat the clip as one block of noise that has both space (the width and height of each frame) and time (the frames in order). The model removes noise from the whole block together.

At every step, the model can look across frames. When it cleans up a face in frame 40, it can check what that face looks like in frames 39 and 41. This is how it keeps things steady over time.

Many recent models use a transformer to do the cleaning. A transformer is good at comparing many pieces of data with each other. Some models cut the clip into small chunks, sometimes called patches, that each cover a small area of the picture across a few frames. The model then compares every chunk with the others to decide what each one should become.

Latent space: why video gets squeezed first

Working on raw pixels would be far too slow for video. A single high quality frame has millions of pixel values. A clip has hundreds of frames.

So video models, like many image models, work in latent space. Before any denoising, a separate part of the system learns to squeeze video into a much smaller form and to expand it back again. The squeezed form keeps the important information, like shapes, colors, and movement, and drops detail the model can rebuild later.

For video, this squeezing happens in two directions:

• In space. Each frame becomes a much smaller grid of numbers.
• In time. Frames that look almost the same can be stored together, since little changes between them.

The model does all its denoising in this small space. At the end, a decoder expands the result back into full frames you can watch. This is the "Decode" step in the diagram above.

This is also why some AI video looks soft or smeared in fine detail. If the squeezing drops too much, small things like text, fingers, or distant faces can come back blurry or wrong.

How video models learn

A video model learns from a huge number of video clips paired with text that describes them. Many of these descriptions are written by another AI that watches the clip and writes a detailed caption. Better captions help the model connect words like "slow pan" or "running toward the camera" to what happens on screen.

From these examples, the model learns:

• What things look like, much like an image model
• How things usually move: walking, waves, smoke, traffic
• How cameras usually move: pans, zooms, tracking shots

It does not learn rules of physics. It learns patterns of what videos tend to look like. That difference explains a lot of the strange mistakes you will see in lesson 3.

What about sound?

Some newer video models also make sound that matches the picture, like footsteps, wind, or speech. The idea is similar. The model learns how sound and picture go together in real videos, and it generates both so they line up in time. Sound is still a newer feature, and quality varies a lot between tools.

Where you see this

Tools like Sora, Veo, Runway, and Kling all make video from prompts. Each company uses its own design, and few share full details. But the broad ideas in this lesson, starting from noise, cleaning up the whole clip, and working in latent space, describe how most of today's video models work.

If you want to learn to use these tools hands-on, the AI Video Generation & Editing course covers that. This course stays on how they work underneath.

Key Takeaways

• AI video builds on image diffusion: start from noise and remove it step by step, guided by a prompt.
• A video is a stack of frames, and each frame has to fit the ones around it. This is called temporal consistency.
• Making one frame at a time causes flicker, so modern models denoise the whole clip at once and look across frames.
• Video models work in latent space, which squeezes video in both space and time, then a decoder turns it back into frames.
• Models learn from huge numbers of clips and captions. They learn patterns of how things look and move, not the rules of physics.