How It Works, in Plain Language

You now know what multimodal means: one model that handles text, images, and audio together. The natural next question is how. How can a single system read a sentence and look at a photo and treat them as the same kind of thing?

The honest answer involves a lot of math. The good news is you do not need any of it. There is a simple intuition that gets you 90 percent of the way, and this lesson is that intuition. No code. No equations.

What You'll Learn

• The one core trick: everything becomes numbers
• Why turning text, images, and sound into numbers lets one model handle all of them
• What "reasoning across modalities" means without the math
• A mental model you can reuse for the rest of the course

The one trick: everything becomes numbers

Here is the whole secret in a sentence. An AI model does not understand text, images, or sound directly. It turns all of them into numbers first, then works with the numbers.

A computer has never actually "seen" a cat or "heard" a word. Deep down it only handles numbers. So before a model can work with anything, that thing has to be converted into a long list of numbers.

• A sentence gets chopped into small pieces called tokens, and each token becomes numbers.
• An image gets broken into small patches, and each patch becomes numbers describing its colors and edges.
• A sound gets sliced into tiny time-slivers, and each sliver becomes numbers describing its pitch and volume.

Different senses, same destination. Words, pixels, and sound waves all end up as numbers.

Why turning everything into numbers is such a big deal

Once your typed question, your uploaded photo, and your voice note are all just numbers, a single model can line them up side by side and treat them as one big pile of information.

Imagine three people who speak different languages: one speaks only text, one speaks only images, one speaks only sound. On their own they cannot talk to each other. Now imagine they all learn one shared language. Suddenly they can sit at the same table and discuss the same problem.

Numbers are that shared language. This shared space is the thing that makes multimodal AI possible. It is why the model can look at a photo of your fridge and your typed question "what can I cook tonight?" and answer sensibly. Both the photo and the question are living in the same numeric space, so the model can connect them.

You may hear this shared numeric space called an embedding space. You do not need the term. Just remember: everything gets translated into the same kind of numbers, and that is what lets one model handle it all.

What "reasoning across modalities" really means

"Reasoning across modalities" sounds impressive. In plain terms it means the model can let one input inform another.

• Your words tell it what to look for in the image ("is the sky in this photo cloudy?").
• The image gives it facts your words did not include (the actual color of the sky).
• It blends both to produce an answer neither could give alone.

This is exactly what you do when a friend hands you a photo and asks a question. The question shapes how you look. The photo gives you the details. You answer using both. The model is imitating that blend.

The mental model to carry forward

Here is the picture to keep in your head for the rest of the course:

1. Whatever you give the model, it turns into numbers.
2. All those numbers live in one shared space.
3. In that shared space, your text, your image, and your audio can be compared and connected.
4. The model uses those connections to predict a helpful response, one piece at a time.

That last point matters and comes with a warning we will return to later: the model is predicting a likely response, not looking up a guaranteed fact. That is why it can be fluent and confident and still be wrong. Hold that thought. It becomes important when we talk about limits and how to verify what the model tells you.

If you want to go deeper on how the text side of this works, the free course How LLMs Actually Work covers tokens and prediction in more detail. For this course, the four-step picture above is all you need.

Key Takeaways

• A model does not read text or see images directly. It turns everything into numbers first.
• Text, images, and audio all become numbers that live in one shared space.
• That shared space is what lets a single model connect your words to your picture to your voice.
• Reasoning across modalities just means letting one input inform another, the way you do naturally.
• The model predicts a likely answer, which is powerful but also why it can be confidently wrong.