How Speech Becomes Text

You talk to your phone, and a second later your words appear on the screen. Your meeting app writes a transcript while people speak. A video gets captions without anyone typing them.

All of these use speech recognition, also called speech-to-text. This lesson explains how a computer turns the sound of your voice into written words. No code and no math, just the main ideas.

What You'll Learn

• What sound looks like to a computer
• What a spectrogram is and why AI models use it
• How a modern speech recognition model turns sound into words
• Why models trained on huge amounts of audio work better
• Where speech recognition still makes guesses

Sound is a wave of air pressure

When you speak, your voice pushes air back and forth. That movement travels as a sound wave. A microphone measures the wave thousands of times every second and stores each measurement as a number.

So to a computer, one second of your voice is a long list of numbers. On its own, that list is hard to work with. The numbers go up and down very fast, and the same word said by two people gives very different lists.

The model needs a better view of the sound. That view is the spectrogram.

The spectrogram: a picture of sound

A spectrogram is a picture that shows which sounds are present at each moment.

• Time runs from left to right.
• Pitch runs from bottom to top. Low sounds sit near the bottom, high sounds near the top.
• Brightness shows how loud each pitch is at that moment.

Think of it like sheet music drawn by a machine. A deep "oo" sound shows bright bands low down. A sharp "s" sound shows a fuzzy patch high up. Silence is dark.

Each speech sound leaves its own pattern on this picture. That is why a spectrogram helps. It turns a messy wave into shapes that a model can learn to read, much like a model learns to read shapes in a photo.

Many speech models use a version called a mel spectrogram. It spaces the pitches the way human hearing does, with more detail in the range where speech lives.

How the model reads the picture

Older systems split the job into separate parts. One part matched sounds to small speech units. Another part used a dictionary of how words are spoken. A third part guessed which word order made sense. Each part was built and tuned by hand.

Modern systems are mostly end-to-end. One neural network takes in the spectrogram and writes out text directly. It learns everything from examples: audio clips paired with correct transcripts.

A well-known example is Whisper, a speech recognition model released by OpenAI. Its design is a good picture of how many modern models work:

1. The audio is cut into short chunks and turned into spectrograms.
2. An encoder reads the spectrogram and builds an understanding of the sounds in it.
3. A decoder writes the text one piece at a time, each time looking back at the sounds and at the words it already wrote.

If that decoder sounds familiar, it should. It works much like the next-token prediction you may know from chatbots. How LLMs Actually Work explains that idea in detail. A speech model is predicting the most likely next words, but it is guided by the sound.

Why context helps the model

Because the decoder looks at the words it already wrote, it uses context. Say you hear a sound that could be "write" or "right." If the sentence so far is "Please turn", the model picks "right." If it is "I need to", it picks "write."

The encoder also hears the whole audio chunk, not just the sound so far. So a later word can help too. In "Please write a letter", the word "letter" later in the clip is one more clue that the sound was "write."

This is also why speech recognition is better on full sentences than on single words. One short word with no context is harder to get right.

Why more training audio matters

A speech model learns from audio paired with text. The more varied that audio is, the more situations the model can handle.

• Many speakers teach it that the same word can sound different.
• Many accents teach it the range of ways people speak a language.
• Noisy recordings teach it to ignore background sounds.
• Many languages let one model handle more than one language.

Large modern models learn from hundreds of thousands of hours of speech. That variety is a big reason they handle real life far better than older systems did.

But variety has limits. If an accent, a language, or a type of speech is rare in the training audio, the model has seen fewer examples of it. It will make more mistakes there. You will see more on this in the last lesson.

Extra jobs a speech system often does

Raw speech recognition gives you a stream of words. Real products usually add more:

• Punctuation and capitals, so the text reads like a sentence.
• Timestamps, so captions line up with the video.
• Speaker labels, so a transcript shows who said what. This step is often called diarization.
• Language detection, so the system knows which language it is hearing.

Some of these come from the same model. Others are separate steps. Either way, each one is another place where small mistakes can happen.

The model is always guessing

It is easy to think a transcript is a recording of what was said. It is not. It is the model's best guess about what was said.

Most of the time the guess is right. But when the sound is unclear, the model still writes something. It picks the most likely words, not the true words. This is why a transcript can contain a clean, confident sentence that nobody said, especially during long silences or loud noise.

Treat a transcript like a helpful draft. Check names, numbers, and anything important before you rely on it.

Key Takeaways

• A microphone turns your voice into a long list of numbers that track the sound wave.
• A spectrogram turns that wave into a picture of pitch and loudness over time, which a model can learn to read.
• Modern speech recognition is mostly end-to-end: one neural network goes from spectrogram to text.
• Models like Whisper use an encoder to read the sound and a decoder that writes text one piece at a time, using context.
• Varied training audio helps with accents and noise, but rare speech still causes more errors.
• A transcript is the model's best guess, so always check names, numbers, and key details.