Why AI Runs on GPUs and AI Chips
GPUs were built to draw video game graphics. Today they are the most important hardware in AI, and the companies that make them sit at the center of the technology industry. The reason is not luck. The math behind video game graphics and the math behind neural networks turned out to be very similar.
This lesson explains what makes GPUs different from normal processors, why that difference suits AI so well, and what the newer specialized AI chips in data centers and phones are doing.
What You'll Learn
- The difference between doing work in sequence and in parallel
- What kind of math AI models mostly do
- Why GPUs fit that math so well
- What specialized AI chips like TPUs and NPUs are
- Why the chip is only part of the story
Few fast workers vs many simple workers
A CPU is built to be flexible. It has a small number of very capable cores, each able to handle complicated, branching tasks quickly: running your operating system, opening apps, responding to clicks. It is great at doing many different things, mostly one after another.
A GPU takes the opposite approach. It has thousands of simpler cores. Each one is less capable on its own, but together they can do a huge number of simple calculations at the same time. This is called parallel processing.
GPUs trade flexibility for massive parallel throughput.
| Criteria | CPU | GPU |
|---|---|---|
| Number of cores | A few, very capable | Thousands, simpler |
| Best at | Varied tasks, one after another | The same simple task on lots of data at once |
| Picture it as | A few expert chefs | A huge team each chopping one vegetable |
| Main AI role | Coordinating the work | Doing the heavy math |
CPU
- Number of cores
- A few, very capable
- Best at
- Varied tasks, one after another
- Picture it as
- A few expert chefs
- Main AI role
- Coordinating the work
GPU
- Number of cores
- Thousands, simpler
- Best at
- The same simple task on lots of data at once
- Picture it as
- A huge team each chopping one vegetable
- Main AI role
- Doing the heavy math
The math AI actually does
Strip away the details, and most of the work inside a neural network is one kind of operation: multiply numbers together, then add up the results. Do that for many numbers at once and you have what mathematicians call matrix multiplication. You do not need to know the math, only its shape:
- There are enormous amounts of these multiply-and-add steps.
- Most of them do not depend on each other, so they can be done at the same time.
- Each one is simple. No complicated decisions are needed.
That is exactly the kind of work a GPU is built for. Drawing a video game frame means calculating the color of millions of pixels, each a simple, independent calculation. Running a neural network means millions of simple, independent multiply-and-add steps. Same shape of problem.
- Neural networksMostly multiply-and-add on huge sets of numbers
- Work is independentMany steps can run at the same time
- GPUsThousands of cores built for parallel simple math
When researchers started training neural networks on GPUs, training that would have taken months on CPUs could be done in days. That speedup is one of the main reasons deep learning took off.
Specialized AI chips
Once it was clear how much AI work there was, companies started designing chips only for it. These drop features AI does not need and add more of what it does.
- TPUs (tensor processing units) and similar data center chips are designed around the multiply-and-add operations of neural networks. Several large tech companies build their own.
- NPUs (neural processing units) are small AI chips built into phones and laptops. They run AI features, like photo enhancement, voice typing, and on-device assistants, using far less battery than the main processor would.
The general term for all of these, including GPUs used for AI, is accelerators: chips that speed up a specific kind of work.
All are accelerators. They differ in where they live and how specialized they are.
| Criteria | GPU | Data center AI chip | NPU |
|---|---|---|---|
| Where | Gaming PCs, workstations, data centers | Data centers | Phones and laptops |
| Built for | Graphics and AI | AI only | Everyday AI features |
| Strength | Flexible and widely supported | Efficient at large scale | Low power use |
GPU
- Where
- Gaming PCs, workstations, data centers
- Built for
- Graphics and AI
- Strength
- Flexible and widely supported
Data center AI chip
- Where
- Data centers
- Built for
- AI only
- Strength
- Efficient at large scale
NPU
- Where
- Phones and laptops
- Built for
- Everyday AI features
- Strength
- Low power use
The chip is only part of the story
A fast chip is useless if it sits idle waiting for data. Remember the kitchen from the previous lesson: a huge team of cooks can only work as fast as ingredients reach them.
That is why AI hardware is judged not just on how many calculations per second it can do, but on:
- How much fast memory sits right next to the chip.
- How quickly that memory can feed the chip.
- How well many chips can share work with each other.
The next lesson covers the first two, which are often the real limit on AI. The last lesson covers the third.
Key Takeaways
- CPUs have a few capable cores for varied tasks. GPUs have thousands of simple cores for parallel work.
- Neural networks mostly do multiply-and-add on huge sets of numbers, and most of it can run in parallel.
- That makes GPUs a natural fit, and moving AI training to GPUs was a key reason deep learning took off.
- Specialized AI chips, like data center AI chips and NPUs in phones, are designed only for this kind of math.
- Raw calculation speed is not enough. Memory and data movement often decide real AI performance.