Inside AI Data Centers: Chips, Networks, Power, and Cooling
The biggest AI models are too large for any single chip, and training them takes more computing than one machine could do in a lifetime. So they run in data centers: buildings full of servers, each packed with AI chips, all connected together and working as one giant computer.
This lesson explains how that works at a conceptual level: how a model is split across many chips, why the network between chips matters so much, and why power and cooling have become central to the story of AI.
What You'll Learn
- Why large models need many chips working together
- The main ways work is split across chips
- Why fast connections between chips are as important as the chips
- How training clusters differ from clusters that serve users
- Why power, cooling, and water have become big AI topics
Why one chip is not enough
The previous lessons gave two reasons:
- Memory. A large model's weights can need far more memory than one chip has. The model has to be divided up.
- Computing. Training processes enormous amounts of data. Even the fastest chip would take far too long alone.
So data centers group chips into servers (often several AI chips per server), servers into racks, and racks into clusters that can contain a very large number of chips.
- Cluster: many racks working as one system
- Rack: a tall cabinet of servers
- Server: a machine holding several AI chips
- AI chip: GPU or specialized accelerator with its own fast memory
- Server: a machine holding several AI chips
- Rack: a tall cabinet of servers
Splitting the work
There are three main ways to spread AI work across many chips. Real systems usually combine them.
Most large systems combine all three.
| Criteria | What is split | How it works | Main benefit |
|---|---|---|---|
| Split the data | The training examples | Each chip has a full copy of the model and works on different examples, then they share what they learned | Train on more data at once |
| Split each layer | The math inside each layer | Chips each hold part of every layer and combine results constantly | Fit models too big for one chip |
| Split the layers | The model's layers | Chips each hold a group of layers and pass results along like an assembly line | Fit very deep models |
What is split
- Split the data
- The training examples
- Split each layer
- The math inside each layer
- Split the layers
- The model's layers
How it works
- Split the data
- Each chip has a full copy of the model and works on different examples, then they share what they learned
- Split each layer
- Chips each hold part of every layer and combine results constantly
- Split the layers
- Chips each hold a group of layers and pass results along like an assembly line
Main benefit
- Split the data
- Train on more data at once
- Split each layer
- Fit models too big for one chip
- Split the layers
- Fit very deep models
All three have one thing in common: chips must constantly exchange data. That leads to the next point.
The network matters as much as the chips
When a model is spread across many chips, each step of the work often needs results from other chips before it can continue. If the connections between chips are slow, expensive chips sit idle waiting.
That is why AI data centers use:
- Very fast direct links between chips inside the same server.
- High-speed networks connecting servers and racks, much faster than normal office networking.
- Careful layouts that keep chips that talk most often physically close together.
This is the same lesson from the start of the course, at a larger scale: moving data is often the bottleneck. Inside a chip it is memory. Across a data center it is the network.
Training clusters vs serving clusters
The two jobs from lesson 1 look different at data center scale.
Training uses a huge cluster working together on one task for a long stretch of time. Every chip must stay in sync. If one fails, the job has to recover from a saved checkpoint. It is one enormous, tightly coordinated project.
Serving (inference) answers millions of separate requests. The work is split into many independent pieces: copies of the model spread across groups of chips, each handling a stream of users. It needs to be fast and reliable for each request, and it runs all the time.
Same kinds of chips, very different ways of using them.
| Criteria | Training | Serving |
|---|---|---|
| Goal | Create the model | Answer users |
| Shape of work | One giant job | Many small independent jobs |
| Timing | Weeks or months, then done | Constant, every day |
| Key need | Chips perfectly in sync | Fast, reliable answers at scale |
Training
- Goal
- Create the model
- Shape of work
- One giant job
- Timing
- Weeks or months, then done
- Key need
- Chips perfectly in sync
Serving
- Goal
- Answer users
- Shape of work
- Many small independent jobs
- Timing
- Constant, every day
- Key need
- Fast, reliable answers at scale
Power and cooling
Every calculation uses electricity, and nearly all of that electricity ends up as heat. A room full of AI chips running constantly produces a great deal of both demand and heat. This has made power and cooling central to how AI grows.
Power. Large AI data centers need a very large, steady electricity supply. Where to find that power, and how clean it is, now shapes where data centers are built and how fast they can expand.
Cooling. Chips stop working properly, and wear out faster, if they get too hot. Data centers use:
- Air cooling: large fans and chilled air moving through the racks.
- Liquid cooling: liquid flowing through plates attached directly to the chips, which removes heat much more efficiently. The densest AI hardware increasingly relies on it.
Water. Some cooling systems use water, often by letting it evaporate to carry heat away. That can put pressure on local water supplies, especially in dry regions. Designs that reuse water or rely on other methods reduce this.
Efficiency. Engineers work constantly to get more AI work from each unit of power: more efficient chips, smaller and quantized models, better cooling, and batching many requests together. The environmental impact of AI depends heavily on these choices and on the energy sources used. AI Ethics & Responsible AI covers the wider debate.
Putting it together
Every time you ask an AI a question, a request travels to a data center, is routed to a group of accelerators holding the model's weights in fast memory, runs through billions of multiply-and-add steps one token at a time, shares data across chips over fast links, and turns electricity into both an answer and heat that has to be removed.
That chain explains most of the practical limits you notice as a user: why bigger models are slower, why long contexts are harder, why providers set usage limits, and why capable small models on your own devices are such an active area of work.
Key Takeaways
- Large models need many chips because of both memory and computing limits.
- Work is split by data, within layers, and across layers, and real systems combine all three.
- Chips must constantly share data, so fast networks between chips matter as much as the chips.
- Training is one giant synchronized job. Serving is many independent requests running constantly.
- Power, cooling, and water have become central to AI's growth, and efficiency choices shape its impact.