Where Data Lives: Cloud Storage and S3
A pipeline is always moving data toward a place where it can rest and be read. For most modern pipelines, that place is cloud storage. And when people talk about cloud storage for data, they very often mean Amazon S3. This lesson explains what cloud object storage is, what S3 does, and how it fits into a pipeline. No account or setup required. This is about the concept.
What You'll Learn
- What cloud object storage is and why pipelines rely on it
- What Amazon S3 is in plain terms
- The difference between a data lake and a data warehouse
- How storage sits inside a pipeline
What Cloud Storage Really Means
Cloud storage is space to keep files on servers you rent instead of computers you own. You upload data over the internet, and the provider handles the hard parts: keeping copies safe, scaling up as you add more, and serving the data back when you ask.
The style of storage that pipelines lean on is called object storage. Instead of a strict folder tree like your laptop, object storage keeps each file as an "object" with a name and some labels. You can throw in almost anything: spreadsheets, images, logs, or huge exports. It scales to enormous sizes without you managing any disks.
This matters for pipelines because data volumes are unpredictable. Object storage grows with you and charges roughly for what you actually store, so it is a natural landing zone for raw and processed data alike.
What Amazon S3 Is
Amazon S3 stands for Simple Storage Service. It is Amazon Web Services' object storage. It is one of the most widely used places to store data in the cloud, and countless pipelines read from and write to it.
A few plain-language basics.
- Buckets. Your data lives in containers called buckets. A bucket is like a top-level drawer with a globally unique name.
- Objects. Each file you store is an object inside a bucket. An object has a key, which is just its name or path, and the data itself.
- Durability. S3 keeps multiple copies of your data across systems, so files are very unlikely to be lost.
- Access control. You decide who and what can read or write each bucket. Good access settings keep private data private.
You do not need to know the buttons. You need to know that when a pipeline "lands data in S3," it is dropping files into a bucket so later steps can pick them up.
Data Lake vs Data Warehouse
Two words come up constantly around storage. They are not the same thing, and the difference is easy once you see it.
A lake stores raw data cheaply, a warehouse serves clean data fast
| Criteria | Data Lake | Data Warehouse |
|---|---|---|
| What it holds | Raw data in any format | Cleaned, structured tables |
| Shape of data | Loose files and objects | Rows and columns |
| Main use | Store everything cheaply | Fast reporting and queries |
| Typical example | Amazon S3 | A cloud query warehouse |
| Read style | Read files, then process | Run SQL queries directly |
Data Lake
- What it holds
- Raw data in any format
- Shape of data
- Loose files and objects
- Main use
- Store everything cheaply
- Typical example
- Amazon S3
- Read style
- Read files, then process
Data Warehouse
- What it holds
- Cleaned, structured tables
- Shape of data
- Rows and columns
- Main use
- Fast reporting and queries
- Typical example
- A cloud query warehouse
- Read style
- Run SQL queries directly
A data lake is a big, cheap pool of raw data. S3 is often used as the storage behind a data lake because it can hold anything at low cost.
A data warehouse is built for fast questions. It holds clean, structured tables so people can run reports and queries quickly.
Many modern setups use both. Raw data lands in a lake, and cleaned data is served from a warehouse. Some newer systems blur the line and query lake data directly, but the two ideas still help you reason about where data sits.
How Storage Sits in a Pipeline
Storage is not just the final destination. It often appears in the middle too.
- SourceOrders, logs, files
- Raw zone in S3Land data as-is
- TransformClean and reshape
- Clean zoneWarehouse or curated S3
- UseReports and AI
A common pattern is to keep two areas. A raw zone holds the data exactly as it arrived, untouched, so you can always go back to the original. A clean zone holds the transformed, ready-to-use version. Keeping the raw copy is cheap and saves you when something downstream goes wrong.
Why This Design Is So Common
- Cost. Object storage is inexpensive, so keeping raw data is affordable.
- Safety. If a transform has a bug, the untouched raw copy lets you rebuild without asking the source again.
- Flexibility. New needs can read from the same raw zone without new extractions.
- Scale. Storage grows as your data grows, with no disks to manage.
This is exactly why the ELT approach from the last lesson fits so well. You land raw data cheaply in storage, then transform it downstream.
Key Takeaways
- Cloud object storage keeps files on rented servers and scales without you managing disks.
- Amazon S3 is a widely used object storage service that keeps files as objects inside buckets.
- A data lake stores raw data cheaply, while a data warehouse serves clean, structured data fast.
- Pipelines often keep a raw zone and a clean zone so the original data is always recoverable.
- Cheap, durable storage is what makes modern load-first pipelines practical.