What a Data Pipeline Is

Every app, dashboard, and report you use runs on data that came from somewhere else. A sales chart pulls numbers from an orders system. A weather app pulls readings from sensors. That data almost never arrives ready to use. Something has to collect it, clean it up, and move it to where it is needed. That "something" is a data pipeline.

This lesson gives you a clear mental model of what a pipeline is and why it exists. You do not need to code. You just need to picture how data travels from one place to another.

What You'll Learn

• What a data pipeline is, in plain terms
• The stages data passes through on its way to being useful
• Why raw data is rarely ready to use as-is
• The difference between batch and streaming pipelines

A Simple Definition

A data pipeline is a set of steps that moves data from a source to a destination, changing it along the way so it is ready to use.

Think of a real water pipeline. Water starts at a reservoir, flows through pipes, gets filtered, and comes out clean at your tap. A data pipeline does the same job for information. Data starts in a source system, flows through processing steps, gets cleaned and reshaped, and arrives somewhere it can be read, searched, or charted.

The key idea is flow. Data does not stay in one place. It moves on a repeatable path, usually on a schedule, so the destination always has fresh, usable information.

The Stages Data Passes Through

Most pipelines follow the same basic path, no matter how big or small they are.

Here is what happens at each stage.

• Source. Where the data is born. This could be an online store, a spreadsheet, a payment system, or a device sending readings.
• Collect. The pipeline reaches into the source and pulls a copy of the data. This is often called extraction.
• Transform. The raw data gets cleaned and reshaped. Dates get a standard format. Duplicates get removed. Columns get renamed. Values get combined.
• Store. The cleaned data lands in a place built for it, such as a data warehouse or a data lake. You will meet both later in this course.
• Use. People and tools read the stored data to build reports, power dashboards, or feed AI models.

Why Raw Data Is Not Ready to Use

It is tempting to think you could skip the middle and read straight from the source. In practice, raw data is messy.

• The same customer might be spelled three different ways.
• Dates might be written as "01/02/2026" in one system and "Feb 1, 2026" in another.
• Numbers might arrive as text, so you cannot add them up.
• Some rows might be missing values or contain obvious errors.

A pipeline fixes these problems in the transform stage so that everyone downstream works from clean, consistent data. Without that step, every report would carry the same mess, and no two people would agree on the numbers.

Batch vs Streaming

Pipelines run on two main rhythms. Knowing the difference helps you understand why a report is fresh or a few hours old.

• Batch pipelines move data in scheduled groups. For example, every night at 2 a.m. the pipeline collects the day's orders and loads them. Batch is simple, cheap, and fine for most reporting. The tradeoff is that data is only as fresh as the last run.
• Streaming pipelines move data continuously, event by event, as it happens. A fraud alert or a live map needs streaming because a few hours of delay would be useless. Streaming is more powerful and more complex to run.

Most organizations use batch for the majority of their work and reserve streaming for the cases that truly need up-to-the-second data.

A Real Example

Imagine a small online shop.

1. Orders are created in the store's checkout system all day. That is the source.
2. Each night, a pipeline collects the new orders.
3. It transforms them: it standardizes country names, converts prices to one currency, and drops test orders.
4. It stores the clean orders in a warehouse.
5. The next morning, the owner opens a dashboard that reads from the warehouse and shows yesterday's sales. That is the use stage.

The owner never touches the raw checkout data. The pipeline does the heavy lifting so the dashboard is always clean and consistent.

Key Takeaways

• A data pipeline moves data from a source to a destination and reshapes it along the way so it is ready to use.
• The common stages are source, collect, transform, store, and use.
• Raw data is almost always messy, so the transform stage is what makes data trustworthy.
• Batch pipelines run on a schedule, while streaming pipelines move data continuously as events happen.
• Understanding this flow is the foundation for everything else in this course.