Data Validation: Keeping Data Trustworthy
A pipeline can move data perfectly and still produce a useless result. If the data flowing through is wrong, everything downstream is wrong too. That is why real pipelines check their data as it moves. This practice is called data validation. This lesson explains what validation is, the common checks, and why catching problems early saves so much pain. No code required.
What You'll Learn
- What data validation is and why pipelines need it
- The most common validation checks
- The difference between stopping and quarantining bad data
- Where validation fits in a pipeline
What Data Validation Is
Data validation is checking that data meets the rules you expect before you trust it. It is the pipeline's quality control.
Think of a factory line with an inspector. As products move past, the inspector checks each one. Is it the right size? Is anything missing? Is it damaged? Products that fail get pulled off the line before they reach a customer. Validation does the same for data. It inspects records against rules and catches problems before they spread.
Without validation, bad data flows straight into reports and dashboards. People make decisions on numbers that are quietly wrong, and no one notices until much later.
The "Garbage In, Garbage Out" Rule
There is an old saying in data work: garbage in, garbage out. If you feed a system bad data, you get bad results, no matter how good the rest of the pipeline is.
A clean, well-built pipeline cannot fix data that was wrong at the source. It can only catch the problem and stop it from spreading. That is exactly what validation does, and it is why teams treat it as essential rather than optional.
Common Validation Checks
Most validation is built from a small set of simple checks. You do not need to write them. You need to recognize them.
- Completeness. Are required fields present? For example, every order must have a customer ID. Rows missing one fail.
- Type. Is each value the right kind of data? A price should be a number, not the word "unknown."
- Range. Does a value fall inside sensible limits? An age of 250 or a negative quantity signals a problem.
- Format. Does the value match an expected pattern? An email should look like an email. A date should be a real date.
- Uniqueness. Are values that should be unique actually unique? Two orders should not share the same order ID.
- Consistency. Do related values agree? An order's total should match the sum of its line items.
Three of the most common validation checks and what they catch
| Criteria | Completeness | Range | Uniqueness |
|---|---|---|---|
| Question it asks | Is the field there? | Is the value sensible? | Is it a duplicate? |
| Example failure | Missing customer ID | Age of 250 | Two rows, same order ID |
| Common cause | Broken source export | Typo or bad sensor | Data loaded twice |
Completeness
- Question it asks
- Is the field there?
- Example failure
- Missing customer ID
- Common cause
- Broken source export
Range
- Question it asks
- Is the value sensible?
- Example failure
- Age of 250
- Common cause
- Typo or bad sensor
Uniqueness
- Question it asks
- Is it a duplicate?
- Example failure
- Two rows, same order ID
- Common cause
- Data loaded twice
Stop or Quarantine
When a record fails a check, the pipeline has to decide what to do. There are two common choices.
- Stop the pipeline. If the problem is serious or widespread, halt the run and alert a person. This prevents bad data from ever landing. It fits situations where wrong data is worse than late data, such as finance.
- Quarantine the bad rows. Set the failing records aside in a separate area, let the good rows continue, and review the failures later. This keeps most of the data flowing while still catching problems.
Neither choice is always right. The point is that a good pipeline has a plan for failure instead of silently passing bad data along.
Where Validation Fits
Validation is most useful early, right after data is collected, so problems are caught before they spread. Many pipelines also check again after transforms, to make sure the cleaning steps did not introduce new errors.
- CollectPull raw data
- ValidateCheck the rules
- TransformClean and reshape
- Validate againConfirm nothing broke
- StoreTrusted data lands
Checking early and checking again is a simple habit that catches both source problems and transform mistakes. The earlier you catch a problem, the cheaper it is to fix.
A Quick Example
Picture a nightly pipeline for a shop.
- It collects yesterday's orders.
- Validation runs. Three rows are missing a customer ID, and one row has a negative quantity.
- The pipeline quarantines those four rows and lets the rest continue.
- A short report notes the four failures so someone can look into the source system.
- The good data is transformed, validated again, and stored.
The dashboard the next morning shows trustworthy numbers, and the team has a clear signal about the four bad rows instead of a silent error hiding in the totals.
Key Takeaways
- Data validation checks that data meets your rules before you trust it.
- Bad data in means bad results out, so validation protects everything downstream.
- Common checks cover completeness, type, range, format, uniqueness, and consistency.
- When a record fails, a pipeline can stop entirely or quarantine the bad rows and continue.
- Validate early and validate again after transforms to catch problems while they are cheap to fix.