Handling Missing and Duplicate Data

Missing values and duplicates are two of the trickiest data cleaning problems. Unlike typos, there's rarely a single "right" answer—you need to make judgment calls based on your data and goals. AI can help you understand your options and make informed decisions.

Finding Missing Values

Start by understanding the scope of missing data:

Strategies for Handling Missing Data

There are several approaches, and the right one depends on your situation:

Option 1: Remove Rows with Missing Data

Best when: Only a few rows are affected and removing them won't bias your results.

Option 2: Fill with a Default Value

Best when: You have a sensible default (e.g., "Unknown" for a category, 0 for a count).

Option 3: Fill with Calculated Values

Best when: You can reasonably estimate the missing value from other data (e.g., the average or median).

Option 4: Flag and Keep

Best when: You want to keep all data but mark which values were missing for transparency.

Ask AI to recommend the best approach for your specific situation:

Implementing Missing Value Fixes

Once you've chosen a strategy, ask AI to apply it:

Finding Duplicates

Duplicates come in two types, and you need to handle them differently:

Exact Duplicates

Rows where every column is identical. These are usually safe to remove.

Near Duplicates

Rows that are almost identical but have small differences—like two entries for the same customer with slightly different spellings. These require more judgment.

Deciding Which Duplicate to Keep

When you find near-duplicates, you need to decide which version is correct:

Removing Duplicates Safely

Before deleting duplicates, create a backup strategy:

Preventing Future Duplicates

After cleaning, ask AI for advice on preventing the problem from recurring:

Key Takeaway

Missing values and duplicates require judgment—there's no one-size-fits-all fix. Use AI to understand the scope of the problem, evaluate your options, and implement the best strategy for your specific situation. Always keep a record of what you changed so you can explain your decisions.