Handling Duplicates

Finding and removing duplicate rows is an important data cleaning step.

Finding Duplicates

Use duplicated() to identify duplicate rows:

Keeping First vs Last

Control which duplicate is marked:

Checking Specific Columns

Check duplicates in subset of columns:

Dropping Duplicates

Remove duplicate rows with drop_duplicates():

Dropping by Specific Columns

Keep unique combinations of selected columns:

Getting Duplicate Rows

Extract the actual duplicates:

Exercise: Remove Duplicates

Exercise: Unique by Column

Key Points

• duplicated() returns boolean Series
• keep='first' marks later occurrences (default)
• keep='last' marks earlier occurrences
• keep=False marks all duplicates
• subset= checks specific columns only
• drop_duplicates() removes duplicate rows