Shifting, Lagging & Period-over-Period Change

Many time series questions are about change: how much did sales grow versus last month, what was yesterday's value, what is the day-over-day difference. Pandas answers these with shift, diff, and pct_change. These tools let you compare each row to an earlier (or later) row without manual bookkeeping.

What You'll Learn

• How shift moves data up or down to create lagged columns
• How diff computes the change between consecutive rows
• How pct_change gives period-over-period growth
• How to build a previous-value comparison column

shift: Lagging a Column

shift(n) moves values down by n rows, leaving NaN at the top. The shifted column represents the value from n periods earlier, which is exactly what you need to compare today against yesterday.

A negative shift (shift(-1)) moves values up, giving you the next period's value instead, useful for building a "what comes after" comparison.

diff: Change Between Rows

diff() subtracts the previous row from the current one. It is shorthand for df['sales'] - df['sales'].shift(1) and answers "how much did this change."

The first change is NaN because there is no prior row to subtract.

pct_change: Growth Rate

pct_change() gives the fractional change from one row to the next. Multiply by 100 for a percentage. This is the standard period-over-period growth metric.

From 100 to 120 is a 20 percent increase, and from 120 to 115 is a small decline, all computed automatically.

Comparing Across a Longer Gap

shift, diff, and pct_change all accept a period argument, so you can compare to a week ago or a month ago instead of just the prior row.

Watch Out for Grouped Data

When a DataFrame holds several series stacked together (for example multiple stores), shift across the whole column would mix one group's last row into the next group's first row. Shift within each group using GroupBy so values do not leak across boundaries.

The first row of store B is NaN, not store A's last value, because the shift respects group boundaries.

Exercise: Day-over-Day Change

Exercise: Previous Value

Key Points

• shift(n) lags a column by n rows so you can compare to an earlier period
• diff() returns the change between consecutive rows
• pct_change() gives period-over-period growth as a fraction
• All three accept a period argument to compare across longer gaps
• For stacked groups, shift within each group using GroupBy to avoid leaking values across boundaries