Handling Missing Values in Pandas

Concept

Missing values are places where a field has no usable value. In Pandas, they commonly appear as NaN for numerical data, but a dataset may also use empty strings, placeholders such as "unknown", or special codes that must be cleaned before Pandas can recognize them as missing.

Why It Matters

Missing data is not just a cleaning inconvenience. It can change distributions, bias analysis, reduce model performance, and sometimes reveal how a business process actually works.

The first question should not be “How do I remove missing values?” It should be “Why are these values missing?”

Detect missing values

import pandas as pd

missing_by_column = df.isna().sum()
print(missing_by_column)

For proportions:

missing_pct = df.isna().mean().mul(100).round(2)

Use notna() when you need rows with a present value:

customers_with_income = df[df["annual_income"].notna()]

A column with 2% missing values deserves a different response from one with 70% missing values.

Real-world example

Imagine a loan dataset where annual_income is missing for some applicants.

Possible reasons include:

• users skipped the field;
• a source system failed to send it;
• income is unavailable for a specific customer type;
• the value was removed by an earlier processing rule.

Those causes have different implications. Imputing every missing income with the mean may hide useful information.

Common handling strategies

Drop rows

clean_df = df.dropna(subset=["annual_income"])

Use this only when losing those rows is acceptable and unlikely to introduce bias.

Median imputation

median_income = df["annual_income"].median()
df["annual_income"] = df["annual_income"].fillna(median_income)

Median is often safer than mean for skewed numerical variables.

Mean, mode, and constant values

df["age"] = df["age"].fillna(df["age"].mean())
df["product_category"] = df["product_category"].fillna(
    df["product_category"].mode().iloc[0]
)
df["campaign_code"] = df["campaign_code"].fillna("not_provided")

Mean can be reasonable for a roughly symmetric numeric feature; median is often more robust to outliers; mode is useful for a categorical field; a constant can preserve the fact that a value was absent. None is automatically correct. A missing income may mean “customer declined to answer,” while a missing transaction amount may indicate a broken source record.

Add a missingness indicator

df["income_was_missing"] = df["annual_income"].isna().astype(int)

Sometimes the fact that a value is missing is predictive.

Leakage warning

For machine learning, fit preprocessing on the training data and apply the learned transformation to validation/test data. Do not calculate a global median using information from the test set.

Code Explanation

isna() and notna() return Boolean tables or Series. dropna(subset=[...]) limits row removal to fields that are essential for the current task. fillna() replaces missing values, but it does not make the original data problem disappear. Always compare distributions and row counts before and after a strategy.

When to Use It

Investigate missingness whenever loading, merging, or preparing a dataset. Use a context-aware strategy before exploratory summaries or ML preprocessing, and keep a record of which values were changed.

Best Practices

Quantify missingness by column and by important segments. Check whether missingness is related to an outcome or process. Preserve a missingness indicator when absence may carry information, and separate training-only preprocessing from validation and test data to avoid leakage.

Data Science Perspective

Missing data can change distributions, bias group comparisons, and distort model inputs. The reason for missingness matters: data absent at random may support one strategy, while systematic absence may require a separate category, indicator, or process investigation.

A Context-Aware Decision Example

Suppose customer_income is missing more often for a particular acquisition channel. Dropping those rows may make that channel look artificially different; filling every value with the overall median may hide a meaningful process difference. A reasonable first response is to report missingness by channel, retain an income_was_missing indicator when appropriate, and choose an imputation strategy using training data only. Missingness can itself carry information, but it is not automatically a useful feature and should be evaluated carefully.

For categorical values, a visible constant such as "not_provided" can be more honest than guessing the mode when absence has business meaning. For essential transaction fields, an unresolved missing value may require source-system investigation rather than imputation.

Interview perspective

Question: How would you handle missing values?

A strong answer starts with investigation: quantify missingness, understand why it occurs, assess whether it is random, consider business meaning, and then select a strategy appropriate for the feature and modeling setup.

Practice Questions

A dataset has 100,000 rows. age is 3% missing, income is 18% missing, and secondary_phone is 82% missing. Propose a different handling strategy for each column and justify it.

2. Write Pandas code that counts missing values by column and sorts the result from largest to smallest.
3. Explain why filling an income column with a global median before a train/test split can be a problem.
4. Give one reason a missingness indicator can be useful even after filling a numeric value.

Quick Quiz

1. Which method produces True for missing values? Answer: isna().
2. Is dropping all missing rows always a safe default? Answer: no.
3. Which fill strategy often resists extreme numeric values better: mean or median? Answer: median.
4. What should you investigate before dropping missing rows? Answer: how many rows are lost and whether missingness is related to a group or outcome.

Next Lesson

Next, summarize meaningful groups of rows with GroupBy and aggregation.