Finding Patterns, Relationships, and Data Quality Issues

Concept

Strong EDA separates a visible observation from an explanation. It looks for trends, clusters, skew, segments, and relationships while also checking whether data-quality problems could be creating the pattern.

Patterns and Relationships

In churn data, month-to-month customers may show higher observed churn than annual-contract customers. A histogram may show right-skewed support_calls; a scatter plot may reveal that high charges and short tenure occur together; a time chart may show seasonal changes. These are observations. They become hypotheses only after asking what else differs between the groups.

An interaction means a relationship changes by another variable. For example, higher monthly charges may be associated with churn for newer customers but not long-tenure customers. Segment comparisons can reveal this better than one overall average.

Data Quality Checks

Inspect missingness, duplicate IDs, impossible values, inconsistent labels, extreme values, dates, and unexpected ranges. age = 250, negative order quantity, or a signup date after a churn date require investigation. Categories such as New York, NY, and new york can split one real group into misleading labels.

df.isna().sum()
df.duplicated("customer_id").sum()
df["region"].value_counts(dropna=False)
df.loc[df["age"].between(0, 110) == False, ["customer_id", "age"]]

An outlier is not automatically an error. A customer with many support calls may be a valid high-risk case; a huge charge may be a currency error. Check the source, unit, and business context before cleaning it.

Leakage Awareness

EDA should ask when each field became available. A cancellation date, final support outcome, or refund created after a customer churned may describe the outcome but must not be used to predict it beforehand. This is leakage: future or target-derived information makes a model look better than it can perform in reality.

From Observation to Action

Use four levels of language. Observation: month-to-month customers have higher observed churn. Interpretation: contract type may be related to retention. Hypothesis: pricing or service differences could help explain it. Conclusion: only after appropriate validation supports a specific claim.

Possible next steps are to inspect pricing and service differences, include contract type as a feature, and test whether the relationship remains after accounting for tenure and charges. This is more useful than declaring contract type the cause.

Failure Signals

Common Mistakes

Deleting every outlier or missing row.
Ignoring inconsistent category labels and date logic.
Using future information during exploration for a predictive dataset.
Treating tiny segments as stable evidence.
Confusing an observed relationship with a causal conclusion.

Best Practices

Validate surprising values against the source, keep a log of quality decisions, compare groups with counts, and revisit the business definition of each field. Build charts around a question and record both evidence and uncertainty.

Data Science Perspective

EDA findings guide cleaning rules, feature choices, cohort definitions, and modeling safeguards. The outcome is a clean analytical dataset plus a defensible set of questions, not a collection of attractive charts.

Interview Perspective

Question: How do you identify data-quality issues? Answer: inspect types, missingness patterns, duplicate keys, ranges, categories, dates, and source definitions; validate anomalies before changing data. Follow-up: give an example of leakage in churn prediction.

Practice Questions

Why might New York, NY, and new york change a regional churn report?
What would you check before removing a customer with 30 support calls?
Explain why a post-churn cancellation field is leakage for churn prediction.

Quick Quiz

Is an outlier always an error? Answer: no.
What comes after an observation but before a conclusion? Answer: interpretation and hypothesis.
What is leakage? Answer: information unavailable at the prediction time that reveals the outcome.

Error or Legitimate Extreme?

An age of 250 and monthly_charges = -40 are likely validation or unit problems. A customer with unusually high charges or thirty support calls may be a legitimate extreme observation. Check source records, units, dates, and business rules before changing either. Duplicate customer IDs require the same care: they may be an accidental repeat, a monthly snapshot, or valid event history. Date inconsistencies such as churn before signup, and inconsistent labels such as New York, NY, and new york, can create false segments.

Leakage and Evidence

A cancellation outcome, final refund amount, or support resolution recorded after churn can appear highly predictive only because it already knows the future. Including it in a churn model produces unrealistically strong validation results that will fail at the real decision time. Keep an evidence ladder: observation, interpretation, hypothesis, conclusion, and next action. Higher churn among month-to-month customers is an observation; contract flexibility may be associated with churn is an interpretation; longer contracts improve retention is a hypothesis, not a conclusion. Next inspect pricing, tenure, and service differences, then validate whether the relationship remains.

Key Takeaway

Key Takeaways

Useful EDA finds patterns, validates quality, and turns observations into testable next steps. It reports uncertainty and avoids causal claims that the data cannot support.

Next Lesson

You have completed the initial Data Science Foundations EDA module. Revisit earlier modules as you apply this workflow to real datasets.