Selecting and Filtering Data

Concept

Selection chooses columns or rows from a DataFrame. Filtering keeps observations that meet a condition. Pandas makes this expressive with column labels, .loc, .iloc, and Boolean masks.

Why It Matters

Most analysis begins with a question about a subset: customers older than 30, orders above a threshold, or active subscribers in selected regions. Clear selection keeps the question and the code aligned.

Column Selection

import pandas as pd

customers = pd.DataFrame({
    "name": ["Asha", "Ben", "Chen"],
    "age": [28, 41, 35],
    "region": ["North", "South", "North"],
    "active": [True, False, True],
    "revenue": [6200, 4100, 7800],
})

print(customers["revenue"])
print(customers[["name", "revenue"]])

One pair of brackets returns a Series; a list inside brackets returns a DataFrame with several columns.

loc and iloc

.loc selects by labels; .iloc selects by integer position.

print(customers.loc[0, "name"])
print(customers.loc[0:1, ["name", "region"]])
print(customers.iloc[0:2, 0:2])

With .loc, label slices include the end label. With .iloc, position slices follow normal Python behavior and exclude the end. Prefer .loc when your code should communicate column names; use .iloc for genuinely positional tasks.

Filtering with Boolean Masks

over_30 = customers[customers["age"] > 30]
selected_regions = customers[customers["region"].isin(["North", "West"])]
print(over_30)
print(selected_regions)

The expression customers["age"] > 30 creates one True or False value per row. Passing that mask inside brackets keeps only the True rows. isin() is useful for membership in a known set of categories.

Multiple Conditions

high_value_active = customers[
    (customers["revenue"] > 5000) & (customers["active"])
]
print(high_value_active)

Expected output keeps Asha and Chen. Use & for “and” and | for “or,” with parentheses around every condition. Python's and and or expect one Boolean value, while a Pandas Series contains many row-level Booleans.

between() can make ranges readable: customers[customers["age"].between(30, 45)] includes both endpoints by default.

Real-World / Data Science Example

A retention team may need active customers with revenue above 5,000 before offering a premium intervention. Filtering gives a candidate group, but it does not prove those customers will churn. The filter is a business rule that should be reviewed alongside its cost and fairness implications.

Code Explanation

Boolean filtering aligns conditions to rows through the DataFrame index. Combining masks with parentheses keeps operator precedence clear. .loc can combine row masks and column lists in one operation, for example customers.loc[mask, ["name", "revenue"]].

Choosing loc or iloc Deliberately

Use .loc when the task is stated in labels: “show revenue and region for customers matching this condition.” Use .iloc when position itself is meaningful, such as reviewing the first ten imported rows. They can look similar but answer different questions. If a DataFrame has an ID-based index, df.loc[101] means the row labeled 101, not necessarily the 102nd row.

Build masks separately when a filter has several rules. This makes each business condition inspectable.

is_active = customers["active"]
is_high_value = customers["revenue"] > 5000
target_customers = customers.loc[is_active & is_high_value, ["name", "region", "revenue"]]

Checking is_active.sum() and target_customers.shape is a quick way to notice an unexpectedly strict condition before sending a list to a business team.

When to Use It

Use selection to focus an analysis, build a feature subset, inspect exceptions, or prepare a report. Filter early only when it matches the question; dropping rows casually can hide data-quality issues or change the population you intend to describe.

Failure Signals

Common Mistakes

Using and or or instead of & or |.
Forgetting parentheses around individual conditions.
Confusing .loc labels with .iloc positions.
Filtering a Series and then assuming it has the same rows as the original table.

Best Practices

Name complex masks, such as high_value_active_mask, and inspect the resulting row count. Select only columns needed for a report, but retain an ID when you need to trace a finding back to the source record.

Data Science Perspective

Boolean masks are used for cohort analysis, data validation, outlier review, and feature preparation. They are the table-oriented form of the comparisons you used with NumPy arrays.

Interview Perspective

Question: Why do Pandas filters use & rather than and? A strong answer: each comparison creates a Series of row-level Booleans, and & combines those element by element.

Practice Questions

Select only region and revenue from a customer table.
Filter orders with revenue greater than 1,000 and status equal to "paid".
Explain why a filter for selected regions should use isin() rather than repeated equality checks.
Rewrite an invalid age > 30 and active filter using Pandas masks and parentheses.

Quick Quiz

Which accessor selects by labels? Answer: .loc.
Which operator combines Pandas conditions as “and”? Answer: &.
Do .iloc slices include their end position? Answer: no.
Why are parentheses used around Pandas conditions? Answer: to make each row-level comparison explicit before & or | combines them.

Key Takeaway

Key Takeaways

Pandas selection makes business questions concrete. Use named columns, Boolean masks, parentheses, and explicit row counts to keep filters accurate and reviewable.

Next Lesson

Next, investigate and handle missing values without losing their business meaning.