pythontojob
Log inSign up
Chapter 05

Filtering rows — asking every row the same question

Keeping only the rows that answer yes: boolean masks, combining them with &, | and ~, isin, between and the .str methods, filtered rows with .loc — and changing just those rows without the chained assignment that silently does nothing.

45 minPython 3.12
  1. 1Encounter
  2. 2Understand
  3. 3Worked
  4. 4Predict
  5. 5Apply
  6. 6Stretch

The problem we are solving

The shop's manager sends a message: "Which orders from the north branch were for more than three items? I want to see them."

With the eight orders from the last chapter you could answer by eye. Run a finger down the branch column, stop at every north, glance across at quantity. Three rows. Done.

Now make it the real file — twelve thousand orders, a year of them. Your eye no longer works, and the first idea most Python programmers have is a loop:

python
import pandas as pd

orders = pd.read_csv("orders.csv")

keep = []
for i in range(len(orders)):
    row = orders.iloc[i]
    if row["branch"] == "north" and row["quantity"] > 3:
        keep.append(int(row["order_id"]))

print(keep)
text
[1001, 1005, 1008]

The answer is right. But look at what it cost: a counter, an empty list, an if, an append, and the table taken apart one row at a time. And the result is not a table any more — it is a bare list of order numbers. To show the manager the products and prices too, you would loop again. On twelve thousand rows this is also slow, because every orders.iloc[i] builds a fresh Series just to read two values from it.

Pandas has a different way to ask, and it is the most common thing you will ever do with a DataFrame: ask every row the same yes-or-no question at once, then keep the rows that said yes. That is filtering, and it is this whole chapter. The loop above becomes one line, the result stays a table, and it reads almost like the manager's sentence.

By the end of this chapter you can

  • Turn a question into a boolean mask — a column of True/False, one per row — and use it to keep rows, or to count them with mask.sum() and mask.mean()
  • Combine conditions with &, | and ~, and explain why the parentheses are compulsory
  • Use isin, between and the .str methods for lists, ranges and text patterns
  • Filter rows and pick columns in one step with .loc[mask, columns]
  • Change values in the filtered rows safely with .loc[mask, column] = value, and recognise the ChainedAssignmentError that the unsafe way produces in pandas 3

Prerequisites: Selecting columns and rows.

Make the file first

This chapter uses the same orders.csv as the previous one. If you do not have it, create it in your project folder with exactly these nine lines:

text
order_id,date,branch,product,category,quantity,price
1001,2024-03-01,north,pen,stationery,12,15.0
1002,2024-03-01,south,notebook,stationery,5,60.0
1003,2024-03-02,north,bag,accessories,2,850.0
1004,2024-03-02,east,bottle,accessories,7,120.0
1005,2024-03-03,north,eraser,stationery,30,8.0
1006,2024-03-03,south,bag,accessories,1,850.0
1007,2024-03-04,east,pen,stationery,20,15.0
1008,2024-03-04,north,bottle,accessories,4,120.0

Before you write the code

A filter is a question written in code. Most filtering bugs are not pandas bugs — they are a question that was never pinned down, or a column that does not hold what you assumed. So before the first line of filtering, do these five things.

1. Write the question as conditions, one per column. Take the manager's sentence apart:

  • "orders from the north branch" → column branch, test equals, value "north"
  • "more than three items" → column quantity, test greater than, value 3
  • "I want to see them" → no test; keep the whole row

Two things this list forces you to decide. "More than three" is > 3, not >= 3 — an order of exactly three items does not count. And the two conditions are joined by and: an order must pass both. Those two decisions are the whole filter; the code is just their spelling.

2. Check the input — shape, types, and the exact values you will compare against.

python
import pandas as pd

orders = pd.read_csv("orders.csv")

print(orders.shape)
print(orders.dtypes)
text
(8, 7)
order_id      int64
date            str
branch          str
product         str
category        str
quantity      int64
price       float64
dtype: object

quantity is int64, so > 3 will compare numbers. If it had come in as str — because someone typed "3 pcs" in one cell — the comparison would fail with a TypeError, and you want to know that now, not halfway through. branch is str, so it will be compared as text, and text comparison is exact: "north", "North" and "north " are three different values. So look at the values that really exist:

python
print(orders["branch"].unique())
text
<StringArray>
['north', 'south', 'east']
Length: 3, dtype: str

Three branches, spelled one way each, no stray spaces. Now "north" in the code is guaranteed to match something.

3. Sketch the output before you produce it. On a small file, answer the question by hand first. The north-branch orders are 1001 (12 items), 1003 (2), 1005 (30) and 1008 (4). More than three items: 1001, 1005, 1008. So the expected result is 3 rows, all 7 columns, and their order numbers are known. If the code later returns 4 rows, or 0, you will know immediately that the code is wrong rather than trusting it. On a large file you cannot do all of this by hand, but you can still estimate: "the north branch takes about half the orders, so the answer should be well under half the rows."

4. Choose the tool from the shape of the condition.

  • one comparison → a mask: df["col"] > value
  • several conditions → masks joined with & (and), | (or), ~ (not)
  • "one of these values" → df["col"].isin([...])
  • "between two values" → df["col"].between(low, high)
  • a pattern inside text → df["col"].str.contains(...), .str.startswith(...)
  • rows and only some columns → df.loc[mask, ["col1", "col2"]]
  • change values in those rows → df.loc[mask, "col"] = new_value

The manager's question is two comparisons joined by and, so: two masks and &.

5. Decide how you will check the answer. After filtering, three cheap checks catch nearly everything: the row count matches your sketch; the condition really holds in every surviving row ((result["quantity"] > 3).all() should print True); and nothing you expected is missing (look for 1008 in the result).

Now the code — one idea at a time.

A comparison gives a mask

Compare a column with a value, and pandas does not give you one True or False. It compares every row and gives you a whole column of answers:

python
big = orders["quantity"] > 5

print(big)
text
0     True
1    False
2    False
3     True
4     True
5    False
6     True
7    False
Name: quantity, dtype: bool

This is a Series of bool — a boolean mask. It has the same index as orders (0 to 7), one answer per row: row 0 has 12 items, so True; row 1 has 5, and 5 is not greater than 5, so False.

Why does pandas work this way, instead of making you loop? Because a column is stored as one block of numbers of a single type, and comparing a whole block against 5 is one operation done in fast compiled code. Your loop asked Python to look at each row separately, eight times — or twelve thousand times. The mask asks once.

Every comparison operator works the same way: ==, !=, <, <=, >, >=. They work on text columns too — orders["branch"] == "north" is a mask with True in rows 0, 2, 4 and 7.

Nothing has been filtered yet. A mask is only the answers. The next step is to use them.

The mask keeps rows

Put the mask inside square brackets, and pandas keeps the rows where the mask is True:

python
print(orders[big])
text
order_id        date branch product     category  quantity  price
0      1001  2024-03-01  north     pen   stationery        12   15.0
3      1004  2024-03-02   east  bottle  accessories         7  120.0
4      1005  2024-03-03  north  eraser   stationery        30    8.0
6      1007  2024-03-04   east     pen   stationery        20   15.0

Four rows out of eight. Read the index down the left: 0, 3, 4, 6. The surviving rows keep their original labels. Filtering does not renumber anything — row 3 is still row 3, the same order 1004 it was in the full table. That turns out to matter, and we will come back to it.

How does pandas line the mask up with the rows? Not by position — by index label. The mask's label 0 decides row 0, label 3 decides row 3. Here the two indexes are identical, because the mask was built from orders itself, so this is invisible. It becomes visible only when you use a mask built from a different table — that is in "When it breaks".

You will usually see the mask written inline, without a name:

python
print(orders[orders["branch"] == "north"])
text
order_id        date branch product     category  quantity  price
0      1001  2024-03-01  north     pen   stationery        12   15.0
2      1003  2024-03-02  north     bag  accessories         2  850.0
4      1005  2024-03-03  north  eraser   stationery        30    8.0
7      1008  2024-03-04  north  bottle  accessories         4  120.0

Read it from the inside out: `orders["branch"] == "north"` builds the mask, `orders[ … ]` keeps the True rows. Naming the mask (big = …) is worth it when the condition is long or when you use it twice; inline is fine for short ones.

Counting with a mask, without filtering

Often the manager does not want the rows, only a number: how many orders were bigger than five items? You do not need to filter for that. In arithmetic, True counts as 1 and False as 0, so:

python
print(big.sum())
print(big.mean())
text
4
0.5

sum() adds up the ones: 4 orders. mean() divides by the number of rows: 4 out of 8 is 0.5, so half the orders were bigger than five items. "How many" is .sum() of a mask; "what share" is .mean() of a mask. Both are one line and neither builds a new table.

And when you want a total over the filtered rows — the number of items in those big orders — filter first, then pick the column, then sum:

python
print(orders[big]["quantity"].sum())
text
69

12 + 7 + 30 + 20 = 69. Reading a column of a filtered table, like this, is fine. Writing into one this way is not — that is the last section of the chapter.

Several conditions: &, |, ~

The manager's question has two conditions. Pandas joins masks with three operators:

  • & means and — the row is kept when both masks are True
  • | means or — the row is kept when at least one mask is True
  • ~ means not — it flips every True to False and back

Here is the manager's question, finally:

python
north_big = orders[(orders["branch"] == "north") & (orders["quantity"] > 3)]

print(north_big)
text
order_id        date branch product     category  quantity  price
0      1001  2024-03-01  north     pen   stationery        12   15.0
4      1005  2024-03-03  north  eraser   stationery        30    8.0
7      1008  2024-03-04  north  bottle  accessories         4  120.0

Three rows: 1001, 1005, 1008 — exactly the sketch from the planning step. Now the check from step 5:

python
print(len(north_big))
print((north_big["quantity"] > 3).all())
print((north_big["branch"] == "north").all())
text
3
True
True

Count matches, and both conditions hold in every surviving row. .all() asks "is every value in this mask True?" — it is the cheapest way to prove a filter did what you meant.

An or question: orders from the east branch, or any order of something costing 850 or more.

python
print(orders[(orders["branch"] == "east") | (orders["price"] >= 850)])
text
order_id        date branch product     category  quantity  price
2      1003  2024-03-02  north     bag  accessories         2  850.0
3      1004  2024-03-02   east  bottle  accessories         7  120.0
5      1006  2024-03-03  south     bag  accessories         1  850.0
6      1007  2024-03-04   east     pen   stationery        20   15.0

Four rows: the two east-branch orders, and the two bag orders from north and south. A row that passed both (none here) would appear once, not twice — a filter only ever keeps or drops a row.

A not question: everything except stationery.

python
print(orders[~(orders["category"] == "stationery")])
text
order_id        date branch product     category  quantity  price
2      1003  2024-03-02  north     bag  accessories         2  850.0
3      1004  2024-03-02   east  bottle  accessories         7  120.0
5      1006  2024-03-03  south     bag  accessories         1  850.0
7      1008  2024-03-04  north  bottle  accessories         4  120.0

~ turns every True into False and back. For a single equality you could write != instead, and that is simpler; ~ earns its place when the thing you are negating is a longer condition or an isin (you will see that shortly).

Why the parentheses are compulsory

Every condition above sits inside its own parentheses. That is not style. Remove them and the code breaks:

python
print(orders[orders["quantity"] > 3 & orders["price"] < 100])
text
TypeError: Cannot perform 'rand_' with a dtyped [float64] array and scalar of type [bool]

What you see above is the last line of the traceback, the line that names the error; the lines above it only say where it happened. The reason is operator precedence — the order in which Python applies operators. In Python, & binds more tightly than > and <. That was decided long before pandas existed, because & was designed for whole numbers (bitwise "and"), where it should happen first. So Python reads the line as:

text
orders["quantity"] > (3 & orders["price"]) < 100

It tries to compute 3 & orders["price"] first — a bitwise "and" between the number 3 and a column of decimals — and that is what the error is about. The message mentions rand_ (the "right-hand and") and an array, but the cause is always the same: a missing pair of parentheses.

The error is the lucky outcome. Wrap only the first condition and forget the second, and there is no error at all:

python
print(len(orders[(orders["quantity"] > 3) & orders["price"] < 100]))
print(len(orders[(orders["quantity"] > 3) & (orders["price"] < 100)]))
text
8
4

The first line is read as ((quantity > 3) & price) < 100. The & mixes a mask with the prices, and every result happens to be below 100, so the "filter" keeps all eight rows. The correct filter keeps four. Same columns, same numbers, one missing pair of parentheses — and the wrong version runs without a sound. Wrap every condition in its own parentheses, every time. Then neither the error nor the silent version can happen.

Why not and?

The other natural instinct is to write the English word:

python
print(orders[(orders["branch"] == "north") and (orders["quantity"] > 3)])
text
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

Python's and, or and not need a single True or False on each side — they were built for if statements. To use and, Python asks the first mask "are you true?", and a column of eight answers cannot reply with one. Is it true if any value is True? If all are? Pandas refuses to guess, and the error lists the ways you could decide (a.any(), a.all()). But you did not want one answer; you wanted row-by-row "and". That is &. The rule to remember: inside a filter, and/or/not become &/|/~.

One of several values: isin

Orders from the south or east branch. You could write two equalities joined with |. With five branches that becomes five comparisons. isin takes a list and asks "is this row's value in the list?":

python
print(orders[orders["branch"].isin(["east", "south"])])
text
order_id        date branch   product     category  quantity  price
1      1002  2024-03-01  south  notebook   stationery         5   60.0
3      1004  2024-03-02   east    bottle  accessories         7  120.0
5      1006  2024-03-03  south       bag  accessories         1  850.0
6      1007  2024-03-04   east       pen   stationery        20   15.0

And here is where ~ earns its keep — every branch except north and east is ~ in front of an isin:

python
print(orders[~orders["branch"].isin(["north", "east"])]["order_id"].tolist())
text
[1002, 1006]

Two orders, both from the south branch. Note that ~orders["branch"].isin([...]) needs no extra parentheses: ~ applies to the result of the method call, and there is no & or | to fight with. As soon as you combine it with another condition, the parentheses come back: (~orders["branch"].isin([...])) & (orders["quantity"] > 3).

A range: between

Orders priced from 15 to 120. That is two comparisons, >= 15 and <= 120. between says it in one:

python
print(orders[orders["price"].between(15, 120)])
text
order_id        date branch   product     category  quantity  price
0      1001  2024-03-01  north       pen   stationery        12   15.0
1      1002  2024-03-01  south  notebook   stationery         5   60.0
3      1004  2024-03-02   east    bottle  accessories         7  120.0
6      1007  2024-03-04   east       pen   stationery        20   15.0
7      1008  2024-03-04  north    bottle  accessories         4  120.0

between includes both ends by default — the pens at exactly 15.0 and the bottles at exactly 120.0 are in. That matches how people usually say "from 15 to 120", but check it against the question. If the question means strictly between, say so:

python
print(orders["price"].between(15, 120, inclusive="neither").sum())
text
1

Only the notebook at 60.0 is strictly between. inclusive also accepts "left" and "right" for half-open ranges — the usual choice for money bands, where you want 0–100, 100–200 and so on without a boundary value landing in two bands.

Text patterns: the .str methods

Equality on text is exact. The .str accessor gives a text column the string methods you know from Python, applied to every row, and each returns a mask.

Products whose name starts with "b":

python
print(orders[orders["product"].str.startswith("b")])
text
order_id        date branch product     category  quantity  price
2      1003  2024-03-02  north     bag  accessories         2  850.0
3      1004  2024-03-02   east  bottle  accessories         7  120.0
5      1006  2024-03-03  south     bag  accessories         1  850.0
7      1008  2024-03-04  north  bottle  accessories         4  120.0

Products containing the letter "o", whatever the case:

python
print(orders[orders["product"].str.contains("O", case=False)])
text
order_id        date branch   product     category  quantity  price
1      1002  2024-03-01  south  notebook   stationery         5   60.0
3      1004  2024-03-02   east    bottle  accessories         7  120.0
7      1008  2024-03-04  north    bottle  accessories         4  120.0

There is no capital O anywhere in the data, yet the notebook and the two bottles are found, because case=False ignores case. Without it, .str.contains("O") would find nothing.

Exact equality has no such option. This is the most common "my filter returns nothing" bug:

python
print(orders[orders["branch"] == "North"])
text
Empty DataFrame
Columns: [order_id, date, branch, product, category, quantity, price]
Index: []

No error, no rows. The table is not wrong and pandas is not wrong — there is simply no branch spelled "North" with a capital letter. This is why step 2 of the plan printed unique(): the values you compare against should be copied from what is actually in the column, not typed from memory.

Text dates compare correctly — when they are ISO

The date column came in as str. Can you filter it with >=? Orders from 3 March onwards:

python
print(orders["date"] >= "2024-03-03")
text
0    False
1    False
2    False
3    False
4     True
5     True
6     True
7     True
Name: date, dtype: bool

It works, but understand why. These are strings, so >= compares them as text, character by character, the way a dictionary is ordered. That gives the right answer only because the dates are written year-month-day with zero padding (2024-03-03), so text order happens to equal time order. A column written 3/3/2024 would compare wrongly — "10/1/2024" sorts before "9/1/2024" because "1" comes before "9". For anything beyond simple cut-offs, convert to real dates with pd.to_datetime, covered in the chapter on adding columns.

Rows and columns together: .loc[mask, columns]

The filters so far kept every column. The manager wanted the north-branch orders, but the report only needs the product and quantity. You met .loc[rows, columns] in the last chapter, with labels. The rows part also accepts a mask:

python
print(orders.loc[orders["branch"] == "north", ["product", "quantity"]])
text
product  quantity
0     pen        12
2     bag         2
4  eraser        30
7  bottle         4

One step, one object, and the reader of your code sees the whole question — which rows, which columns — in one place. .loc[mask, "quantity"] (a single column name, not a list) gives you a Series instead, ready for arithmetic:

python
print(orders.loc[orders["branch"] == "north", "quantity"].sum())
text
48

12 + 2 + 30 + 4 = 48 items sold by the north branch.

query(): the same filter as a sentence

There is a second spelling for filters, query(), which takes the condition as a string:

python
print(orders.query("branch == 'north' and quantity > 3"))
text
order_id        date branch product     category  quantity  price
0      1001  2024-03-01  north     pen   stationery        12   15.0
4      1005  2024-03-03  north  eraser   stationery        30    8.0
7      1008  2024-03-04  north  bottle  accessories         4  120.0

The same three rows. Inside the string you write plain column names, and and/or/not are allowed, because query parses the string itself instead of handing it to Python's operators. To use a Python variable, prefix it with @:

python
min_qty = 5
print(orders.query("quantity >= @min_qty")["order_id"].tolist())
text
[1001, 1002, 1004, 1005, 1007]

query reads nicely for long conditions. Its costs: a typo inside the string is only discovered when the line runs, your editor cannot help you inside it, and column names with spaces need backticks. This course uses masks as the main tool, because masks are what everything else in pandas — .loc, assignment, counting — is built on. Know query so you can read it in other people's code.

The filtered table keeps its old labels

Back to the index. Save the north-branch orders and look at them:

python
north = orders[orders["branch"] == "north"]

print(north.index.tolist())
text
[0, 2, 4, 7]

Labels 0, 2, 4, 7. So what is north.loc[1]? There is no label 1 in this table — order 1002 came from the south branch and was filtered out:

python
print(north.loc[1])
text
KeyError: 1

.loc looks up a label, and label 1 does not exist here. If you meant "the second north-branch order", that is a position, and positions are .iloc:

python
print(north.iloc[1])
text
order_id           1003
date         2024-03-02
branch            north
product             bag
category    accessories
quantity              2
price             850.0
Name: 2, dtype: object

The second row by position is the bag order — and its Name is 2, its label from the original table. This is exactly the loc/iloc distinction from the last chapter, and filtering is where it bites, because after a filter labels and positions stop matching.

Keeping the labels is deliberate: it lets you trace any row in a result back to the original table. When you no longer need that — say the filtered table is the final report — renumber it:

python
print(north.reset_index(drop=True))
text
order_id        date branch product     category  quantity  price
0      1001  2024-03-01  north     pen   stationery        12   15.0
1      1003  2024-03-02  north     bag  accessories         2  850.0
2      1005  2024-03-03  north  eraser   stationery        30    8.0
3      1008  2024-03-04  north  bottle  accessories         4  120.0

drop=True throws the old labels away. Without it, they would be kept as a new column called index, which is rarely what you want in a report.

An empty result is not an error

python
none = orders[orders["branch"] == "west"]

print(none)
print(none.shape)
print(none.empty)
text
Empty DataFrame
Columns: [order_id, date, branch, product, category, quantity, price]
Index: []
(0, 7)
True

There is no west branch, so no rows — and no error. That is correct behaviour (zero is a valid answer), but it means a misspelt filter fails silently. Sums over an empty result are 0 and means are nan, which can travel a long way into a report before anyone notices. When a filter feeds something important, check result.empty or the row count, and say so out loud in the output: "No orders from the west branch." is better than a blank table.

Changing the filtered rows: .loc[mask, column] = value

Sometimes you filter in order to change something. The pens sold by the east branch were mispriced; they should have been 13.5, not 15.0. The tempting code does the filter and then picks the column — two pairs of brackets, the way you read values earlier:

python
import pandas as pd

orders = pd.read_csv("orders.csv")

orders[orders["branch"] == "east"]["price"] = 13.5

print(orders["price"].tolist())
text
[15.0, 60.0, 850.0, 120.0, 8.0, 850.0, 15.0, 120.0]
ChainedAssignmentError: A value is being set on a copy of a DataFrame or Series through chained assignment.
Such chained assignment never works to update the original DataFrame or Series, because the intermediate object on which we are setting values always behaves as a copy (due to Copy-on-Write).

Try using '.loc[row_indexer, col_indexer] = value' instead, to perform the assignment in a single step.

See the documentation for a more detailed explanation: https://pandas.pydata.org/pandas-docs/stable/user_guide/copy_on_write.html#chained-assignment

Read the prices: nothing changed. And pandas warned you, at length. (In a terminal the warning goes to stderr, so it appears above the list, at the moment the assignment line runs, and it starts with the file name and line number. Here it is shown after the output, without them.) This is chained assignment — two indexing steps, orders[mask] and then ["price"], with the assignment on the end of the chain. The first step, orders[mask], builds a new table holding the east-branch rows. The second step sets prices in that new table. Then the new table is thrown away, because nothing holds on to it. orders itself was never touched.

In pandas 3 this is guaranteed behaviour, called Copy-on-Write: any table you get by indexing behaves as its own copy, so writing into it can never reach back into the original. The ChainedAssignmentError is a warning, not an exception — the program keeps running — which is exactly why it is dangerous: a script with this line finishes "successfully" with the old prices still in place.

(If you read older tutorials, you will see this situation described with a SettingWithCopyWarning and a note that it "may or may not" work. In pandas 3 the answer is simple: it never works, so never write it.)

The fix is the warning's own advice: one indexing step, .loc[rows, column], with the assignment on it:

python
orders.loc[orders["branch"] == "east", "price"] = 13.5

print(orders["price"].tolist())
text
[15.0, 60.0, 850.0, 13.5, 8.0, 850.0, 13.5, 120.0]

Wait — that changed both east-branch orders, the pen and the bottle. The bottle was not mispriced. The code did exactly what it said; the question in the code was wrong. This is step 1 of the plan coming back: "pens sold by the east branch" is two conditions, and only one was written. Start again from a fresh copy of the file and write both:

python
import pandas as pd

orders = pd.read_csv("orders.csv")

fix = (orders["branch"] == "east") & (orders["product"] == "pen")
print(fix.sum())

orders.loc[fix, "price"] = 13.5
print(orders.loc[orders["branch"] == "east", ["product", "price"]])
text
1
  product  price
3  bottle  120.0
6     pen   13.5

Two habits are visible here. The mask gets a name (fix) because it is used for something important; and before writing, fix.sum() says how many rows are about to change — one, which is what we expected. Count before you overwrite, every time.

To read, df[mask]["col"] is fine. To write, always use df.loc[mask, "col"] = value. The chained form raises no exception, and the change silently disappears.

Saving a filtered table to work on is fine

What about this?

python
import pandas as pd

orders = pd.read_csv("orders.csv")

east = orders[orders["branch"] == "east"]
east["price"] = 0.0

print(east["price"].tolist())
print(orders["price"].tolist())
text
[0.0, 0.0]
[15.0, 60.0, 850.0, 120.0, 8.0, 850.0, 15.0, 120.0]

No warning, and that is right. Here you meant to make a separate table: you named it east, and changing it changes east only. orders is untouched. Copy-on-Write makes this behaviour predictable — a table you got by filtering is always independent of the original. The bug is only ever the chained form, where the intermediate table has no name and the change disappears with it.

A complete example

The manager now wants a short report. report.py:

python
import pandas as pd

orders = pd.read_csv("orders.csv")

# Check what arrived before asking it anything.
print("Rows, columns:", orders.shape)
print("Branches:", list(orders["branch"].unique()))
print()

# 1. The original question: north-branch orders of more than three items.
north_big = orders.loc[
    (orders["branch"] == "north") & (orders["quantity"] > 3),
    ["order_id", "product", "quantity"],
]
print("North orders over 3 items:")
print(north_big.reset_index(drop=True))
print()

# 2. Counting with masks: no new table needed.
accessories = orders["category"] == "accessories"
print("Accessory orders   :", accessories.sum())
print("Share of all orders:", accessories.mean())
print("Accessory items    :", orders.loc[accessories, "quantity"].sum())
print()

# 3. Outside north, cheap items (under 100).
cheap_outside = orders[
    (orders["branch"] != "north") & (orders["price"] < 100)
]
print("Cheap orders outside north:", cheap_outside["order_id"].tolist())

# 4. A branch we do not have: say so instead of printing an empty table.
west = orders[orders["branch"] == "west"]
if west.empty:
    print("No orders from the west branch.")
text
Rows, columns: (8, 7)
Branches: ['north', 'south', 'east']

North orders over 3 items:
   order_id product  quantity
0      1001     pen        12
1      1005  eraser        30
2      1008  bottle         4

Accessory orders   : 4
Share of all orders: 0.5
Accessory items    : 14

Cheap orders outside north: [1002, 1007]
No orders from the west branch.

Why it is written this way:

  • Check first, then ask. shape and the list of branches are printed before any filter. If the file had arrived with North capitalised, the branches line would show it before the report printed an empty table.
  • .loc[mask, columns] for the main question. The rows and the columns of the answer are decided in one place, and reset_index(drop=True) renumbers the result because this is the final report, not a step on the way.
  • A named mask, used three times. accessories is computed once and then counted (sum), measured (mean) and used to select a column. Naming a mask is how you avoid writing the same condition three times and getting one of them slightly different.
  • Every condition in its own parentheses, even in the short ones, so adding a third condition later cannot trigger the precedence error.
  • The empty case is handled out loud. Zero rows is a valid answer, and the report says it in words.

When it breaks

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). You used and, or or not between two masks — or an if on a whole mask. Python's keywords need one True/False, and a mask has one per row. Inside a filter, write &, |, ~. Inside an if, decide what you mean: if mask.any(): ("is at least one row true?") or if mask.all(): ("are all rows true?").

TypeError: Cannot perform 'rand_' with a dtyped [float64] array and scalar of type [bool] (or 'ror_', or [int64]) Missing parentheses around a condition next to & or |. Python applied & before >, so it tried a bitwise "and" between a number and a column. Put each condition in its own parentheses: (a > 3) & (b < 100).

The filter returns an empty table, with no error Your value does not occur in the column exactly as typed. Print df["col"].unique() — or better, list(df["col"].unique()), which shows each value inside quotes, so a trailing space in 'north ' becomes visible. Fix the data with df["col"].str.strip(), or match loosely with .str.contains(..., case=False).

KeyError: 1 (or another number) right after filtering You used .loc[n] on a filtered table as if it still had labels 0, 1, 2…. It keeps the original labels. Use .iloc[n] for "the n-th row", or reset_index(drop=True) first if the old labels are no longer useful.

ChainedAssignmentError: A value is being set on a copy of a DataFrame or Series through chained assignment. You wrote df[mask]["col"] = value. It never changes df in pandas 3 — the warning is telling you the line did nothing. Write df.loc[mask, "col"] = value.

UserWarning: Boolean Series key will be reindexed to match DataFrame index. The mask was built from a different table than the one you are filtering — typically a mask from the full orders used on the filtered north. Pandas lines it up by label and silently drops labels it cannot find:

python
import pandas as pd

orders = pd.read_csv("orders.csv")

north = orders[orders["branch"] == "north"]
big = orders["quantity"] > 5

print(north[big])
text
order_id        date branch product    category  quantity  price
0      1001  2024-03-01  north     pen  stationery        12   15.0
4      1005  2024-03-03  north  eraser  stationery        30    8.0
UserWarning: Boolean Series key will be reindexed to match DataFrame index.

(In a terminal the warning appears above the table.) It happens to give a sensible answer here, but only by luck of the labels. Build the mask from the table you are filtering: north[north["quantity"] > 5].