Lists, dictionaries, sets, loops and comprehensions

The problem we are solving

The head of support wants to know what customers complained about most in this week's tickets. The tickets are messy text — some in capitals, some full of punctuation, some saying the same thing twice. A colleague wrote a script in five minutes and sent a list: the at the top, then refund, then Refund and refund. as separate entries.

The second run brought another problem: a small change in the order of the tickets changed the fifth word of the top five, even though no word's count had changed. Nobody reading the report could tell which version was right.

The answer is not a special library. It is Python's four basic containers — lists, tuples, dictionaries and sets — and loops and conditions used properly. A large share of an AI engineer's daily work is exactly this: cleaning text, counting, filtering and sorting, in a way that gives the same result every time for the same input.

By the end of this chapter you can

• Choose between a list, a tuple, a dictionary and a set for a job, and say why
• Write clean loops with for, while, if/elif/else, enumerate and .items()
• Filter and transform data with list, dictionary and set comprehensions
• Extract words from messy text and count them with a dictionary and with Counter
• Sort words with equal counts by an explicit rule, so the result is the same every time

Prerequisites: pathlib, JSON/YAML config and environment variables — and being able to run a script with uv run.

Four containers, four jobs

First, the four side by side:

tickets = ["refund", "login", "refund", "billing"]
latest = ("T-1042", "refund", 3)
owners = {"refund": "Maya", "login": "Omar"}
seen = set(tickets)

print(tickets[0], tickets[-1], len(tickets))
print(latest[1])
print(owners["login"])
print(sorted(seen))
print("login" in seen, "shipping" in seen)

refund billing 4
refund
Omar
['billing', 'login', 'refund']
True False

Each one makes a different promise, and you choose by the promise:

• List ([...]) — keeps order, allows the same value more than once, can be changed. For "a row of things that come one after another": a list of tickets, a batch of prompts.
• Tuple ((...)) — keeps order but cannot be changed after it is made. For a few values that together form one record, like ("T-1042", "refund", 3) — id, category, priority. Position carries meaning here.
• Dictionary ({key: value}) — look up a value by key. "Who owns which category", "how many times each word" — anywhere you go from one thing to another.
• Set (set(...) or {a, b}) — each value once, no guaranteed order. For when the question is only "is it there" or "how many distinct values".

sorted(seen) is printed because the order of a printed set is not guaranteed — it can look one way on one run and another way on the next. If a program's output depends on a set's order, that output will not be stable either.

in on a list checks every element one by one; on a set or dictionary it answers almost immediately, whether there are ten elements or ten million. If you check if word in stop_words over and over, stop_words should be a set, not a list. Why that is, and how to measure the difference, is coming later in the course.

Loops: say what you are looping over

Loops over tuples and dictionaries read most clearly with unpacking — values go straight into named variables:

ticket_id, category, priority = ("T-1042", "refund", 3)
print(ticket_id, category, priority)

owners = {"refund": "Maya", "login": "Omar"}
for category, owner in owners.items():
    print(f"{category} -> {owner}")

for position, category in enumerate(["refund", "login"], start=1):
    print(position, category)

T-1042 refund 3
refund -> Maya
login -> Omar
1 refund
2 login

.items() gives a (key, value) pair each time, and enumerate gives each element along with its position — no counting indexes with range(len(...)). Since Python 3.7 a dictionary remembers the order its keys were inserted, so the loop over owners starts with refund.

What happens if you forget .items()?

owners = {"refund": "Maya", "login": "Omar"}
for category, owner in owners:
    print(category, owner)

Traceback (most recent call last):
  File "main.py", line 2, in <module>
    for category, owner in owners:
        ^^^^^^^^^^^^^^^
ValueError: too many values to unpack (expected 2)

Looping directly over a dictionary gives only the keys. The first key, "refund", is a six-character string, and Python tries to split it into two variables — six characters, two slots. The message looks strange until you remember that a string is a sequence too.

if/elif/else stops at the first condition that is true, so the order of the conditions is the logic:

latencies_ms = [120, 340, 95, 1800, 210, 2600]

for latency in latencies_ms:
    if latency >= 2000:
        label = "critical"
    elif latency >= 1000:
        label = "slow"
    else:
        label = "ok"
    print(latency, label)

120 ok
340 ok
95 ok
1800 slow
210 ok
2600 critical

Write the conditions in the opposite order — >= 1000 first, >= 2000 second — and there is no error, but 2600 becomes slow, because the first condition is true and the chain stops there. The critical branch never runs again. When you split by thresholds, put the strictest condition first.

for runs over a known collection. while runs as long as a condition holds — when you do not know in advance how many times it will take. Fetching data from an API page by page is the classic case. Here a dictionary plays the part of the API:

pages = {1: ["t1", "t2"], 2: ["t3"], 3: []}

page = 1
collected = []
while True:
    batch = pages[page]
    if not batch:
        break
    collected.extend(batch)
    page += 1

print(collected, "pages read:", page - 1)

['t1', 't2', 't3'] pages read: 2

An empty page means break. if not batch works because an empty list is falsy. In loops like this, write the stopping condition first and the rest after — a while True that forgets to stop runs forever, with no error at all.

Comprehensions: one collection from another

"An empty list, a loop, an if, an append" — this four-line pattern is so common that Python has a one-line form for it:

latencies_ms = [120, 340, 95, 1800, 210, 2600]

slow = [ms for ms in latencies_ms if ms >= 1000]
seconds = [ms / 1000 for ms in latencies_ms]
labels = {ms: ("slow" if ms >= 1000 else "ok") for ms in latencies_ms}
buckets = {ms // 1000 for ms in latencies_ms}

print(slow)
print(seconds)
print(labels)
print(sorted(buckets))

[1800, 2600]
[0.12, 0.34, 0.095, 1.8, 0.21, 2.6]
{120: 'ok', 340: 'ok', 95: 'ok', 1800: 'slow', 210: 'ok', 2600: 'slow'}
[0, 1, 2]

How to read it: [result for element in collection if condition]. The left part says what to build from each element; the trailing if says which ones to keep. The same shape with {key: value for ...} builds a dictionary, and {value for ...} a set — buckets holds three distinct one-second buckets, each once.

Do not confuse the two kinds of if. The trailing if ms >= 1000 is a filter — when it is false, the element is dropped. "slow" if ms >= 1000 else "ok" is a choice — every element stays, and its value is one of the two. That is why labels has all six.

A comprehension is for building a new collection. When you only want to print something or write to a file — when the result would be thrown away — write an ordinary for loop. And a three-level nested comprehension may fit on one line, but a five-line loop is often easier for a colleague to read.

Counting words: by hand, then with Counter

Almost everyone's first attempt at counting with a dictionary looks like this:

words = ["refund", "late", "refund", "login", "refund", "late"]

counts = {}
for word in words:
    counts[word] += 1

Traceback (most recent call last):
  File "main.py", line 5, in <module>
    counts[word] += 1
    ~~~~~~^^^^^^
KeyError: 'refund'

counts[word] += 1 means counts[word] = counts[word] + 1 — and reading counts[word] on the right finds that the key "refund" does not exist yet. A word seen for the first time needs a starting value. .get(key, 0) provides exactly that:

words = ["refund", "late", "refund", "login", "refund", "late"]

counts = {}
for word in words:
    counts[word] = counts.get(word, 0) + 1

print(counts)

{'refund': 3, 'late': 2, 'login': 1}

The job is common enough that the standard library has a type for it — collections.Counter, a kind of dictionary:

from collections import Counter

words = ["refund", "late", "login", "refund", "login", "late", "slow"]
counts = Counter(words)

print(counts)
print(counts.most_common(2))
print(counts["missing"])

Counter({'refund': 2, 'late': 2, 'login': 2, 'slow': 1})
[('refund', 2), ('late', 2)]
0

Reading a missing key from a Counter gives 0, not a KeyError — exactly right for counting. But look at most_common(2): refund, late and login all have a count of 2, and the first two came back. Why was login dropped? Because for equal counts, most_common keeps the order in which the words were first seen. login first appeared third in the list, so it lost. Change the order of the tickets and the result changes too — the mystery from the start of the chapter.

Cleaning text: which words are really words?

split() only breaks on whitespace:

text = "Refund not received. REFUND please! Login fails; login page is slow."

print(text.split())
print(text.lower().split())

['Refund', 'not', 'received.', 'REFUND', 'please!', 'Login', 'fails;', 'login', 'page', 'is', 'slow.']
['refund', 'not', 'received.', 'refund', 'please!', 'login', 'fails;', 'login', 'page', 'is', 'slow.']

.lower() settled the upper/lower case difference, but received., please! and slow. still carry punctuation — and when counting, slow. and slow are two different words. This is where a regular expression helps: instead of saying what to throw away, you say what a word looks like:

import re

text = "Refund not received. REFUND please! Login fails; login page is slow. Can't pay."

words = re.findall(r"[a-z']+", text.lower())
print(words)

['refund', 'not', 'received', 'refund', 'please', 'login', 'fails', 'login', 'page', 'is', 'slow', "can't", 'pay']

[a-z']+ means "one or more lowercase English letters or apostrophes". re.findall returns every place the pattern matches in the text, and everything else — full stops, commas, digits — is dropped. can't stays one word.

This pattern has a gap: if someone writes 'invalid token', the first word comes out as 'invalid — with the quote attached. The next example tightens the pattern so an apostrophe can only sit between two letters. And remember that [a-z] is English letters only; text in other scripts needs a different pattern.

Equal counts: decide the sorting rule yourself

To turn counts into a ranking you have to sort. Three ways, on a dictionary where refund and login both have a count of 3:

counts = {"refund": 3, "late": 2, "login": 3, "slow": 1}

print(sorted(counts, reverse=True))
print(sorted(counts.items(), key=lambda item: item[1], reverse=True))
print(sorted(counts.items(), key=lambda item: (-item[1], item[0])))

['slow', 'refund', 'login', 'late']
[('refund', 3), ('login', 3), ('late', 2), ('slow', 1)]
[('login', 3), ('refund', 3), ('late', 2), ('slow', 1)]

The first line is the most dangerous, because it looks reasonable. Sorting a dictionary sorts its keys, not its values — so this is reverse alphabetical order and has nothing to do with the counts. slow appeared once, yet it is on top. No error.

The second line sorts by count. But which comes first, refund or login? Python's sorted is "stable" — elements with equal keys keep the order they had in the input. refund was inserted into the dictionary first, so it comes first. In other words, the result depends on the order the data arrived in.

The third line states the rule itself. key returns a tuple, and tuples compare element by element: first -item[1] — the negative count, so bigger counts come first — and on a tie, item[0], the word itself, alphabetically. Now the result is the same whatever order the input arrives in. There is no need for reverse=True, because the minus sign already reversed the count order while leaving the alphabetical order as it is.

Wherever results will be compared — reports, tests, evaluations — give an explicit rule for ordering equal values. Otherwise today's report and tomorrow's will differ where the data does not.

A complete example

Now all of it together: the top five words in the tickets. Stop words — words like the and and that appear in every text but say nothing — are dropped, two-letter words are dropped, and ties are broken alphabetically. top_words.py:

import re
from collections import Counter

STOP_WORDS = {
    "a", "and", "for", "i", "is", "it", "my", "not", "of", "on", "the", "to", "was", "with",
}

TICKETS = [
    "Refund not received for order 1182. I was charged twice!",
    "The app is slow and the login page freezes.",
    "Charged twice again. Please refund the second charge.",
    "Login fails with 'invalid token' on my phone.",
    "Still waiting on my refund, it was promised last week.",
    "App slow after the update; login takes a minute.",
]


def tokenize(text: str) -> list[str]:
    return re.findall(r"[a-z]+(?:'[a-z]+)?", text.lower())


def top_words(texts: list[str], n: int = 5) -> list[tuple[str, int]]:
    counts = Counter(
        word
        for text in texts
        for word in tokenize(text)
        if word not in STOP_WORDS and len(word) > 2
    )
    ranked = sorted(counts.items(), key=lambda item: (-item[1], item[0]))
    return ranked[:n]


def print_ranking(ranking: list[tuple[str, int]]) -> None:
    if not ranking:
        print("no words to rank")
        return
    width = max(len(word) for word, _ in ranking)
    for rank, (word, count) in enumerate(ranking, start=1):
        print(f"{rank}. {word:<{width}}  {count}  {'#' * count}")


if __name__ == "__main__":
    print_ranking(top_words(TICKETS))
    print_ranking(top_words([]))

uv run top_words.py

1. login    3  ###
2. refund   3  ###
3. app      2  ##
4. charged  2  ##
5. slow     2  ##
no words to rank

The decisions behind it:

The word pattern [a-z]+(?:'[a-z]+)?. One or more letters, then optionally an apostrophe and more letters. can't stays one word, but the quote around 'invalid token' no longer becomes part of a word. (?:...) is a group that is not captured — with a plain (...), re.findall would return only the group's part instead of the whole word.

Stop words are a set. Every word is checked with word not in STOP_WORDS; on a set that check is near-instant however long the list grows.

Counter fed directly by a generator. Two fors and an if make one stream — every word of every ticket that passes the filter. No big intermediate list of all the words is built. The two fors are in the same order as ordinary nested loops: the outer one first.

An explicit tie-break. login and refund both have 3; app, charged, slow and twice all have 2. Alphabetical order decided who got fifth place, so twice was dropped. The rule is not perfect — twice was not dropped for being less important — but it is declared and it is the same every time. If someone reading the report asks, there is an answer.

Empty input handled on purpose. max() on an empty collection raises ValueError: max() iterable argument is empty. A quiet week with no tickets is normal, not an error — so print_ranking checks for an empty ranking first. The second output line shows exactly that.

Counting and displaying kept apart. top_words returns a list and prints nothing; print_ranking only prints. So the same counts can be used from a test, an API or a dashboard. {word:<{width}} left-aligns each word in a space as wide as the longest one, so the count column lines up.

One limitation: charged and charge are two separate words here. Merging the forms of a word is its own problem, and AI systems usually match by meaning rather than by counting words — that approach is coming later in the course. But even there, cleaning text, filtering and sorting in a stable order — the work of this chapter — are the first steps.

When it breaks

KeyError: 'refund' counts[word] += 1 for a key that is not in the dictionary yet. Write counts[word] = counts.get(word, 0) + 1, or use collections.Counter.

ValueError: too many values to unpack (expected 2) The code says for key, value in d: over a dictionary. Looping directly over a dictionary gives only keys; write d.items() to get pairs.

TypeError: unhashable type: 'list' A list was put into a set or used as a dictionary key. Set elements and dictionary keys must be things that cannot change. Use a tuple instead of a list: seen.add(("refund", "late")).

RuntimeError: dictionary changed size during iteration Keys were deleted from or added to a dictionary while looping over it. Build a new dictionary instead of deleting: counts = {word: n for word, n in counts.items() if n >= 2}.

AttributeError: 'tuple' object has no attribute 'append' A tuple cannot be changed after it is made. If the values really grow and shrink, it should have been a list; if the tuple is a record, build a new tuple with the new value.

ValueError: max() iterable argument is empty max() or min() on an empty collection. Check whether it is empty first, or give a default: max(lengths, default=0).

Your turn

Now find the top words per category. In a file called categories.py, copy the chapter's STOP_WORDS and tokenize, and add these tickets:

TICKETS = [
    ("billing", "Charged twice for one order."),
    ("billing", "Refund for the double charge, please."),
    ("login", "Login fails on the new phone."),
    ("billing", "Still no refund after the charge."),
    ("login", "Password reset email never arrives."),
    ("login", "Login page shows a blank screen."),
]

Then write code so that:

1. Each category has its own counts — a dictionary whose keys are category names and whose values are that category's Counter (hint: dict.setdefault and Counter.update)
2. Stop words and words of two letters or fewer are not counted
3. Each category shows its top two words, higher counts first and alphabetical order on a tie
4. Categories are printed in alphabetical order, whatever order the tickets come in

uv run categories.py should print exactly this:

billing  charge=2, refund=2
login    login=2, arrives=1

Then reverse the TICKETS list (TICKETS[::-1]) and run it again — does the output change at all? If it does, which condition is not met? Finally, remove the tie-break and sort by count alone: what is the second word for login, and does it stay the same when you reverse the order?