Python Advanced – functools, operator, and reduce

The Python track makes the case that a comprehension beats map and filter, and it is right. This post covers what survives that argument: functools, the operator module, and the cases where reduce is genuinely the clearest thing to write.

Why map and filter mostly lose

amounts = [100, -25, 250, -40]

print(list(map(lambda n: n * 2, amounts)))        # Output: [200, -50, 500, -80]
print([n * 2 for n in amounts])                   # Output: [200, -50, 500, -80]

print(list(filter(lambda n: n > 0, amounts)))     # Output: [100, 250]
print([n for n in amounts if n > 0])              # Output: [100, 250]

The comprehension is shorter, needs no list() to be printable, and reads in the order the work happens. That is the whole case, and it holds for nearly every map or filter written with a lambda.

The exception is when you already have a named function, where map is genuinely tidy: map(str.strip, lines) beats [line.strip() for line in lines] by a nose, and map over a large file is lazy where the comprehension is not.

reduce, and when it is the right shape

from functools import reduce
import operator

amounts = [100, 25, 250]

print(reduce(operator.add, amounts))          # Output: 375
print(sum(amounts))                           # Output: 375

That first line is the example everyone teaches and it is the wrong one — sum already exists and is clearer. Guido removed reduce from the builtins in Python 3 for precisely this reason.

It earns its place when the combining step is not a builtin and the accumulator is not a number:

from functools import reduce

permissions = [{"read"}, {"read", "write"}, {"write", "admin"}]
print(sorted(reduce(set.union, permissions)))     # Output: ['admin', 'read', 'write']

overrides = [{"a": 1}, {"b": 2}, {"a": 99}]
print(reduce(lambda acc, d: acc | d, overrides, {}))   # Output: {'a': 99, 'b': 2}

Merging a list of dicts in precedence order, unioning permission sets, composing functions — one expression each. The third argument is the starting value, which is what makes an empty input return {} rather than raising TypeError.

Note the merge result: a is 99, not 1. Later dicts win, and the key keeps its original position while taking the new value — which is exactly the semantics you want for "defaults, then environment, then command line", and worth confirming rather than assuming.

The test: if a for loop with an accumulator would be clearer, write the loop. Most of the time it is.

operator, instead of small lambdas

import operator

accounts = [
    {"kind": "Savings", "balance": 8400.5},
    {"kind": "Checking", "balance": 1250.0},
]

by_balance = sorted(accounts, key=operator.itemgetter("balance"))
print(by_balance[0]["kind"])              # Output: Checking

rows = [("b", 2), ("a", 1)]
print(sorted(rows, key=operator.itemgetter(0))[0])     # Output: ('a', 1)

print(operator.itemgetter("kind", "balance")(accounts[0]))
# Output: ('Savings', 8400.5)

itemgetter("balance") does what lambda a: a["balance"] does, is implemented in C, and — the actual reason to prefer it — asking for two keys returns a tuple, which is the multi-column sort key in one call.

attrgetter is the same for attributes, and methodcaller("strip") for methods. Use them for key functions; do not reach for operator.add where + would do.

partial, for pre-filling arguments

from functools import partial

def format_money(value, symbol="$", places=2):
    return f"{symbol}{value:,.{places}f}"

as_gbp = partial(format_money, symbol="£")
whole = partial(format_money, places=0)

print(format_money(1250.5))     # Output: $1,250.50
print(as_gbp(1250.5))           # Output: £1,250.50
print(whole(1250.5))            # Output: $1,250

partial builds a new callable with some arguments already supplied. Where it pays off is any API that takes a callback and gives you no way to pass extra arguments — a key= function, a signal handler, a thread target.

The alternative is a lambda that closes over a variable, which is where the late-binding trap lives: partial captures values at the moment you call it, so a loop producing partials produces correct ones.

cache and lru_cache

from functools import cache

calls = []

@cache
def fib(n):
    calls.append(n)
    return n if n < 2 else fib(n - 1) + fib(n - 2)

print(fib(30))                # Output: 832040
print(len(calls))             # Output: 31
print(fib.cache_info().hits)  # Output: 28

Thirty-one calls instead of about 2.7 million, from one decorator line. @cache (3.9) is unbounded; @lru_cache(maxsize=128) evicts the least recently used, which is what you want when the inputs are open-ended.

Three conditions, all required. The function must be pure — same input, same output, no side effects. Its arguments must be hashable, so no lists or dicts. And the cache lives for the life of the process, so caching something that reads a file means never noticing the file changed.

singledispatch, instead of isinstance chains

from functools import singledispatch
from decimal import Decimal

@singledispatch
def to_csv(value):
    return str(value)

@to_csv.register
def _(value: Decimal):
    return f"{value:f}"

@to_csv.register
def _(value: bool):
    return "1" if value else "0"

@to_csv.register
def _(value: list):
    return ";".join(to_csv(v) for v in value)

print(to_csv(Decimal("1250.00")))          # Output: 1250.00
print(to_csv(True))                        # Output: 1
print(to_csv([Decimal("1.5"), False]))     # Output: 1.5;0
print(to_csv(42))                          # Output: 42

The dispatch is on the type of the first argument, taken from the annotation. It replaces a chain of isinstance checks with something you can extend from another module without editing the original function — which is the real benefit, and why serialization libraries are built on it.

Note bool registered separately from int. Since bool is a subclass of int, dispatch picks the most specific registration, so the order of the register calls does not matter.

What to reach for

Next: Serialization.