Functional Patterns in Python: Comprehensions, itertools and functools
Comprehensions, functools and itertools working together: how I show in projects when a comprehension is a better choice than map/filter, where lru_cache and groupby shine, and which pitfalls are waiting.
from itertools import groupby
from operator import itemgetter
records = [
{"team": "a", "pts": 3},
{"team": "b", "pts": 5},
{"team": "a", "pts": 2},
]
totals = {
team: sum(r["pts"] for r in group)
for team, group in groupby(records, key=itemgetter("team"))
}
# {"a": 2, "b": 5} – erwartet war {"a": 5, "b": 5}
No traceback, no warning, just a wrong result. groupby only groups consecutive equal keys. Without sorting first, team a falls apart into two groups, and the second one overwrites the first in the dictionary. If you know GROUP BY from SQL, you expect the exact opposite here.
That is how Python’s functional tools behave across the board: they carry familiar names from Java, C# or SQL, but follow their own rules. If you want to work functionally in Python, you reach for map and filter less often and for comprehensions, functools and itertools more often. Together these three cover almost everything you know from functional programming – in a style that is typical for Python.
I go through them in order: first comprehensions as the idiomatic replacement for map/filter, then the lazy combinators from itertools, then the helpers from functools – above all lru_cache – and finally the leaning toward immutability. In several places I show not only how it works, but where I regularly see mistakes in practice. If you have already looked at advanced Python, you know generators and decorators – here we build on that.
Comprehensions instead of map and filter#
Let us start with a concrete example. I want the squares of all even numbers from 0 to 9. In the map/filter version it reads like this:
nums = range(10)
# map/filter with lambdas – works, but reads inside out
squares_even = list(map(lambda n: n * n, filter(lambda n: n % 2 == 0, nums)))
That works, but you read it from the inside out: first filter, then map, and the two lambda expressions sit between the arguments. The comprehension turns this around and puts the transformation in front and the condition at the end:
nums = range(10)
# comprehension – transform first, condition last, one line
squares_even = [n * n for n in nums if n % 2 == 0]
This is the form that dominates in Python code, and the reason is simply readability. Idiomatic Python code usually prefers a comprehension over map or filter with lambda; PEP 8, however, does not prescribe this particular choice. Comprehensions come in three flavors – for lists, dictionaries and sets:
words = ["alpha", "beta", "gamma"]
# list, dict and set comprehensions
lengths = [len(w) for w in words]
by_length = {w: len(w) for w in words}
initials = {w[0] for w in words}
One detail is easy to miss: since Python 3, a comprehension has its own namespace. The loop variable w stays inside the comprehension and no longer leaks into the surrounding code – an old stumbling block from Python 2 days that you no longer need to fear.
Does that mean map and filter are superfluous? No. They are still idiomatic when a named function already exists. map(str.upper, words) is clearer than [w.upper() for w in words] – there is no lambda here, just a ready-made function, and the expression stays short:
words = ["alpha", "beta", "gamma"]
# map shines with an existing named function – no lambda noise
shouted = list(map(str.upper, words))
So the lambda works as a signal: if there is one in map or filter, a comprehension would be the better choice. A named function in map, on the other hand, is perfectly fine.
A related detail regularly causes confusion: the difference between the filter at the end and a conditional expression in front. Both use the keyword if, but they do opposite things:
nums = range(10)
# if at the end filters elements out – only evens survive
evens = [n for n in nums if n % 2 == 0]
# if ... else at the front transforms – all elements stay
masked = [n if n % 2 == 0 else 0 for n in nums]
The if at the end selects and drops the odd numbers; the if ... else in front keeps every element and only replaces the odd ones with 0. Two constructs that are easy to mix up and produce completely different results – I trip over this regularly in code reviews.
Lazy evaluation with map, filter and generator expressions#
One point is often overlooked: in Python 3, map and filter no longer return lists, but iterators. They compute on demand – values are only produced when you ask for them. That is exactly why the examples above wrap them in list(...); otherwise you would just get the iterator object.
A comprehension, by contrast, materializes immediately: it builds the complete list in memory. For small amounts of data that does not matter; for large ones it can get expensive. This is where the generator expression comes in – syntactically almost identical to the list comprehension, just with parentheses instead of square brackets:
# list comprehension: builds the whole list in memory
total_eager = sum([n * n for n in range(1_000_000)])
# generator expression: yields one value at a time, constant memory
total_lazy = sum(n * n for n in range(1_000_000))
Both return the same result, but the second version does not materialize a complete result list. It produces its values on demand; upstream data sources or individual operations can of course still take up memory. When I only iterate over an expression once anyway – say as an argument to sum, any or max – the generator expression is usually the right choice.
A word of caution: a generator is exhausted after one pass. If you want to iterate over the same generator twice, the second pass returns an empty result. For repeated access you need a materialized list – or you design the pipeline so that one pass is enough.
So we have three ways to process a sequence, and each has its place:
- Comprehension – readable, materializes immediately, its own namespace, filter and transformation in one expression.
map/filter– lazy, ideal in combination with an existing named function, less readable withlambda.itertools– lazy combinators for long or infinite streams, constant memory, composes well.
Let us take a closer look at the third one now.
itertools: lazy combinators for streams#
itertools is a collection of building blocks that all work lazily. They produce iterators, not lists, and can be plugged together into pipelines without intermediate results piling up in memory. A few that I use almost daily:
from itertools import chain, islice, count
# count: infinite counter, here 1, 3, 5, 7, 9
first_five_odds = list(islice(count(1, 2), 5))
# chain.from_iterable: flatten a list of lists lazily
flat = list(chain.from_iterable([[1, 2], [3, 4], [5, 6]]))
Besides chain, islice and count, I often reach for takewhile and dropwhile. Both take a predicate and a stream: takewhile yields values as long as the condition holds and stops for good at the first False; dropwhile skips the leading block for which the predicate is true and returns everything from there on. On a sorted stream, this lets you cut out a range without checking the whole rest.
count(1, 2) is an infinite counter – without islice it would never stop. islice cuts out the first five values and stops evaluation exactly there. That is the pattern behind lazy pipelines: you describe a potentially infinite source and limit it only at the end. chain.from_iterable, in turn, flattens nested sequences without copying the sublists together first.
The building block that triggers the most aha moments is groupby. It lets you group records by a key and aggregate them:
from itertools import groupby
from operator import itemgetter
records = [
{"team": "a", "pts": 3},
{"team": "a", "pts": 2},
{"team": "b", "pts": 5},
]
# IMPORTANT: sort by the same key before grouping
records.sort(key=itemgetter("team"))
totals = {
team: sum(r["pts"] for r in group)
for team, group in groupby(records, key=itemgetter("team"))
}
# {"a": 5, "b": 5}
This brings us back to the example from the beginning: if the input is not sorted by the same key first, you get several groups for the same key, and the result is silently wrong. The records.sort(...) before groupby is not optional; it is mandatory – it is the one line that makes the difference between {"a": 5, "b": 5} and {"a": 2, "b": 5}.
A second point: the groups that groupby returns are transient iterators. As soon as you move on in the outer loop, the previous group becomes invalid. If you want to keep a group beyond the iteration, you have to materialize it with list(group) before moving on. In the example above that is no problem, because sum(...) evaluates the group directly in the same step.
Before we get to functools, a look at the big picture. The following sketch shows how raw data moves through such a pipeline – from the eagerly materializing comprehension through lazy generator expressions and itertools combinators to the final fold into a single value.
graph LR
A[Raw data] --> B[Comprehension<br/>eager, materialised]
B --> C[Generator expr<br/>lazy, constant memory]
C --> D[itertools<br/>chain / islice / groupby]
D --> E[reduce / sum / prod<br/>fold to a value]
E --> F[Result]functools: partial, reduce and lru_cache#
functools bundles tools that act on functions themselves. Three of them are central to functional patterns.
partial binds some of a function’s arguments in advance and returns a new callable function. That is handy for specializing general functions:
from functools import partial
def power(base, exponent):
return base ** exponent
# fix the exponent, get a specialised function
square = partial(power, exponent=2)
cube = partial(power, exponent=3)
print(square(5)) # 25
print(cube(5)) # 125
reduce folds a sequence into a single value – the classic of functional programming. In Python 3, reduce is deliberately no longer a builtin but was moved to functools. That is a clear signal: use it sparingly.
from functools import reduce
flat = [1, 2, 3, 4]
# reduce works, but a built-in is almost always clearer
total = reduce(lambda acc, x: acc + x, flat, 0) # 10
better = sum(flat) # same result, obvious intent
One detail of reduce is easy to overlook: the third argument. It is the initial value of the fold and at the same time decides what happens with an empty sequence. With an initial value of 0, reduce returns exactly that 0 for empty input; without an initial value, reduce raises a TypeError on an empty sequence instead. If you leave out the initial value, you buy yourself a special case that is easy to forget in tests – and that then strikes precisely in the edge case.
This is where the most common functools mistake I see shows up: reduce gets overused. Nested reduce lambdas are hard to read, and for most cases there are clearer alternatives – sum for sums, math.prod for products (since Python 3.8), a comprehension or simply a loop. reduce is justified when the fold is genuinely non-trivial and no specialized function exists. In all other cases, the more explicit way wins.
For me, the star of functools is lru_cache. The decorator memoizes a function’s results – the same arguments return the stored result instead of computing it again. The classic textbook example is the naive recursive Fibonacci function, whose runtime explodes exponentially without memoization:
from functools import lru_cache
@lru_cache(maxsize=None)
def fib(n: int) -> int:
return n if n < 2 else fib(n - 1) + fib(n - 2)
print(fib(50)) # 12586269025 – instant instead of minutes
print(fib.cache_info()) # CacheInfo(hits=48, misses=51, maxsize=None, currsize=51)
With lru_cache, every Fibonacci value is computed exactly once – the complexity drops from exponential to linear. cache_info() gives insight into hits and misses, cache_clear() empties the cache. A note on style: I consistently write @lru_cache(maxsize=None) with parentheses. That is unambiguous across all versions and saves discussions about which Python version allows the call without parentheses.
There are two conditions you need to know. First: the arguments must be hashable – lists or dictionaries as parameters do not work, and tuples only if their elements are hashable too. Second: lru_cache fits when the result depends only on those arguments and the returned object can safely be reused. With side effects or mutable external state, the cache easily returns wrong results. And maxsize=None means an unbounded cache – with a great many different arguments, memory usage can grow without limit. For such cases you set a concrete maxsize.
functools offers more than fits into one article – for example singledispatch, which lets a function resolve differently depending on the type of its first argument, or cmp_to_key, which turns an old comparison function into a key for sorted. For functional patterns in the narrower sense, though, partial, reduce and lru_cache are the three I reach for most often in practice.
The leaning toward immutability#
Functional work and immutability go together. Python offers two immutable containers that help here: tuple for ordered sequences and frozenset for sets. Both are hashable as long as their elements are hashable, and can then be used as dictionary keys or as elements of a set – something you cannot do with list or set:
# immutable, hashable – usable as dict keys or set members
point = (3, 4)
allowed = frozenset({"read", "write"})
permissions = {
frozenset({"read"}): "viewer",
frozenset({"read", "write"}): "editor",
}
print(permissions[allowed]) # editor
The advantage is not just hashability. Immutable data cannot change unnoticed while you send it through a pipeline. That is exactly the foundation of side-effect-free work: a function takes data in, returns new data and leaves the input untouched. Especially in larger systems, this separation makes transitions traceable.
That leads directly to the last pitfall I see often: side effects in comprehensions. A comprehension is supposed to produce values, not call print, append to a list or change state:
# don't do this: a comprehension used only for side effects
[print(w) for w in words] # builds a useless list of None values
# do this instead
for w in words:
print(w)
The first version builds a list full of None values on the side that nobody needs. It misuses the comprehension for something the normal loop is there for. Comprehensions are meant for transformations – if no value is produced, the code belongs in a loop.
Making the right choice#
What matters is less knowing every tool than reaching for the right one at the right moment. The following decision tree sums up how I weigh things in practice.
graph TD
A[Sequence to process] --> B{Build a new container?}
B -->|Yes| C[Comprehension<br/>list / dict / set]
B -->|No| D{Transform with a<br/>named function?}
D -->|Yes| E[map]
D -->|No| F{Large or infinite,<br/>save memory?}
F -->|Yes| G[Generator expr<br/>or itertools]
F -->|No| H{Fold to one value?}
H -->|Yes| I[sum / prod / reduce]The thread running through all of this: Python provides the functional tools, but expects you to use them in its own idiom. Comprehensions come before map/filter with lambda because they read better. Generator expressions and itertools step in when data gets large or streams are infinite – lazy and easy on memory. functools contributes partial for specialization, reduce for the rare genuine folds and lru_cache for memoizing pure functions. And the immutable containers tuple and frozenset keep the whole thing free of side effects.
The pitfalls I have shown are all of the same kind: reduce where sum would have been enough; groupby without sorting first; side effects in a comprehension. They do not come from missing knowledge, but from the habit of using a tool beyond its purpose. If you know these three groups of tools and use them with discipline, you write Python that stays compact and is still easy to read – and that is exactly the standard the language sets for functional code.
Further reading#
- Slide deck on this topic: https://slides.mikebild.dev/introduction-python.en.html
- Repository
introduction-python– the examples from this article to try out. - functools – Higher-order functions (Python 3.8)
- itertools – Functions creating iterators (Python 3.8)
- List Comprehensions – Python Tutorial (3.8)
- map and filter – Built-in Functions (Python 3.8)
Revised and translated from German with the help of AI.