Advanced Python: Decorators, Magic Methods and Generators
Three building blocks that make code Pythonic: decorators wrap functions, magic methods bind your own objects to Python’s protocols, generators produce lazy sequences.
Python code that you could have written line for line just the same in Java or C# is rarely wrong – it just doesn’t use the language. Three mechanisms make the difference, and none of them is magic: the @ decorator, the dunder method and yield. All three are documented in the language core, each can be rebuilt yourself in a few lines, and each solves a problem you would otherwise solve with copy-paste or with too much memory.
The state here is early 2020, so Python 3.8. All examples run there without extra packages – just with functools, time and the standard library. The prerequisites are Python objects, functions and the usual built-in data types.
Decorators: functions that wrap functions#
Getting into decorators is easier than the @ sugar suggests. In Python, functions are perfectly ordinary objects. You can assign them to a variable, pass them as an argument and return them from another function. A decorator is nothing more than a Higher-Order Function: it takes a function and returns a – usually wrapping – function.
import functools
import time
def timer(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
start = time.perf_counter()
result = func(*args, **kwargs)
elapsed = time.perf_counter() - start
print(f"{func.__name__} took {elapsed:.4f}s")
return result
return wrapper
@timer
def slow_sum(n):
return sum(range(n))
The notation @timer above def slow_sum is exactly syntactic sugar for slow_sum = timer(slow_sum). Nothing else happens. After decorating, the name slow_sum no longer points to the original function but to wrapper. On every call, timing starts first, then the real function runs, and afterward the elapsed time is printed. The *args, **kwargs makes sure the wrapper passes through any signature without having to know anything about the arguments itself.
Here’s the flow when calling a decorated function, drawn out – the wrapper brackets the actual work:
flowchart LR
Call["call slow_sum()"] --> W["wrapper()"]
W -->|"before:<br/>start timer"| F["original func()"]
F -->|"return value"| W
W -->|"after:<br/>print elapsed"| Ret["return result"]functools.wraps isn’t optional#
One detail is missing from hand-written decorators with remarkable regularity: the line @functools.wraps(func). Without it, the decorator still works, but the wrapper doesn’t take on the identity of the original function. slow_sum.__name__ then returns "wrapper" instead of "slow_sum", slow_sum.__doc__ is the wrapper’s docstring, and help(slow_sum) shows the wrong signature.
def timer_broken(func):
def wrapper(*args, **kwargs):
return func(*args, **kwargs)
return wrapper
@timer_broken
def greet(name):
"""Say hello to someone."""
return f"Hello, {name}"
print(greet.__name__) # -> "wrapper"
print(greet.__doc__) # -> None
functools.wraps is a convenience wrapper around functools.update_wrapper. It copies __module__, __name__, __qualname__, __annotations__ and __doc__ from the original function to the wrapper and also sets __wrapped__, so that you can get back to the undecorated function if needed. For debugging, introspection and tools that look at metadata, this is the difference between useful and confusing. That’s why this line goes into every decorator I write.
Decorators with arguments#
Often you want to configure a decorator, for example: “Run the function three times.” For that you need one more level. A decorator with arguments is a function that returns a decorator, which in turn returns the wrapper – three nested functions.
def repeat(times):
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
result = None
for _ in range(times):
result = func(*args, **kwargs)
return result
return wrapper
return decorator
@repeat(times=3)
def ping():
print("ping")
@repeat(times=3) is evaluated first: the call repeat(times=3) returns decorator, and this decorator is then applied to ping. Once you’ve read the structure from the inside out, the pattern opens up – and it shows up in exactly this form in many libraries, for example in routing decorators in web frameworks.
Magic methods: your own objects behaving like built-in ones#
The second building block reaches deeper into the language. Magic methods – also called dunder methods, because of the double underscores – are the way your own classes plug into Python’s protocols. You never call these methods directly. Instead, the interpreter calls them when you put an object into a known context: len(obj) triggers __len__, == triggers __eq__, a for loop triggers __iter__, a with block triggers __enter__ and __exit__.
A few of the most common dunder methods and what triggers them:
__init__– construction when an instance is created__repr__–repr()and the display in the REPL, ideally unambiguous and at besteval-able__str__–str()andprint(); if it’s missing,__repr__steps in__eq__– comparison with==__len__– the calllen(obj), which is also used for truth testing when__bool__is missing__iter__and__next__– iteration inforloops and when unpacking__enter__and__exit__– entering and leaving awithblock__call__– calling an instance like a function,obj()
To make this concrete, I build a class that provides lines from a file. It should have a proper __repr__, be comparable with ==, work as a context manager and be iterable.
class LineStore:
def __init__(self, path):
self.path = path
def __repr__(self):
return f"LineStore({self.path!r})"
def __eq__(self, other):
return isinstance(other, LineStore) and self.path == other.path
def __enter__(self):
self._file = open(self.path, encoding="utf-8")
return self
def __exit__(self, exc_type, exc_value, traceback):
self._file.close()
return False # do not swallow exceptions
def __iter__(self):
return (line.rstrip("\n") for line in self._file)
A LineStore now behaves like a built-in object. In the REPL it shows LineStore('access.log') instead of a meaningless memory address. Two instances with the same path are equal. And the real highlight is in the with block:
with LineStore("access.log") as store:
for line in store:
if "ERROR" in line:
print(line)
The with calls __enter__, binds the return value to store, runs the body and then – guaranteed, even on an exception – calls __exit__. That’s where the file gets closed. That’s exactly the point of a context manager: cleanup you can’t forget. The return value of __exit__ decides how errors are handled. If the method returns True, an exception raised in the block is suppressed; a falsy value like False passes it on. I deliberately return False here, because an error while reading the file shouldn’t silently disappear.
For simple cases, by the way, you don’t have to implement the protocol by hand. contextlib.contextmanager turns a generator function into a context manager, with the yield separating setup and teardown. That leads us straight to the third building block.
Generators: lazy sequences with yield#
The LineStore carries a peculiarity with it: its __iter__ already returns a generator expression. Generators are the third pillar, and for me the most important in practice as soon as data grows larger than memory.
As soon as the keyword yield appears in a function, it’s no longer a normal callable but a generator function. Calling it doesn’t run the body but immediately returns a generator object. The code only runs when you iterate over the object – and it stops at every yield, remembers its state and continues exactly there on the next step.
def stream_lines(path):
with open(path, encoding="utf-8") as fh:
for line in fh: # a file object is itself an iterator
yield line.rstrip("\n")
for line in stream_lines("big.log"):
if line.startswith("2020"):
print(line)
Here only one line is in memory at a time. Whether the file is ten kilobytes or ten gigabytes – memory usage stays constant. That’s exactly the difference from a list. If I wrote fh.readlines(), Python would hold all lines at once. The generator produces them one at a time, on demand.
The same principle exists in compact form as a generator expression – syntactically like a list comprehension, just with round instead of square brackets:
# list comprehension: builds the whole list in memory
lengths_list = [len(line) for line in stream_lines("big.log")]
# generator expression: yields one length at a time
total = sum(len(line) for line in stream_lines("big.log"))
The second variant sums the line lengths without ever creating a complete list. For aggregations over large amounts of data, that’s the standard way. And if you want to chain generators, yield from helps you delegate to another iterable without writing a loop yourself:
def chained(*paths):
for path in paths:
yield from stream_lines(path)
Three pitfalls#
As clear as the three building blocks are, the same stumbling blocks show up just as reliably in practice. These three cost the most time.
The classic is mutable default arguments. If you write an empty list as a default, you don’t get a fresh list on every call but always the same one.
def append_item(item, acc=[]): # trap: shared across calls
acc.append(item)
return acc
print(append_item(1)) # [1]
print(append_item(2)) # [1, 2] – surprise
The default is evaluated exactly once, namely at def. After that, all calls share the same object. The solution is a guard on None:
def append_item(item, acc=None):
if acc is None:
acc = []
acc.append(item)
return acc
The second stumbling block is the forgotten functools.wraps, the case shown above. A decorator without this line works, but it swallows the name, docstring and signature of the decorated function. In small scripts you don’t notice; as soon as someone introspects the function or a framework looks at the metadata, it becomes a puzzle.
The third hits generators: after one pass they’re exhausted, and that’s the flip side of lazy evaluation. A generator remembers its progress and doesn’t reset it. If you run through it twice, you get nothing the second time.
gen = stream_lines("big.log")
first = list(gen) # reads all lines
second = list(gen) # empty – the generator is exhausted
You can iterate over a list as often as you like, over a generator only once. If you need the data several times, deliberately materialize it into a list – or call the generator function again to get a fresh object. Related to this: len() doesn’t work on a generator, because its length isn’t even known without a full pass.
Conclusion#
Decorators, magic methods and generators seem like advanced extras at first contact, but they’re basic tools in everyday work. Decorators cleanly separate recurring behavior – timing, logging, repetition – from the actual logic, as long as you don’t forget functools.wraps. Magic methods connect your own objects to the language, so that len(), ==, for and with just work without the calling code having to know anything about the internals. And generators turn a function into a lazy sequence that delivers large amounts of data element by element instead of holding everything at once.
What connects these three building blocks is a shared idea: Python provides clearly defined protocols, and your own code hooks into them. You don’t write against the language but with it. Once you’ve internalized that, you recognize the patterns everywhere in the standard library – and write code that can not only be run but also read.
Further reading#
- Slide deck on this topic: https://slides.mikebild.dev/introduction-python.en.html
- Repository
introduction-python– chapters13-decorators.md,14-magic-methods.md,15-iter.mdwith the corresponding examples underexamples/(decorators.py,classes.py,functional.py) - Python Data Model / Special Methods: https://docs.python.org/3.8/reference/datamodel.html
- The
withstatement and the context manager protocol: https://docs.python.org/3.8/reference/compound_stmts.html#the-with-statement functoolswithwrapsandupdate_wrapper: https://docs.python.org/3.8/library/functools.htmlcontextlib.contextmanager: https://docs.python.org/3.8/library/contextlib.html- Yield expressions and generators: https://docs.python.org/3.8/reference/expressions.html#yield-expressions
- Tutorial on generators and generator expressions: https://docs.python.org/3.8/tutorial/classes.html#generators
- PEP 318 – Decorators for Functions and Methods: https://www.python.org/dev/peps/pep-0318/
Revised and translated from German with the help of AI.