Python Descriptors, Metaclasses and Generator Coroutines

Quick answer: Every piece of Python syntax is a thin wrapper over a dunder method — a + b calls __add__, for x in y calls __iter__/__next__, with calls __enter__/__exit__. The three mechanisms that unlock the language are descriptors (which is how property, classmethod, staticmethod, and every ORM field in Django and SQLAlchemy work), __init_subclass__ and metaclasses (which let a class police its own subclasses), and the fact that a generator is a coroutine with send() and yield from for two-way data flow. Write these three once and forty lines of framework boilerplate disappear.

Part 14 of our Python series — Module 3, the deep dive. Previous: performance engineering. Start at Module 1.

Syntax is sugar over dunders

This is not trivia. It means that any object implementing the right dunders becomes the thing: an object with __iter__ is iterable, one with __enter__/__exit__ is a context manager, one with __call__ is callable. There is no interface to declare; behaviour is structural.

Attribute access, in the right order

__getattr__ is only called when normal lookup fails, which makes it the safe hook for optional attributes and lazy loading. __getattribute__ intercepts every access and is easy to break into infinite recursion — only override it if __getattr__ genuinely cannot do the job.

class Settings:
    def __init__(self, data):
        self._data = data

    def __getattr__(self, name):        # only on a miss
        try:
            return self._data[name]
        except KeyError as exc:
            raise AttributeError(name) from exc    # AttributeError, not KeyError

class Proxy:
    def __getattribute__(self, name):   # every access: recursion risk lives here
        if name.startswith("_"):
            return object.__getattribute__(self, name)
        return "blocked"

Note the translation in __getattr__: raising AttributeError (not KeyError) is what makes hasattr() and getattr(s, "x", default) behave correctly — a subtle bug you will meet in real code.

Descriptors: the machinery behind property

A descriptor is any object defining __get__, __set__, or __delete__, stored as a class attribute. When you read obj.attr, Python checks type(obj)‘s attributes first and honours them if they are descriptors. That is the whole trick behind the standard library:

class Temperature:
    def __init__(self, celsius):
        self._c = celsius

    @property
    def fahrenheit(self):
        return self._c * 9 / 5 + 32

    @classmethod
    def from_fahrenheit(cls, f):
        return cls((f - 32) * 5 / 9)

    @staticmethod
    def is_freezing(celsius):
        return celsius <= 0

property is a data descriptor (it defines __set__ too, even if only to raise AttributeError), so it takes precedence over the instance __dict__; a plain function is a non-data descriptor, which is why methods can be shadowed by instance attributes. Data descriptors win, non-data descriptors lose — that single rule explains most “why can’t I assign to this attribute?” confusion.

Writing your own descriptor is how frameworks validate fields at the class level. __set_name__ (3.6+) tells the descriptor what it was named, so it can store a private slot without you repeating the string:

class Positive:
    def __set_name__(self, owner, name):
        self.name = "_" + name

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self                       # accessed on the class: return the descriptor
        return getattr(obj, self.name)

    def __set__(self, obj, value):
        if value <= 0:
            raise ValueError(f"{self.name[1:]} must be positive, got {value}")
        setattr(obj, self.name, value)


class Order:
    quantity = Positive()
    price = Positive()

    def __init__(self, quantity, price):
        self.quantity = quantity              # goes through __set__
        self.price = price

Order(0, 1) now raises at construction — validation that cannot be bypassed by assigning later, because the descriptor is on the class, not on the instance. Scale that idea to 80 fields and you have Django’s model system.

__init_subclass__ before metaclasses

When a base class needs to react to its subclasses, reach for __init_subclass__ first: it is a normal classmethod hook, needs no metaclass, and cannot conflict with anyone else’s metaclass.

class Plugin:
    registry = {}

    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        Plugin.registry[cls.__name__] = cls


class Csv(Plugin): ...
class Json(Plugin): ...

print(sorted(Plugin.registry))      # ['Csv', 'Json']

A metaclass is type subclassed — the class of a class — and it is the right tool only when you must reject a class definition, or modify every attribute namespace before the class exists:

class RequireRun(type):
    def __new__(mcls, name, bases, namespace):
        if bases and "run" not in namespace:
            raise TypeError(f"{name} must define run()")
        return super().__new__(mcls, name, bases, namespace)


class Task(metaclass=RequireRun):
    def run(self):
        return "ok"


class Broken(Task):        # TypeError: Broken must define run()
    pass

try:
    class Broken(Task):
        pass
except TypeError as exc:
    print("rejected:", exc)     # Broken must define run()

The cost of metaclasses is real: they are inherited, they collide with other metaclasses (TypeError: metaclass conflict), and ABCMeta, EnumMeta, and ORM bases are already metaclasses — which is why the modern advice is “__init_subclass__ and __set_name__ first, metaclasses when you truly need to refuse a definition”.

Generators are coroutines, not just lazy lists

A generator is a resumable frame. next() resumes it; send(value) resumes it and hands a value back into the yield expression; .throw() raises inside it; .close() finishes it. That two-way channel is a full coroutine protocol, and it predates async/await.

def accumulator():
    total = 0
    while True:
        value = yield total        # receive from send(), emit total
        if value is None:
            break
        total += value

acc = accumulator()
next(acc)              # prime the generator: runs to the first yield
print(acc.send(5))     # 5
print(acc.send(7))     # 12

yield from delegates to a sub-iterable and forwards send/throw/close through it, which is how generator pipelines are composed:

def chain(*iterables):
    for it in iterables:
        yield from it

print(list(chain([1, 2], [3], [4, 5])))     # [1, 2, 3, 4, 5]

Two practical consequences. First, generators are single-use: iterating a generator twice yields nothing the second time, because it is exhausted — convert to a list when you need repeat passes. Second, any function containing yield returns a generator even if the yield is never reached; calling it runs no code until the first next(). That is why a generator that raises on its first line can appear to “do nothing” when you call it.

Context managers: exception behaviour matters

class Guard:
    def __enter__(self):
        return "locked"

    def __exit__(self, exc_type, exc, tb):
        print("cleaning up;", exc_type.__name__ if exc_type else "no error")
        return False        # False: let the exception propagate (True would SWALLOW it)

Returning True from __exit__ silently suppresses the exception — powerful for retry and cleanup wrappers, and a debugging nightmare if it happens by accident. contextlib.contextmanager gives you the same power from a generator function:

import time
from contextlib import contextmanager

@contextmanager
def timer(label):
    start = time.perf_counter()
    try:
        yield                       # everything inside the with-block runs here
    finally:
        print(f"{label}: {time.perf_counter() - start:.3f}s")

with timer("work"):
    sum(range(200_000))

Anything after the yield is __exit__, and a try/finally there guarantees cleanup even when the body raises. For stacking several optional context managers, contextlib.ExitStack avoids nested ladders.

Protocol classes: structural typing on demand

abc.ABC requires inheritance, which is useless for code you do not own. typing.Protocol describes a shape instead:

from typing import Protocol, runtime_checkable

@runtime_checkable
class Reader(Protocol):
    def read(self, size: int = -1) -> bytes: ...

def consume(source: Reader) -> int:
    return len(source.read())

consume now accepts sockets, files, io.BytesIO, and any object with a compatible read, with no shared base class. Be careful with @runtime_checkable: isinstance() then checks only that the attribute exists, not that its signature matches. Static type checkers do the rest of the work; runtime checks are shallow by design.

Complete executable example

# data_model.py -- descriptors, class hooks, context managers, and coroutines
import functools
import time
from contextlib import contextmanager


def section(title):
    print("\n" + title)
    print("=" * len(title))


section("1. A validating descriptor: validation you cannot bypass")


class Positive:
    def __set_name__(self, owner, name):
        self.name = "_" + name

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        return getattr(obj, self.name)

    def __set__(self, obj, value):
        if value <= 0:
            raise ValueError(f"{self.name[1:]} must be positive, got {value}")
        setattr(obj, self.name, value)


class Order:
    quantity = Positive()
    price = Positive()

    def __init__(self, quantity, price):
        self.quantity = quantity
        self.price = price

    @property
    def total(self):
        return self.quantity * self.price


order = Order(3, 2.5)
print("total:", order.total)
print("class access returns:", type(Order.quantity).__name__)
try:
    Order(0, 1)
except ValueError as exc:
    print("rejected at construction:", exc)
try:
    order.quantity = -5
except ValueError as exc:
    print("rejected on assignment:", exc)

section("2. property, classmethod, staticmethod are all descriptors")


class Temperature:
    def __init__(self, celsius):
        self._c = celsius

    @property
    def fahrenheit(self):
        return self._c * 9 / 5 + 32

    @classmethod
    def from_fahrenheit(cls, f):
        return cls((f - 32) * 5 / 9)

    @staticmethod
    def is_freezing(celsius):
        return celsius <= 0


print("212F ->", round(Temperature.from_fahrenheit(212).fahrenheit, 2), "C->F")
print("freezing at -5:", Temperature.is_freezing(-5))
assert isinstance(Temperature.__dict__["fahrenheit"], property)   # it IS a descriptor

section("3. __init_subclass__ builds a registry with no metaclass")


class Plugin:
    registry = {}

    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        Plugin.registry[cls.__name__] = cls


class Csv(Plugin): ...
class Json(Plugin): ...


print("registered:", sorted(Plugin.registry))
assert sorted(Plugin.registry) == ["Csv", "Json"]

section("4. A metaclass can refuse a class definition")


class RequireRun(type):
    def __new__(mcls, name, bases, namespace):
        if bases and "run" not in namespace:
            raise TypeError(f"{name} must define run()")
        return super().__new__(mcls, name, bases, namespace)


class Task(metaclass=RequireRun):
    def run(self):
        return "ok"


print("valid subclass:", Task().run())
try:
    class Broken(Task):
        pass
except TypeError as exc:
    print("rejected at class definition:", exc)

section("5. Context manager, both spellings")


class Guard:
    def __enter__(self):
        return "locked"

    def __exit__(self, exc_type, exc, tb):
        print("  cleanup; error:", exc_type.__name__ if exc_type else "none")
        return False


with Guard() as state:
    print("  inside:", state)


@contextmanager
def timer(label):
    start = time.perf_counter()
    try:
        yield
    finally:
        print(f"  {label}: {time.perf_counter() - start:.3f}s")


with timer("sum of 200k"):
    sum(range(200_000))

section("6. A generator is a coroutine you can send values into")


def accumulator():
    total = 0
    while True:
        value = yield total
        if value is None:
            break
        total += value


acc = accumulator()
next(acc)                      # prime it: run to the first yield
print("send 5 ->", acc.send(5))
print("send 7 ->", acc.send(7))
try:
    acc.send(None)              # None breaks the loop and finishes the generator
    raise AssertionError("expected StopIteration")
except StopIteration:
    print("sending the terminating sentinel closed it: StopIteration")

section("7. yield from delegates")


def chain(*iterables):
    for it in iterables:
        yield from it


print("chain:", list(chain([1, 2], [3], [4, 5])))
assert list(chain([1, 2], [3])) == [1, 2, 3]

section("8. singledispatch: dispatch on type without if/elif")


@functools.singledispatch
def describe(value):
    return "generic object"


@describe.register(int)
def _(value):
    return f"int {value}"


@describe.register(list)
def _(value):
    return f"list of {len(value)}"


print(describe(1), "|", describe([1, 2, 3]), "|", describe(1.5))
assert describe(1) == "int 1" and describe([1]) == "list of 1"

print("\nAll data-model assertions passed.")

Line by line: Positive stores its private name in __set_name__, so the same descriptor class works for quantity and price without arguments; __get__(None, ...) returning self is what lets class-level access print the descriptor instead of an error; the Order(0, 1) call proves validation happens inside __init__ through the descriptor, not in a separate check; the metaclass __new__ inspects the namespace before the class exists, which is the only moment run can still be missing; Guard.__exit__ returns False so a raised exception would propagate rather than vanish; the generator is primed with next() before any send() — skipping that raises TypeError: can't send non-None value to a just-started generator — and sending the terminating sentinel correctly ends the loop and raises StopIteration, exactly as for relies on; and singledispatch shows dispatch chosen by type at call time, which is simpler and faster than an if/elif isinstance chain.

Common mistakes

• Raising KeyError from __getattr__. Always raise AttributeError, or hasattr and default lookups will misbehave.
• Overriding __getattribute__ when __getattr__ would do — and recursing into self.__class__ while doing it.
• Metaclass overuse. __init_subclass__ and __set_name__ cover most real needs with no conflict risk.
• Reusing an exhausted generator. Iterating it twice gives nothing; materialise a list when you need repeat passes.
• Forgetting to prime a coroutine-style generator before send().
• Returning True from __exit__ by accident, which silently eats exceptions.

Key takeaways and challenge

• Syntax maps to dunders; implementing the right ones makes your object be iterable, callable, or a context manager.
• __getattr__ runs only on a miss and is the safe hook; __getattribute__ runs always.
• Descriptors are how property, classmethod, staticmethod, and ORM fields work — data descriptors beat instance attributes.
• __set_name__ plus a descriptor gives you class-level validation; __init_subclass__ gives you registration.
• Metaclasses exist to refuse or transform class definitions; use them last.
• Generators are single-use coroutines: send, throw, close, yield from.
• typing.Protocol gives structural typing without inheritance; runtime_checkable only checks attribute names.

Challenge: build a Validated descriptor family (Positive, NonEmpty, InRange) and a Model base class that uses __init_subclass__ to collect every Validated field it finds. Then give the base class a from_dict classmethod that constructs an instance from a plain dict and raises a single ValidationError listing every field that failed. That is the core of a data-validation library in under 60 lines — and you will understand exactly what your ORM is doing.

Want to go deeper with a mentor? Ampersand Academy runs one-to-one Python training that goes well past the tutorial level.