Induction in metaclasses ======================== .. class:: title ----- .. class:: center .. class:: title Induction in metaclasses .. image:: induction.svg :height: 250px | | | **Ionel Cristian Mărieș** — **Python** / **OSS** enthusiast .. epigraph:: `blog.ionelmc.ro `_ `github.com/ionelmc `_ .footer: `Conference.py #1 `_ — generated with `Darkslide `_ Presenter Notes --------------- * So, "induction"? So it wasn't some typo on the conference website. * The idea was that I'm going to give this talk and induce the knowledge into your brains. * How many have written a metaclass? * Metaclasses: a form of meta-programming - code manipulating code. ----- Why? ==== Lots of unwarranted fear ... "`it's magic`", "`they are bad`" etc They present lots of interesting opportunities for: * Reducing boilerplate. * Nicer APIs Especially useful in designs that rely on subclassing (subclasses will use the same metaclass). They have a `tax`: * Introspection issues, unhappy linters. * Need to understand them to use effectively. Because some interactions are not explicit. Presenter Notes --------------- * I like to think that it's best to understand the implications of using them and be able to reason yourself whether they are good or not for your code. * You can't just copy paste metaclasses and expect to understand how everything works just by looking at the code. * What exactly is the problem with the linters? Even if a linter could statically understand them, it would be very hard to know where to draw the line. * Subclasses: decorators vs metaclasses ---- Few basic things =========================== * Objects allow overriding certain object interfaces and operations * Like the function call operator: .. sourcecode:: pycon >>> class Funky: ... def __call__(self): ... print("Look at me, I work like a function!") >>> f = Funky() >>> f() Look at me, I work like a function! * Or attribute access: .. sourcecode:: pycon >>> class MagicHat: ... def __getattr__(self, name): ... return "Imagine a %a ..." % name >>> hat = MagicHat() >>> hat.rabbit "Imagine a 'rabbit' ..." ---- But there are some caveats ============================= Magic methods are looked up (resolved) on the class, not on the instance. [1] .. sourcecode:: pycon >>> class Simple: ... pass >>> s = Simple() Doesn't work: .. sourcecode:: pycon >>> s.__call__ = lambda self: "x" >>> s() Traceback (most recent call last): ... TypeError: 'Simple' object is not callable Aha! It is resolved on the class: .. sourcecode:: pycon >>> Simple.__call__ = lambda self: "x" >>> s() 'x' .. [1] Exceptions: Python 2 old-style objects and few `special-cased` methods. ---- Constructors and initialisers ================================ Instance creation is just a magic method you can override: .. sourcecode:: pycon >>> class WithConstructor(object): ... def __new__(cls, value): ... print("Creating the instance") ... return super().__new__(cls) ... ... def __init__(self, value): ... print("Initialising the instance") ... self.value = value >>> WithConstructor(1) Creating the instance Initialising the instance <__main__.WithConstructor object at 0x...> .. ``__new__`` is a `special-cased` static method. Note that if ``__new__`` returns a different type of object then a different ``__init__`` would be called (from the other type's class). ---- Types and instances =========================== .. .. sourcecode:: pycon >>> type(s) >>> type(Simple) >>> type(type) .. class:: center | | | | | | | | | | | A class is just an instance of something else, a metaclass. .. image:: simple.png :width: 900px ---- You can have a custom metaclass ================================= So, if a class is just an instance, we can customise the creation process: .. sourcecode:: pycon >>> class Meta(type): ... pass >>> class Complex(metaclass=Meta): ... pass >>> Complex() <__main__.Complex object at 0x...> Normally, the metaclass should be a subclass of ``type``. More on this later. ---- The instantiation dance in detail =================================== .. class:: center .. image:: dimensional.png :width: 600px Remember that: * ``__init__`` does `not` create instances, ``__new__`` does. * Magic methods are resolved on the metaclass. ``Complex()`` is equivalent to ``Meta.__call__``. Presenter notes --------------- * What do I see here: 5 ways to make a singleton :-) * ``__new__`` is a special case, it's not resolved to the ``__new__`` from the metaclass. * "Called to create a new instance of class cls. __new__() is a static method (special-cased so you need not declare it as such) that takes the class of which an instance was requested as its first argument." ---- The full interface =================================== ``__prepare__(mcs, name, bases, **kwargs)`` - New in Python 3, returns the `namespace dictionary` ``__new__(mcs, name, bases, attrs, **kwargs)`` - Returns an instance of Meta ``__init__(cls, name, bases, attrs, **kwargs)`` - Runs initialisation code, `type`'s `__init__` doesn't do anything **kwargs**: only in Python 3, example: | .. sourcecode:: python class Class(object, metaclass=Meta, a=1, b=2, c=3): pass ``__call__`` - Returns and instance of Complex (which in turn is instance of Meta). Because magic methods are resolved on the class. Presenter notes --------------- * An evil metaclass could even change the baseclasses. ---- Syntactic sugar =================================== .. | >>> class Base: ... pass .. sourcecode:: pycon >>> class Simple(Base): ... foo = 'bar' Is equivalent to: .. sourcecode:: pycon >>> Simple = type('Simple', (Base, ), {'foo': 'bar'}) .. class:: hr ~ .. sourcecode:: pycon >>> class Simple(Base, metaclass=Meta): ... foo = 'bar' Is equivalent to: .. sourcecode:: pycon >>> Simple = Meta('Simple', (Base, ), {'foo': 'bar'}) Presenter notes --------------- * Note the name duplication. What could a different name break? `Hint: pickling`. ---- Going back to the `type` of the metaclass ========================================== Normally you inherit from type because it implements all the necessary interface for a well functioning class. But you can use just as well a plain callable. | | | And let horrible stuff like this happen: .. sourcecode:: pycon >>> class NotAnymore(metaclass=print): ... pass NotAnymore () {'__qualname__': 'NotAnymore', '__module__': '__main__'} >>> repr(NotAnymore) 'None' ---- Why can the metaclass be any callable ======================================== .. class:: center .. image:: n-dimensional.png :width: 600px ---- Why can the metaclass be any callable ======================================== .. class:: center .. image:: n-dimensional-aligned.png :width: 850px ---- Useless tricks ============== .. sourcecode:: pycon >>> type(type) is type True But how about: .. sourcecode:: pycon >>> class mutable(type): # dummy type, so the instance has a __dict__ ... pass # (mutable in other words) ... >>> class typeish(type, metaclass=mutable): ... pass ... >>> typeish.__class__ = typeish >>> print(type(typeish) is typeish) True Presenter notes --------------- * Type returns a class according to the given specification (be it an object or parameters for a new class). ---- New in Python 3: ``__prepare__`` ================================ Allows users to customize the class creation before the body of the class is executed. Basically allows you to return a different object as the namespace (instead of a dict). What can you do with it? Lots of interesting stuff: * Single dispatch * Duplicate validators * Field order aware objects ---- Disallow overrides with ``__prepare__`` ============================================= .. sourcecode:: pycon >>> class StrictDict(dict): ... def __setitem__(self, name, value): ... if name in self: ... raise RuntimeError('You already defined %r!' % name) ... super().__setitem__(name, value) ... >>> class StrictMeta(type): ... def __prepare__(name, bases): ... return StrictDict() ... >>> class Strict(metaclass=StrictMeta): ... a = 1 ... a = 2 # Ooops. Will ever anyone notice this? Traceback (most recent call last): ... RuntimeError: You already defined 'a'! Presenter notes --------------- * Defensive programming. ---- Single dispatch with ``__prepare__`` (1/3) ========================================== Suppose we have this code: .. | >>> import dispatching .. sourcecode:: pycon >>> class Int(int): ... def __repr__(self): ... return "Int(%d)" % self ... ... @dispatching.dispatch ... def __add__(self, other:int): ... return Int(int.__add__(self, other)) ... ... @__add__.dispatch ... def __add__(self, other:str): ... return Int(int.__add__(self, int(other))) >>> i = Int(5) >>> i + 1 Int(6) >>> i + "2" Int(7) ---- Single dispatch with ``__prepare__`` (2/3) ========================================== We can make it *seamless* using ``__prepare__``: .. | >>> import dispatching >>> class SingleDispatchCollector(dict): ... def __setitem__(self, name, value): ... if callable(value): ... if name in self: ... self[name].dispatch(value) ... else: ... super().__setitem__(name, ... dispatching.dispatch(value)) ... else: ... super().__setitem__(name, value) >>> class SingleDispatchMeta(type): ... def __prepare__(name, bases): ... return SingleDispatchCollector() .. sourcecode:: pycon >>> class Int(int, metaclass=SingleDispatchMeta): ... def __repr__(self): ... return "Int(%d)" % self ... ... def __add__(self, other:int): ... return Int(int.__add__(self, other)) ... ... def __add__(self, other:str): ... return Int(int.__add__(self, int(other))) >>> i = Int(5) >>> i + 1 Int(6) >>> i + "2" Int(7) ---- Single dispatch with ``__prepare__`` (3/3) ========================================== .. sourcecode:: pycon >>> import dispatching >>> class SingleDispatchCollector(dict): ... def __setitem__(self, name, value): ... if callable(value): ... if name in self: ... self[name].dispatch(value) ... else: ... super().__setitem__(name, ... dispatching.dispatch(value)) ... else: ... super().__setitem__(name, value) >>> class SingleDispatchMeta(type): ... def __prepare__(name, bases): ... return SingleDispatchCollector() ---- What else can you do with metaclasses ======================================= * Docstring fixers * DSLs, Models * Class or usage validators * Subclass registry systems * All sorts of behavior changing * Warning/deprecation systems * Automatic function decoration | ---- Metaclasses vs class decorators ===================================== A class decorator takes a class as input and, hopefully, returns a class. The decorator usually returns the same class after making some changes to it (eg: monkey-patching). Disadvantages of a decorator: * Lack of flexibility. They are equivalent to a ``__init__`` method in the metaclass. * When accessing methods through the class or instance they become `bound functions` (which is not the same thing as the original function). Solution is to pull them out of ``__dict__``. * It becomes boilerplate. * Doesn't work well with subclassing. ---- Known uses ========================== Python's abc Django SQLAlchemy fields .. sourcecode:: sh $ pip install fields .. sourcecode:: pycon >>> from fields import Fields >>> class Pair(Fields.a.b): ... pass ... >>> Pair(1, 2) Pair(a=1, b=2) ---- Where do we draw the line? ==================================== Metaclasses have some disadvantages. What is a good tradeoff? What's worse, boilerplate or implicit behavior? Two important things to consider: * Is the implicit behavior intuitive? * Does the abstraction leak? ---- And last .... ============================ Q: What does the metaclass say to the class? A: You're ``__new__`` to me.