Induction in metaclasses

Ionel Cristian Mărieș — Python / OSS enthusiast

blog.ionelmc.ro

github.com/ionelmc

Lots of unwarranted fear ... "it's magic", "they are bad" etc

They present lots of interesting opportunities for:

• Reducing boilerplate.
• Nicer APIs

Especially useful in designs that rely on subclassing (subclasses will use the same metaclass).

They have a tax:

• Introspection issues, unhappy linters.
• Need to understand them to use effectively. Because some interactions are not explicit.

• Objects allow overriding certain object interfaces and operations

  • Like the function call operator:

    >>> class Funky:
    ...     def __call__(self):
    ...         print("Look at me, I work like a function!")
    >>> f = Funky()
    >>> f()
    Look at me, I work like a function!
    

  • Or attribute access:

    >>> class MagicHat:
    ...     def __getattr__(self, name):
    ...         return "Imagine a %a ..." % name
    
    >>> hat = MagicHat()
    >>> hat.rabbit
    "Imagine a 'rabbit' ..."
    

Magic methods are looked up (resolved) on the class, not on the instance. [1]

>>> class Simple:
...     pass
>>> s = Simple()

Doesn't work:

>>> s.__call__ = lambda self: "x"
>>> s()
Traceback (most recent call last):
  ...
TypeError: 'Simple' object is not callable

Aha! It is resolved on the class:

>>> Simple.__call__ = lambda self: "x"
>>> s()
'x'

Instance creation is just a magic method you can override:

>>> class WithConstructor(object):
...     def __new__(cls, value):
...         print("Creating the instance")
...         return super().__new__(cls)
...
...     def __init__(self, value):
...         print("Initialising the instance")
...         self.value = value

>>> WithConstructor(1)
Creating the instance
Initialising the instance
<__main__.WithConstructor object at 0x...>

Note that if __new__ returns a different type of object then a different __init__ would be called (from the other type's class).

A class is just an instance of something else, a metaclass.

So, if a class is just an instance, we can customise the creation process:

>>> class Meta(type):
...     pass
>>> class Complex(metaclass=Meta):
...     pass
>>> Complex()
<__main__.Complex object at 0x...>

Normally, the metaclass should be a subclass of type. More on this later.

Remember that:

• __init__ does not create instances, __new__ does.
• Magic methods are resolved on the metaclass. Complex() is equivalent to Meta.__call__.

__prepare__(mcs, name, bases, **kwargs) - New in Python 3, returns the namespace dictionary

__new__(mcs, name, bases, attrs, **kwargs) - Returns an instance of Meta

__init__(cls, name, bases, attrs, **kwargs) - Runs initialisation code, type's __init__ doesn't do anything

kwargs: only in Python 3, example:

class Class(object, metaclass=Meta, a=1, b=2, c=3):
    pass

__call__ - Returns and instance of Complex (which in turn is instance of Meta). Because magic methods are resolved on the class.

>>> class Simple(Base):
...     foo = 'bar'

Is equivalent to:

>>> Simple = type('Simple', (Base, ), {'foo': 'bar'})

~

>>> class Simple(Base, metaclass=Meta):
...     foo = 'bar'

Is equivalent to:

>>> Simple = Meta('Simple', (Base, ), {'foo': 'bar'})

Normally you inherit from type because it implements all the necessary interface for a well functioning class.

But you can use just as well a plain callable.

And let horrible stuff like this happen:

>>> class NotAnymore(metaclass=print):
...     pass
NotAnymore () {'__qualname__': 'NotAnymore', '__module__': '__main__'}

>>> repr(NotAnymore)
'None'

>>> type(type) is type
True

But how about:

>>> class mutable(type):  # dummy type, so the instance has a __dict__
...     pass              # (mutable in other words)
...
>>> class typeish(type, metaclass=mutable):
...     pass
...
>>> typeish.__class__ = typeish
>>> print(type(typeish) is typeish)
True

Allows users to customize the class creation before the body of the class is executed.

Basically allows you to return a different object as the namespace (instead of a dict).

What can you do with it? Lots of interesting stuff:

• Single dispatch
• Duplicate validators
• Field order aware objects

>>> class StrictDict(dict):
...     def __setitem__(self, name, value):
...         if name in self:
...             raise RuntimeError('You already defined %r!' % name)
...         super().__setitem__(name, value)
...

>>> class StrictMeta(type):
...     def __prepare__(name, bases):
...         return StrictDict()
...

>>> class Strict(metaclass=StrictMeta):
...     a = 1
...     a = 2  # Ooops. Will ever anyone notice this?
Traceback (most recent call last):
  ...
RuntimeError: You already defined 'a'!

Suppose we have this code:

>>> class Int(int):
...     def __repr__(self):
...         return "Int(%d)" % self
...
...     @dispatching.dispatch
...     def __add__(self, other:int):
...         return Int(int.__add__(self, other))
...
...     @__add__.dispatch
...     def __add__(self, other:str):
...         return Int(int.__add__(self, int(other)))


>>> i = Int(5)
>>> i + 1
Int(6)
>>> i + "2"
Int(7)

We can make it seamless using __prepare__:

>>> class Int(int, metaclass=SingleDispatchMeta):
...     def __repr__(self):
...         return "Int(%d)" % self
...
...     def __add__(self, other:int):
...         return Int(int.__add__(self, other))
...
...     def __add__(self, other:str):
...         return Int(int.__add__(self, int(other)))


>>> i = Int(5)
>>> i + 1
Int(6)
>>> i + "2"
Int(7)

>>> import dispatching

>>> class SingleDispatchCollector(dict):
...     def __setitem__(self, name, value):
...         if callable(value):
...             if name in self:
...                 self[name].dispatch(value)
...             else:
...                 super().__setitem__(name,
...                                     dispatching.dispatch(value))
...         else:
...             super().__setitem__(name, value)

>>> class SingleDispatchMeta(type):
...     def __prepare__(name, bases):
...         return SingleDispatchCollector()

• Docstring fixers

• DSLs, Models

• Class or usage validators

• Subclass registry systems

• All sorts of behavior changing

• Warning/deprecation systems

• Automatic function decoration

  
  

A class decorator takes a class as input and, hopefully, returns a class.

The decorator usually returns the same class after making some changes to it (eg: monkey-patching).

Disadvantages of a decorator:

• Lack of flexibility. They are equivalent to a __init__ method in the metaclass.

• When accessing methods through the class or instance they become bound functions (which is not the same thing as the original function).

  Solution is to pull them out of __dict__.

• It becomes boilerplate.

• Doesn't work well with subclassing.

Python's abc

Django

SQLAlchemy

fields

$ pip install fields

>>> from fields import Fields
>>> class Pair(Fields.a.b):
...     pass
...
>>> Pair(1, 2)
Pair(a=1, b=2)

Metaclasses have some disadvantages. What is a good tradeoff?

What's worse, boilerplate or implicit behavior?

Two important things to consider:

• Is the implicit behavior intuitive?
• Does the abstraction leak?

Q: What does the metaclass say to the class?

A: You're __new__ to me.