Learning Python (2013)

---

^[72^] And to quote a Python 3.3 error message I just came across: “TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases” (!). This reflects an erroneous use of a module as a superclass, but metaclasses may not be as optional as developers imply—a theme we’ll revisit in the next chapter’s conclusion to this book.

The Metaclass Model

To understand metaclasses, you first need to understand a bit more about Python’s type model and what happens at the end of a class statement. As we’ll see here, the two are intimately related.

Classes Are Instances of type

So far in this book, we’ve done most of our work by making instances of built-in types like lists and strings, as well as instances of classes we code ourselves. As we’ve seen, instances of classes have some state information attributes of their own, but they also inherit behavioral attributes from the classes from which they are made. The same holds true for built-in types; list instances, for example, have values of their own, but they inherit methods from the list type.

While we can get a lot done with such instance objects, Python’s type model turns out to be a bit richer than I’ve formally described. Really, there’s a hole in the model we’ve seen thus far: if instances are created from classes, what is it that creates our classes? It turns out that classes are instances of something, too:

§ In Python 3.X, user-defined class objects are instances of the object named type, which is itself a class.

§ In Python 2.X, new-style classes inherit from object, which is a subclass of type; classic classes are instances of type and are not created from a class.

We explored the notion of types in Chapter 9 and the relationship of classes to types in Chapter 32, but let’s review the basics here so we can see how they apply to metaclasses.

Recall that the type built-in returns the type of any object (which is itself an object) when called with a single argument. For built-in types like lists, the type of the instance is the built-in list type, but the type of the list type is the type type itself—the type object at the top of the hierarchy creates specific types, and specific types create instances. You can see this for yourself at the interactive prompt. In Python 3.X, for example, the type of a list instance is the list class, and the type of the list class is the type class:

C:\code> py −3 # In 3.X:

>>> type([]), type(type([])) # List instance is created from list class

(<class 'list'>, <class 'type'>) # List class is created from type class

>>> type(list), type(type) # Same, but with type names

(<class 'type'>, <class 'type'>) # Type of type is type: top of hierarchy

As we learned when studying new-style class changes in Chapter 32, the same is generally true in Python 2.X, but types are not quite the same as classes—type is a unique kind of built-in object that caps the type hierarchy and is used to construct types:

C:\code> py −2

>>> type([]), type(type([])) # In 2.X, type is a bit different

(<type 'list'>, <type 'type'>)

>>> type(list), type(type)

(<type 'type'>, <type 'type'>)

As it happens, the type/instance relationship holds true for user-defined classes as well: instances are created from classes, and classes are created from type. In Python 3.X, though, the notion of a “type” is merged with the notion of a “class.” In fact, the two are essentially synonyms—classes are types, and types are classes. That is:

§ Types are defined by classes that derive from type.

§ User-defined classes are instances of type classes.

§ User-defined classes are types that generate instances of their own.

As we saw earlier, this equivalence affects code that tests the type of instances: the type of an instance is the class from which it was generated. It also has implications for the way that classes are created that turn out to be the key to this chapter’s subject. Because classes are normally created from a root type class by default, most programmers don’t need to think about this type/class equivalence. However, it opens up new possibilities for customizing both classes and their instances.

For example, all user-defined classes in 3.X (and new-style classes in 2.X) are instances of the type class, and instance objects are instances of their classes; in fact, classes now have a __class__ that links to type, just as an instance has a __class__ that links to the class from which it was made:

C:\code> py −3

>>> class C: pass # 3.X class object (new-style)

>>> X = C() # Class instance object

>>> type(X) # Instance is instance of class

<class '__main__.C'>

>>> X.__class__ # Instance's class

<class '__main__.C'>

>>> type(C) # Class is instance of type

<class 'type'>

>>> C.__class__ # Class's class is type

<class 'type'>

Notice especially the last two lines here—classes are instances of the type class, just as normal instances are instances of a user-defined class. This works the same for both built-ins and user-defined class types in 3.X. In fact, classes are not really a separate concept at all: they are simply user-defined types, and type itself is defined by a class.

In Python 2.X, things work similarly for new-style classes derived from object, because this enables 3.X class behavior (as we’ve seen, 3.X adds object to the __bases__ superclass tuple of top-level root classes automatically to qualify them as new-style):

C:\code> py −2

>>> class C(object): pass # In 2.X new-style classes,

>>> X = C() # classes have a class too

>>> type(X)

<class '__main__.C'>

>>> X.__class__

<class '__main__.C'>

>>> type(C)

<type 'type'>

>>> C.__class__

<type 'type'>

Classic classes in 2.X are a bit different, though—because they reflect the original class model in older Pythons, they do not have a __class__ link, and like built-in types in 2.X they are instances of type, not a type class (I’ve shortened some of the hex addresses in object displays in this chapter for clarity):

C:\code> py −2

>>> class C: pass # In 2.X classic classes,

>>> X = C() # classes have no class themselves

>>> type(X)

<type 'instance'>

>>> X.__class__

<class __main__.C at 0x005F85A0>

>>> type(C)

<type 'classobj'>

>>> C.__class__

AttributeError: class C has no attribute '__class__'

Metaclasses Are Subclasses of Type

Why would we care that classes are instances of a type class in 3.X? It turns out that this is the hook that allows us to code metaclasses. Because the notion of type is the same as class today, we can subclass type with normal object-oriented techniques and class syntax to customize it. And because classes are really instances of the type class, creating classes from customized subclasses of type allows us to implement custom kinds of classes. In full detail, this all works out quite naturally—in 3.X, and in 2.X new-style classes:

§ type is a class that generates user-defined classes.

§ Metaclasses are subclasses of the type class.

§ Class objects are instances of the type class, or a subclass thereof.

§ Instance objects are generated from a class.

In other words, to control the way classes are created and augment their behavior, all we need to do is specify that a user-defined class be created from a user-defined metaclass instead of the normal type class.

Notice that this type instance relationship is not quite the same as normal inheritance. User-defined classes may also have superclasses from which they and their instances inherit attributes as usual. As we’ve seen, inheritance superclasses are listed in parentheses in the class statement and show up in a class’s __bases__ tuple. The type from which a class is created, though, and of which it is an instance, is a different relationship. Inheritance searches instance and class namespace dictionaries, but classes may also acquire behavior from their type that is not exposed to the normal inheritance search.

To lay the groundwork for understanding this distinction, the next section describes the procedure Python follows to implement this instance-of type relationship.

Class Statement Protocol

Subclassing the type class to customize it is really only half of the magic behind metaclasses. We still need to somehow route a class’s creation to the metaclass, instead of the default type. To fully understand how this is arranged, we also need to know how class statements do their business.

We’ve already learned that when Python reaches a class statement, it runs its nested block of code to create its attributes—all the names assigned at the top level of the nested code block generate attributes in the resulting class object. These names are usually method functions created by nested defs, but they can also be arbitrary attributes assigned to create class data shared by all instances.

Technically speaking, Python follows a standard protocol to make this happen: at the end of a class statement, and after running all its nested code in a namespace dictionary corresponding to the class’s local scope, Python calls the type object to create the class object like this:

class = type(classname, superclasses, attributedict)

The type object in turn defines a __call__ operator overloading method that runs two other methods when the type object is called:

type.__new__(typeclass, classname, superclasses, attributedict)

type.__init__(class, classname, superclasses, attributedict)

The __new__ method creates and returns the new class object, and then the __init__ method initializes the newly created object. As we’ll see in a moment, these are the hooks that metaclass subclasses of type generally use to customize classes.

For example, given a class definition like the following for Spam:

class Eggs: ... # Inherited names here

class Spam(Eggs): # Inherits from Eggs

data = 1 # Class data attribute

def meth(self, arg): # Class method attribute

return self.data + arg

Python will internally run the nested code block to create two attributes of the class (data and meth), and then call the type object to generate the class object at the end of the class statement:

Spam = type('Spam', (Eggs,), {'data': 1, 'meth': meth, '__module__': '__main__'})

In fact, you can call type this way yourself to create a class dynamically—albeit here with a fabricated method function and empty superclasses tuple (Python adds object automatically in both 3.X and 2.X):

>>> x = type('Spam', (), {'data': 1, 'meth': (lambda x, y: x.data + y)})

>>> i = x()

>>> x, i

(<class '__main__.Spam'>, <__main__.Spam object at 0x029E7780>)

>>> i.data, i.meth(2)

(1, 3)

The class produced is exactly like that you’d get from running a class statement:

>>> x.__bases__

(<class 'object'>,)

>>> [(a, v) for (a, v) in x.__dict__.items() if not a.startswith('__')]

[('data', 1), ('meth', <function <lambda> at 0x0297A158>)]

Because this type call is made automatically at the end of the class statement, though, it’s an ideal hook for augmenting or otherwise processing a class. The trick lies in replacing the default type with a custom subclass that will intercept this call. The next section shows how.

Declaring Metaclasses

As we’ve just seen, classes are created by the type class by default. To tell Python to create a class with a custom metaclass instead, you simply need to declare a metaclass to intercept the normal instance creation call in a user-defined class. How you do so depends on which Python version you are using.

Declaration in 3.X

In Python 3.X, list the desired metaclass as a keyword argument in the class header:

class Spam(metaclass=Meta): # 3.X version (only)

Inheritance superclasses can be listed in the header as well. In the following, for example, the new class Spam inherits from superclass Eggs, but is also an instance of and is created by metaclass Meta:

class Spam(Eggs, metaclass=Meta): # Normal supers OK: must list first

In this form, superclasses must be listed before the metaclass; in effect, the ordering rules used for keyword arguments in function calls apply here.

Declaration in 2.X

We can get the same effect in Python 2.X, but we must specify the metaclass differently—using a class attribute instead of a keyword argument:

class Spam(object): # 2.X version (only), object optional?

__metaclass__ = Meta

class Spam(Eggs, object): # Normal supers OK: object suggested

__metaclass__ = Meta

Technically, some classes in 2.X do not have to derive from object explicitly to make use of metaclasses. The generalized metaclass dispatch mechanism was added at the same time as new-style classes, but is not itself bound to them. It does, however, produce them—in the presence of a__metaclass__ declaration, 2.X makes the resulting class new-style automatically, adding object to its __bases__ sequence. In the absence of this declaration, 2.X simply uses the classic class creator as the metaclass default. Because of this, some classes in 2.X require only the__metaclass__ attribute.

On the other hand, notice that metaclasses imply that your class will be new-style in 2.X even without an explicit object. They’ll behave somewhat differently as outlined in Chapter 32, and as we’ll see ahead 2.X may require that they or their superclasses derive from object explicitly, because a new-style class cannot have only classic superclasses in this context. Given this, deriving from object doesn’t hurt as a sort of warning about the class’s nature, and may be required to avoid potential problems.

Also in 2.X, a module level __metaclass__ global variable is available to link all classes in the module to a metaclass. This is no longer supported in 3.X, as it was intended as a temporary measure to make it easier to default to new-style classes without deriving every class from object. Python 3.X also ignores the 2.X class attribute, and the 3.X keyword form is a syntax error in 2.X, so there is no simple portability route. Apart from differing syntax, though, metaclass declaration in 2.X and 3.X has the same effect, which we turn to next.

Metaclass Dispatch in Both 3.X and 2.X

When a specific metaclass is declared per the prior sections’ syntax, the call to create the class object run at the end of the class statement is modified to invoke the metaclass instead of the type default:

class = Meta(classname, superclasses, attributedict)

And because the metaclass is a subclass of type, the type class’s __call__ delegates the calls to create and initialize the new class object to the metaclass, if it defines custom versions of these methods:

Meta.__new__(Meta, classname, superclasses, attributedict)

Meta.__init__(class, classname, superclasses, attributedict)

To demonstrate, here’s the prior section’s example again, augmented with a 3.X metaclass specification:

class Spam(Eggs, metaclass=Meta): # Inherits from Eggs, instance of Meta

data = 1 # Class data attribute

def meth(self, arg): # Class method attribute

return self.data + arg

At the end of this class statement, Python internally runs the following to create the class object—again, a call you could make manually too, but automatically run by Python’s class machinery:

Spam = Meta('Spam', (Eggs,), {'data': 1, 'meth': meth, '__module__': '__main__'})

If the metaclass defines its own versions of __new__ or __init__, they will be invoked in turn during this call by the inherited type class’s __call__ method, to create and initialize the new class. The net effect is to automatically run methods the metaclass provides, as part of the class construction process. The next section shows how we might go about coding this final piece of the metaclass puzzle.

NOTE

This chapter uses Python 3.X metaclass keyword argument syntax, not the 2.X class attribute. 2.X readers will need to translate, but version neutrality is not straightforward here—3.X doesn’t recognize the attribute and 2.X doesn’t allow keyword syntax—and listing examples twice doesn’t address portability (or chapter size!).

Coding Metaclasses

So far, we’ve seen how Python routes class creation calls to a metaclass, if one is specified and provided. How, though, do we actually code a metaclass that customizes type?

It turns out that you already know most of the story—metaclasses are coded with normal Python class statements and semantics. By definition, they are simply classes that inherit from type. Their only substantial distinctions are that Python calls them automatically at the end of a classstatement, and that they must adhere to the interface expected by the type superclass.

A Basic Metaclass

Perhaps the simplest metaclass you can code is simply a subclass of type with a __new__ method that creates the class object by running the default version in type. A metaclass __new__ like this is run by the __call__ method inherited from type; it typically performs whatever customization is required and calls the type superclass’s __new__ method to create and return the new class object:

class Meta(type):

def __new__(meta, classname, supers, classdict):

# Run by inherited type.__call__

return type.__new__(meta, classname, supers, classdict)

This metaclass doesn’t really do anything (we might as well let the default type class create the class), but it demonstrates the way a metaclass taps into the metaclass hook to customize—because the metaclass is called at the end of a class statement, and because the type object’s__call__ dispatches to the __new__ and __init__ methods, code we provide in these methods can manage all the classes created from the metaclass.

Here’s our example in action again, with prints added to the metaclass and the file at large to trace (again, some filenames are implied by later command-lines in this chapter):

class MetaOne(type):

def __new__(meta, classname, supers, classdict):

print('In MetaOne.new:', meta, classname, supers, classdict, sep='\n...')

return type.__new__(meta, classname, supers, classdict)

class Eggs:

pass

print('making class')

class Spam(Eggs, metaclass=MetaOne): # Inherits from Eggs, instance of MetaOne

data = 1 # Class data attribute

def meth(self, arg): # Class method attribute

return self.data + arg

print('making instance')

X = Spam()

print('data:', X.data, X.meth(2))

Here, Spam inherits from Eggs and is an instance of MetaOne, but X is an instance of and inherits from Spam. When this code is run with Python 3.X, notice how the metaclass is invoked at the end of the class statement, before we ever make an instance—metaclasses are for processingclasses, and classes are for processing normal instances:

c:\code> py −3 metaclass1.py

making class

In MetaOne.new:

...<class '__main__.MetaOne'>

...Spam

...(<class '__main__.Eggs'>,)

...{'data': 1, 'meth': <function Spam.meth at 0x02A191E0>, '__module__': '__main__'}

making instance

data: 1 3

Presentation note: I’m truncating addresses and omitting some irrelevant built-in __X__ names in namespace dictionaries in this chapter for brevity, and as noted earlier am forgoing 2.X portability due to differing declaration syntax. To run in 2.X, use the class attribute form, and change print operations as desired. This example works in 2.X with the following modifications, in the file metaclass1-2x.py; notice that either Eggs or Spam must be derived from object explicitly, or else 2.X issues a warning because new-style class can’t have only classic bases here—when in doubt, use object in 2.X metaclasses clients:

from __future__ import print_function # To run the same in 2.X (only)

class Eggs(object): # One of the "object" optional

class Spam(Eggs, object):

__metaclass__ = MetaOne

Customizing Construction and Initialization

Metaclasses can also tap into the __init__ protocol invoked by the type object’s __call__. In general, __new__ creates and returns the class object, and __init__ initializes the already created class passed in as an argument. Metaclasses can use either or both hooks to manage the class at creation time:

class MetaTwo(type):

def __new__(meta, classname, supers, classdict):

print('In MetaTwo.new: ', classname, supers, classdict, sep='\n...')

return type.__new__(meta, classname, supers, classdict)

def __init__(Class, classname, supers, classdict):

print('In MetaTwo.init:', classname, supers, classdict, sep='\n...')

print('...init class object:', list(Class.__dict__.keys()))

class Eggs:

pass

print('making class')

class Spam(Eggs, metaclass=MetaTwo): # Inherits from Eggs, instance of MetaTwo

data = 1 # Class data attribute

def meth(self, arg): # Class method attribute

return self.data + arg

print('making instance')

X = Spam()

print('data:', X.data, X.meth(2))

In this case, the class initialization method is run after the class construction method, but both run at the end of the class statement before any instances are made. Conversely, an __init__ in Spam would run at instance creation time, and is not affected or run by the metaclass’s__init__:

c:\code> py −3 metaclass2.py

making class

In MetaTwo.new:

...Spam

...(<class '__main__.Eggs'>,)

...{'data': 1, 'meth': <function Spam.meth at 0x02967268>, '__module__': '__main__'}

In MetaTwo.init:

...Spam

...(<class '__main__.Eggs'>,)

...{'data': 1, 'meth': <function Spam.meth at 0x02967268>, '__module__': '__main__'}

...init class object: ['__qualname__', 'data', '__module__', 'meth', '__doc__']

making instance

data: 1 3

Other Metaclass Coding Techniques

Although redefining the type superclass’s __new__ and __init__ methods is the most common way to insert logic into the class object creation process with the metaclass hook, other schemes are possible.

Using simple factory functions

For example, metaclasses need not really be classes at all. As we’ve learned, the class statement issues a simple call to create a class at the conclusion of its processing. Because of this, any callable object can in principle be used as a metaclass, provided it accepts the arguments passed and returns an object compatible with the intended class. In fact, a simple object factory function may serve just as well as a type subclass:

# A simple function can serve as a metaclass too

def MetaFunc(classname, supers, classdict):

print('In MetaFunc: ', classname, supers, classdict, sep='\n...')

return type(classname, supers, classdict)

class Eggs:

pass

print('making class')

class Spam(Eggs, metaclass=MetaFunc): # Run simple function at end

data = 1 # Function returns class

def meth(self, arg):

return self.data + arg

print('making instance')

X = Spam()

print('data:', X.data, X.meth(2))

When run, the function is called at the end of the declaring class statement, and it returns the expected new class object. The function is simply catching the call that the type object’s __call__ normally intercepts by default:

c:\code> py −3 metaclass3.py

making class

In MetaFunc:

...Spam

...(<class '__main__.Eggs'>,)

...{'data': 1, 'meth': <function Spam.meth at 0x029471E0>, '__module__': '__main__'}

making instance

data: 1 3

Overloading class creation calls with normal classes

Because normal class instances can respond to call operations with operator overloading, they can serve in some metaclass roles too, much like the preceding function. The output of the following is similar to the prior class-based versions, but it’s based on a simple class—one that doesn’t inherit from type at all, and provides a __call__ for its instances that catches the metaclass call using normal operator overloading. Note that __new__ and __init__ must have different names here, or else they will run when the Meta instance is created, not when it is later called in the role of metaclass:

# A normal class instance can serve as a metaclass too

class MetaObj:

def __call__(self, classname, supers, classdict):

print('In MetaObj.call: ', classname, supers, classdict, sep='\n...')

Class = self.__New__(classname, supers, classdict)

self.__Init__(Class, classname, supers, classdict)

return Class

def __New__(self, classname, supers, classdict):

print('In MetaObj.new: ', classname, supers, classdict, sep='\n...')

return type(classname, supers, classdict)

def __Init__(self, Class, classname, supers, classdict):

print('In MetaObj.init:', classname, supers, classdict, sep='\n...')

print('...init class object:', list(Class.__dict__.keys()))

class Eggs:

pass

print('making class')

class Spam(Eggs, metaclass=MetaObj()): # MetaObj is normal class instance

data = 1 # Called at end of statement

def meth(self, arg):

return self.data + arg

print('making instance')

X = Spam()

print('data:', X.data, X.meth(2))

When run, the three methods are dispatched via the normal instance’s __call__ inherited from its normal class, but without any dependence on type dispatch mechanics or semantics:

c:\code> py −3 metaclass4.py

making class

In MetaObj.call:

...Spam

...(<class '__main__.Eggs'>,)

...{'data': 1, 'meth': <function Spam.meth at 0x029492F0>, '__module__': '__main__'}

In MetaObj.new:

...Spam

...(<class '__main__.Eggs'>,)

...{'data': 1, 'meth': <function Spam.meth at 0x029492F0>, '__module__': '__main__'}

In MetaObj.init:

...Spam

...(<class '__main__.Eggs'>,)

...{'data': 1, 'meth': <function Spam.meth at 0x029492F0>, '__module__': '__main__'}

...init class object: ['__module__', '__doc__', 'data', '__qualname__', 'meth']

making instance

data: 1 3

In fact, we can use normal superclass inheritance to acquire the call interceptor in this coding model—the superclass here is serving essentially the same role as type, at least in terms of metaclass dispatch:

# Instances inherit from classes and their supers normally

class SuperMetaObj:

def __call__(self, classname, supers, classdict):

print('In SuperMetaObj.call: ', classname, supers, classdict, sep='\n...')

Class = self.__New__(classname, supers, classdict)

self.__Init__(Class, classname, supers, classdict)

return Class

class SubMetaObj(SuperMetaObj):

def __New__(self, classname, supers, classdict):

print('In SubMetaObj.new: ', classname, supers, classdict, sep='\n...')

return type(classname, supers, classdict)

def __Init__(self, Class, classname, supers, classdict):

print('In SubMetaObj.init:', classname, supers, classdict, sep='\n...')

print('...init class object:', list(Class.__dict__.keys()))

class Spam(Eggs, metaclass=SubMetaObj()): # Invoke Sub instance via Super.__call__

...rest of file unchanged...

c:\code> py −3 metaclass4-super.py

making class

In SuperMetaObj.call:

...as before...

In SubMetaObj.new:

...as before...

In SubMetaObj.init:

...as before...

making instance

data: 1 3

Although such alternative forms work, most metaclasses get their work done by redefining the type superclass’s __new__ and __init__; in practice, this is usually as much control as is required, and it’s often simpler than other schemes. Moreover, metaclasses have access to additional tools, such as class methods we’ll explore ahead, which can influence class behavior more directly than some other schemes.

Still, we’ll see later that a simple callable-based metaclass can often work much like a class decorator, which allows the metaclasses to manage instances as well as classes. First, though, the next section presents an example drawn from the Python “Twilight Zone” to introduce metaclass name resolution concepts.

Overloading class creation calls with metaclasses

Since they participate in normal OOP mechanics, it’s also possible for metaclasses to catch the creation call at the end of a class statement directly, by redefining the type object’s __call__. The redefinitions of both __new__ and __call__ must be careful to call back to their defaults intype if they mean to make a class in the end, and __call__ must invoke type to kick off the other two here:

# Classes can catch calls too (but built-ins look in metas, not supers!)

class SuperMeta(type):

def __call__(meta, classname, supers, classdict):

print('In SuperMeta.call: ', classname, supers, classdict, sep='\n...')

return type.__call__(meta, classname, supers, classdict)

def __init__(Class, classname, supers, classdict):

print('In SuperMeta init:', classname, supers, classdict, sep='\n...')

print('...init class object:', list(Class.__dict__.keys()))

print('making metaclass')

class SubMeta(type, metaclass=SuperMeta):

def __new__(meta, classname, supers, classdict):

print('In SubMeta.new: ', classname, supers, classdict, sep='\n...')

return type.__new__(meta, classname, supers, classdict)

def __init__(Class, classname, supers, classdict):

print('In SubMeta init:', classname, supers, classdict, sep='\n...')

print('...init class object:', list(Class.__dict__.keys()))

class Eggs:

pass

print('making class')

class Spam(Eggs, metaclass=SubMeta): # Invoke SubMeta, via SuperMeta.__call__

data = 1

def meth(self, arg):

return self.data + arg

print('making instance')

X = Spam()

print('data:', X.data, X.meth(2))

This code has some oddities I’ll explain in a moment. When run, though, all three redefined methods run in turn for Spam as in the prior section. This is again essentially what the type object does by default, but there’s an additional metaclass call for the metaclass subclass (metasubclass?):

c:\code> py −3 metaclass5.py

making metaclass

In SuperMeta init:

...SubMeta

...(<class 'type'>,)

...{'__init__': <function SubMeta.__init__ at 0x028F92F0>, ...}

...init class object: ['__doc__', '__module__', '__new__', '__init__, ...]

making class

In SuperMeta.call:

...Spam

...(<class '__main__.Eggs'>,)

...{'data': 1, 'meth': <function Spam.meth at 0x028F9378>, '__module__': '__main__'}

In SubMeta.new:

...Spam

...(<class '__main__.Eggs'>,)

...{'data': 1, 'meth': <function Spam.meth at 0x028F9378>, '__module__': '__main__'}

In SubMeta init:

...Spam

...(<class '__main__.Eggs'>,)

...{'data': 1, 'meth': <function Spam.meth at 0x028F9378>, '__module__': '__main__'}

...init class object: ['__qualname__', '__module__', '__doc__', 'data', 'meth']

making instance

data: 1 3

This example is complicated by the fact that it overrides a method invoked by a built-in operation—in this case, the call run automatically to create a class. Metaclasses are used to create class objects, but only generate instances of themselves when called in a metaclass role. Because of this, name lookup with metaclasses may be somewhat different than what we are accustomed to. The __call__ method, for example, is looked up by built-ins in the class (a.k.a. type) of an object; for metaclasses, this means the metaclass of a metaclass!

As we’ll see ahead, metaclasses also inherit names from other metaclasses normally, but as for normal classes, this seems to apply to explicit name fetches only, not to the implicit lookup of names for built-in operations such as calls. The latter appears to look in the metaclass’s class, available in its __class__ link—which is either the default type or a metaclass. This is the same built-ins routing issue we’ve seen so often in this book for normal class instances. The metaclass in SubMeta is required to set this link, though this also kicks off a metaclass construction step for the metaclass itself.

Trace the invocations in the output. SuperMeta’s __call__ method is not run for the call to SuperMeta when making SubMeta (this goes to type instead), but is run for the SubMeta call when making Spam. Inheriting normally from SuperMeta does not suffice to catch SubMeta calls, and for reasons we’ll see later is actually the wrong thing to do for operator overloading methods: SuperMeta’s __call__ is then acquired by Spam, causing Spam instance creation calls to fail before any instance is ever created. Subtle but true!

Here’s an illustration of the issue in simpler terms—a normal superclass is skipped for built-ins, but not for explicit fetches and calls, the latter relying on normal attribute name inheritance:

class SuperMeta(type):

def __call__(meta, classname, supers, classdict): # By name, not built-in

print('In SuperMeta.call:', classname)

return type.__call__(meta, classname, supers, classdict)

class SubMeta(SuperMeta): # Created by type default

def __init__(Class, classname, supers, classdict): # Overrides type.__init__

print('In SubMeta init:', classname)

print(SubMeta.__class__)

print([n.__name__ for n in SubMeta.__mro__])

print()

print(SubMeta.__call__) # Not a data descriptor if found by name

print()

SubMeta.__call__(SubMeta, 'xxx', (), {}) # Explicit calls work: class inheritance

print()

SubMeta('yyy', (), {}) # But implicit built-in calls do not: type

c:\code> py −3 metaclass5b.py

<class 'type'>

['SubMeta', 'SuperMeta', 'type', 'object']

<function SuperMeta.__call__ at 0x029B9158>

In SuperMeta.call: xxx

In SubMeta init: xxx

In SubMeta init: yyy

Of course, this specific example is a special case: catching a built-in run on a metaclass, a likely rare usage related to __call__ here. But it underscores a core asymmetry and apparent inconsistency: normal attribute inheritance is not fully used for built-in dispatch—for both instances and classes.

To truly understand this example’s subtleties, though, we need to get more formal about what metaclasses mean for Python name resolution in general.

Inheritance and Instance

Because metaclasses are specified in similar ways to inheritance superclasses, they can be a bit confusing at first glance. A few key points should help summarize and clarify the model:

Metaclasses inherit from the type class (usually)

Although they have a special role, metaclasses are coded with class statements and follow the usual OOP model in Python. For example, as subclasses of type, they can redefine the type object’s methods, overriding and customizing them as needed. Metaclasses typically redefine thetype class’s __new__ and __init__ to customize class creation and initialization. Although it’s less common, they can also redefine __call__ if they wish to catch the end-of-class creation call directly (albeit with the complexities we saw in the prior section), and can even be simple functions or other callables that return arbitrary objects, instead of type subclasses.

Metaclass declarations are inherited by subclasses

The metaclass=M declaration in a user-defined class is inherited by the class’s normal subclasses, too, so the metaclass will run for the construction of each class that inherits this specification in a superclass inheritance chain.

Metaclass attributes are not inherited by class instances

Metaclass declarations specify an instance relationship, which is not the same as what we’ve called inheritance thus far. Because classes are instances of metaclasses, the behavior defined in a metaclass applies to the class, but not the class’s later instances. Instances obtain behavior from their classes and superclasses, but not from any metaclasses. Technically, attribute inheritance for normal instances usually searches only the __dict__ dictionaries of the instance, its class, and all its superclasses; metaclasses are not included in inheritance lookup for normal instances.

Metaclass attributes are acquired by classes

By contrast, classes do acquire methods of their metaclasses by virtue of the instance relationship. This is a source of class behavior that processes classes themselves. Technically, classes acquire metaclass attributes through the class’s __class__ link just as normal instances acquire names from their class, but inheritance via __dict__ search is attempted first: when the same name is available to a class in both a metaclass and a superclass, the superclass (inheritance) version is used instead of that on a metaclass (instance). The class’s __class__, however, is not followed for its own instances: metaclass attributes are made available to their instance classes, but not to instances of those instance classes (and see the earlier reference to Dr. Seuss...).

This may be easier to understand in code than in prose. To illustrate all these points, consider the following example:

# File metainstance.py

class MetaOne(type):

def __new__(meta, classname, supers, classdict): # Redefine type method

print('In MetaOne.new:', classname)

return type.__new__(meta, classname, supers, classdict)

def toast(self):

return 'toast'

class Super(metaclass=MetaOne): # Metaclass inherited by subs too

def spam(self): # MetaOne run twice for two classes

return 'spam'

class Sub(Super): # Superclass: inheritance versus instance

def eggs(self): # Classes inherit from superclasses

return 'eggs' # But not from metaclasses

When this code is run (as a script or module), the metaclass handles construction of both client classes, and instances inherit class attributes but not metaclass attributes:

>>> from metainstance import * # Runs class statements: metaclass run twice

In MetaOne.new: Super

In MetaOne.new: Sub

>>> X = Sub() # Normal instance of user-defined class

>>> X.eggs() # Inherited from Sub

'eggs'

>>> X.spam() # Inherited from Super

'spam'

>>> X.toast() # Not inherited from metaclass

AttributeError: 'Sub' object has no attribute 'toast'

By contrast, classes both inherit names from their superclasses, and acquire names from their metaclass (which in this example is itself inherited from a superclass):

>>> Sub.eggs(X) # Own method

'eggs'

>>> Sub.spam(X) # Inherited from Super

'spam'

>>> Sub.toast() # Acquired from metaclass

'toast'

>>> Sub.toast(X) # Not a normal class method

TypeError: toast() takes 1 positional argument but 2 were given

Notice how the last of the preceding calls fails when we pass in an instance, because the name resolves to a metaclass method, not a normal class method. In fact, both the object you fetch a name from and its source become crucial here. Methods acquired from metaclasses are bound to the subject class, while methods from normal classes are unbound if fetched through the class but bound when fetched through the instance:

>>> Sub.toast

<bound method MetaOne.toast of <class 'metainstance.Sub'>>

>>> Sub.spam

<function Super.spam at 0x0298A2F0>

>>> X.spam

<bound method Sub.spam of <metainstance.Sub object at 0x02987438>>

We’ve studied the last two of these rules before in Chapter 31’s bound method coverage; the first is new, but reminiscent of class methods. To understand why this works the way it does, we need to explore the metaclass instance relationship further.

Metaclass Versus Superclass

In even simpler terms, watch what happens in the following: as an instance of the A metaclass type, class B acquires A’s attribute, but this attribute is not made available for inheritance by B’s own instances—the acquisition of names by metaclass instances is distinct from the normal inheritance used for class instances:

>>> class A(type): attr = 1

>>> class B(metaclass=A): pass # B is meta instance and acquires meta attr

>>> I = B() # I inherits from class but not meta!

>>> B.attr

1

>>> I.attr

AttributeError: 'B' object has no attribute 'attr'

>>> 'attr' in B.__dict__, 'attr' in A.__dict__

(False, True)

By contrast, if A morphs from metaclass to superclass, then names inherited from an A superclass become available to later instances of B, and are located by searching namespace dictionaries in classes in the tree—that is, by checking the __dict__ of objects in the method resolution order (MRO), much like the mapattrs example we coded back in Chapter 32:

>>> class A: attr = 1

>>> class B(A): pass # I inherits from class and supers

>>> I = B()

>>> B.attr

1

>>> I.attr

1

>>> 'attr' in B.__dict__, 'attr' in A.__dict__

(False, True)

This is why metaclasses often do their work by manipulating a new class’s namespace dictionary, if they wish to influence the behavior of later instance objects—instances will see names in a class, but not its metaclass. Watch what happens, though, if the same name is available in bothattribute sources—the inheritance name is used instead of instance acquisition:

>>> class M(type): attr = 1

>>> class A: attr = 2

>>> class B(A, metaclass=M): pass # Supers have precedence over metas

>>> I = B()

>>> B.attr, I.attr

(2, 2)

>>> 'attr' in B.__dict__, 'attr' in A.__dict__, 'attr' in M.__dict__

(False, True, True)

This is true regardless of the relative height of the inheritance and instance sources—Python checks the __dict__ of each class on the MRO (inheritance), before falling back on metaclass acquisition (instance):

>>> class M(type): attr = 1

>>> class A: attr = 2

>>> class B(A): pass

>>> class C(B, metaclass=M): pass # Super two levels above meta: still wins

>>> I = C()

>>> I.attr, C.attr

(2, 2)

>>> [x.__name__ for x in C.__mro__] # See Chapter 32 for all things MRO

['C', 'B', 'A', 'object']

In fact, classes acquire metaclass attributes through their __class__ link, in the same way that normal instances inherit from classes through their __class__, which makes sense, given that classes are also instances of metaclasses. The chief distinction is that instance inheritance does not follow a class’s __class__, but instead restricts its scope to the __dict__ of each class in a tree per the MRO—following __bases__ at each class only, and using only the instance’s __class__ link once:

>>> I.__class__ # Followed by inheritance: instance's class

<class '__main__.C'>

>>> C.__bases__ # Followed by inheritance: class's supers

(<class '__main__.B'>,)

>>> C.__class__ # Followed by instance acquisition: metaclass

<class '__main__.M'>

>>> C.__class__.attr # Another way to get to metaclass attributes

1

If you study this, you’ll probably notice a nearly glaring symmetry here, which leads us to the next section.

Inheritance: The Full Story

As it turns out, instance inheritance works in similar ways, whether the “instance” is created from a normal class, or is a class created from a metaclass subclass of type—a single attribute search rule, which fosters the grander and parallel notion of metaclass inheritance hierarchies. To illustrate the basics of this conceptual merger, in the following, the instance inherits from all its classes; the class inherits from both classes and metaclasses; and metaclasses inherit from higher metaclasses (supermetaclasses?):

>>> class M1(type): attr1 = 1 # Metaclass inheritance tree

>>> class M2(M1): attr2 = 2 # Gets __bases__, __class__, __mro__

>>> class C1: attr3 = 3 # Superclass inheritance tree

>>> class C2(C1,metaclass=M2): attr4 = 4 # Gets __bases__, __class__, __mro__

>>> I = C2() # I gets __class__ but not others

>>> I.attr3, I.attr4 # Instance inherits from super tree

(3, 4)

>>> C2.attr1, C2.attr2, C2.attr3, C2.attr4 # Class gets names from both trees!

(1, 2, 3, 4)

>>> M2.attr1, M2.attr2 # Metaclass inherits names too!

(1, 2)

Both inheritance paths—class and metaclass—employ the same links, though not recursively: instances do not inherit their class’s metaclass names, but may request them explicitly:

>>> I.__class__ # Links followed at instance with no __bases__

<class '__main__.C2'>

>>> C2.__bases__

(<class '__main__.C1'>,)

>>> C2.__class__ # Links followed at class after __bases__

<class '__main__.M2'>

>>> M2.__bases__

(<class '__main__.M1'>,)

>>> I.__class__.attr1 # Route inheritance to the class's meta tree

1

>>> I.attr1 # Though class's __class__ not followed normally

AttributeError: 'C2' object has no attribute 'attr1'

>>> M2.__class__ # Both trees have MROs and instance links

<class 'type'>

>>> [x.__name__ for x in C2.__mro__] # __bases__ tree from I.__class__

['C2', 'C1', 'object']

>>> [x.__name__ for x in M2.__mro__] # __bases__ tree from C2.__class__

['M2', 'M1', 'type', 'object']

If you care about metaclasses, or must use code that does, study these examples, and then study them again. In effect, inheritance follows __bases__ before following a single __class__; normal instances have no __bases__; and classes have both—whether normal or metaclass. In fact, understanding this example is important to Python name resolution in general, as the next section explains.

Python’s inheritance algorithm: The simple version

Now that we know about metaclass acquisition, we’re finally able to formalize the inheritance rules that they augment. Technically, inheritance deploys two distinct but similar lookup routines, and is based on MROs. Because __bases__ are used to construct the __mro__ ordering at class creation time, and because a class’s __mro__ includes itself, the prior section’s generalization is the same as the following—a first-cut definition of Python’s new-style inheritance algorithm:

To look up an explicit attribute name:

1. From an instance I, search the instance, then its class, and then all its superclasses, using:

a. The __dict__ of the instance I

b. The __dict__ of all classes on the __mro__ found at I’s __class__, from left to right

2. From a class C, search the class, then all its superclasses, and then its metaclasses tree, using:

a. The __dict__ of all classes on the __mro__ found at C itself, from left to right

b. The __dict__ of all metaclasses on the __mro__ found at C’s __class__, from left to right

3. In both rule 1 and 2, give precedence to data descriptors located in step b sources (see ahead).

4. In both rule 1 and 2, skip step a and begin the search at step b for built-in operations (see ahead).

The first two steps are followed for normal, explicit attribute fetch only. There are exceptions for both built-ins and descriptors, both of which we’ll clarify in a moment. In addition, a __getattr__ or __getattribute__ may also be used for missing or all names, respectively, perChapter 38.

Most programmers need only be aware of the first of these rules, and perhaps the first step of the second—which taken together correspond to 2.X classic class inheritance. There’s an extra acquisition step added for metaclasses (2b), but it’s essentially the same as others—a fairly subtle equivalence to be sure, but metaclass acquisition is not as novel as it may seem. In fact, it’s just one component of the larger model.

The descriptors special case

At least that’s the normal—and simplistic—case. I listed step 3 in the prior section specially, because it doesn’t apply to most code, and complicates the algorithm substantially. It turns out, though, that inheritance also has a special case interaction with Chapter 38’s attribute descriptors. In short, some descriptors known as data descriptors—those that define __set__ methods to intercept assignments—are given precedence, such that their names override other inheritance sources.

This exception serves some practical roles. For example, it is used to ensure that the special __class__ and __dict__ attributes cannot be redefined by the same names in an instance’s own __dict__:

>>> class C: pass # Inheritance special case #1...

>>> I = C() # Class data descriptors have precedence

>>> I.__class__, I.__dict__

(<class '__main__.C'>, {})

>>> I.__dict__['name'] = 'bob' # Dynamic data in the instance

>>> I.__dict__['__class__'] = 'spam' # Assign keys, not attributes

>>> I.__dict__['__dict__'] = {}

>>> I.name # I.name comes from I.__dict__ as usual

'bob' # But I.__class__ and I.__dict__ do not!

>>> I.__class__, I.__dict__

(<class '__main__.C'>, {'__class__': 'spam', '__dict__': {}, 'name': 'bob'})

This data descriptor exception is tested before the preceding two inheritance rules as a preliminary step, may be more important to Python implementers than Python programmers, and can be reasonably ignored by most application code in any event—that is, unless you code data descriptors of your own, which follow the same inheritance special case precedence rule:

>>> class D:

def __get__(self, instance, owner): print('__get__')

def __set__(self, instance, value): print('__set__')

>>> class C: d = D() # Data descriptor attribute

>>> I = C()

>>> I.d # Inherited data descriptor access

__get__

>>> I.d = 1

__set__

>>> I.__dict__['d'] = 'spam' # Define same name in instance namespace dict

>>> I.d # But doesn't hide data descriptor in class!

__get__

Conversely, if this descriptor did not define a __set__, the name in the instance’s dictionary would hide the name in its class instead, per normal inheritance:

>>> class D:

def __get__(self, instance, owner): print('__get__')

>>> class C: d = D()

>>> I = C()

>>> I.d # Inherited nondata descriptor access

__get__

>>> I.__dict__['d'] = 'spam' # Hides class names per normal inheritance rules

>>> I.d

'spam'

In both cases, Python automatically runs the descriptor’s __get__ when it’s found by inheritance, rather than returning the descriptor object itself—part of the attribute magic we met earlier in the book. The special status afforded to data descriptors, however, also modifies the meaning of attribute inheritance, and thus the meaning of names in your code.

Python’s inheritance algorithm: The somewhat-more-complete version

With both the data descriptor special case and general descriptor invocation factored in with class and metaclass trees, Python’s full new-style inheritance algorithm can be stated as follows—a complex procedure, which assumes knowledge of descriptors, metaclasses, and MROs, but is the final arbiter of attribute names nonetheless (in the following, items are attempted in sequence either as numbered, or per their left-to-right order in “or” conjunctions):

To look up an explicit attribute name:

1. From an instance I, search the instance, its class, and its superclasses, as follows:

a. Search the __dict__ of all classes on the __mro__ found at I’s __class__

b. If a data descriptor was found in step a, call it and exit

c. Else, return a value in the __dict__ of the instance I

d. Else, call a nondata descriptor or return a value found in step a

2. From a class C, search the class, its superclasses, and its metaclasses tree, as follows:

a. Search the __dict__ of all metaclasses on the __mro__ found at C’s __class__

b. If a data descriptor was found in step a, call it and exit

c. Else, call a descriptor or return a value in the __dict__ of a class on C’s own __mro__

d. Else, call a nondata descriptor or return a value found in step a

3. In both rule 1 and 2, built-in operations essentially use just step a sources (see ahead)

Note here again that this applies to normal, explicit attribute fetch only. The implicit lookup of method names for built-ins doesn’t follow these rules, and essentially uses just step a sources in both cases, as the next section will demonstrate.

On top of all this, method __getattr__ may be run if defined when an attribute is not found, and method __getattribute__ may be run for every attribute fetch, though they are special-case extensions to the name lookup model. See Chapter 38 for more on these tools and descriptors.

Assignment inheritance

Also note that the prior section defines inheritance in terms of attribute reference (lookup), but parts of it apply to attribute assignment as well. As we’ve learned, assignment normally changes attributes in the subject object itself, but inheritance is also invoked on assignment to test first for some of Chapter 38’s attribute management tools, including descriptors and properties. When present, such tools intercept attribute assignment, and may route it arbitrarily.

For example, when an attribute assignment is run for new-style classes, a data descriptor with a __set__ method is acquired from a class by inheritance using the MRO, and has precedence over the normal storage model. In terms of the prior section’s rules:

§ When applied to an instance, such assignments essentially follow steps a through c of rule 1, searching the instance’s class tree, though step b calls __set__ instead of __get__, and step c stops and stores in the instance instead of attempting a fetch.

§ When applied to a class, such assignments run the same procedure on the class’s metaclass tree: roughly the same as rule 2, but step c stops and stores in the class.

Because descriptors are also the basis for other advanced attribute tools such as properties and slots, this inheritance pre-check on assignment is utilized in multiple contexts. The net effect is that descriptors are treated as an inheritance special case in new-style classes, for both reference andassignment.

The built-ins special case

At least that’s almost the full story. As we’ve seen, built-ins don’t follow these rules. Instances and classes may both be skipped for built-in operations only, as a special case that differs from normal or explicit name inheritance. Because this is a context-specific divergence, it’s easier to demonstrate in code than to weave into a single algorithm. In the following, str is the built-in, __str__ is its explicit name equivalent, and the instance is skipped for the built-in only:

>>> class C: # Inheritance special case #2...

attr = 1 # Built-ins skip a step

def __str__(self): return('class')

>>> I = C()

>>> I.__str__(), str(I) # Both from class if not in instance

('class', 'class')

>>> I.__str__ = lambda: 'instance'

>>> I.__str__(), str(I) # Explicit=>instance, built-in=>class!

('instance', 'class')

>>> I.attr # Asymmetric with normal or explicit names

1

>>> I.attr = 2; I.attr

2

As we saw in metaclass5.py earlier, the same holds true for classes: explicit names start at the class, but built-ins start at the class’s class, which is its metaclass, and defaults to type:

>>> class D(type):

def __str__(self): return('D class')

>>> class C(D):

pass

>>> C.__str__(C), str(C) # Explicit=>super, built-in=>metaclass!

('D class', "<class '__main__.C'>")

>>> class C(D):

def __str__(self): return('C class')

>>> C.__str__(C), str(C) # Explicit=>class, built-in=>metaclass!

('C class', "<class '__main__.C'>")

>>> class C(metaclass=D):

def __str__(self): return('C class')

>>> C.__str__(C), str(C) # Built-in=>user-defined metaclass

('C class', 'D class')

In fact, it can sometimes be nontrivial to know where a name comes from in this model, since all classes also inherit from object—including the default type metaclass. In the following’s explicit call, C appears to get a default __str__ from object instead of the metaclass, per the first source of class inheritance (the class’s own MRO); by contrast, the built-in skips ahead to the metaclass as before:

>>> class C(metaclass=D):

pass

>>> C.__str__(C), str(C) # Explicit=>object, built-in=>metaclass

("<class '__main__.C'>", 'D class')

>>> C.__str__

<slot wrapper '__str__' of 'object' objects>

>>> for k in (C, C.__class__, type): print([x.__name__ for x in k.__mro__])

['C', 'object']

['D', 'type', 'object']

['type', 'object']

All of which leads us to this book’s final import this quote—a tenet that seems to conflict with the status given to descriptors and built-ins in the attribute inheritance mechanism of new-style classes:

Special cases aren’t special enough to break the rules.

Some practical needs warrant exceptions, of course. We’ll forgo rationales here, but you should carefully consider the implications of an object-oriented language that applies inheritance—its foundational operation—in such an uneven and inconsistent fashion. At a minimum, this should underscore the importance of keeping your code simple, to avoid making it dependent on such convoluted rules. As always, your code’s users and maintainers will be glad you did.

For more fidelity on this story, see Python’s internal implementation of inheritance—a complete saga chronicled today in its object.c and typeobject.c, the former for normal instances, and the latter for classes. Delving into internals shouldn’t be required to use Python, of course, but it’s the ultimate source of truth in a complex and evolving system, and sometimes the best you’ll find. This is especially true in boundary cases born of accrued exceptions. For our purposes here, let’s move on to the last bit of metaclass magic.

Metaclass Methods

Just as important as the inheritance of names, methods in metaclasses process their instance classes—not the normal instance objects we’ve known as “self,” but classes themselves. This makes them similar in spirit and form to the class methods we studied in Chapter 32, though they again are available in the metaclasses instance realm only, not to normal instance inheritance. The failure at the end of the following, for example, stems from the explicit name inheritance rules of the prior section:

>>> class A(type):

def x(cls): print('ax', cls) # A metaclass (instances=classes)

def y(cls): print('ay', cls) # y is overridden by instance B

>>> class B(metaclass=A):

def y(self): print('by', self) # A normal class (normal instances)

def z(self): print('bz', self) # Namespace dict holds y and z

>>> B.x # x acquired from metaclass

<bound method A.x of <class '__main__.B'>>

>>> B.y # y and z defined in class itself

<function B.y at 0x0295F1E0>

>>> B.z

<function B.z at 0x0295F378>

>>> B.x() # Metaclass method call: gets cls

ax <class '__main__.B'>

>>> I = B() # Instance method calls: get inst

>>> I.y()

by <__main__.B object at 0x02963BE0>

>>> I.z()

bz <__main__.B object at 0x02963BE0>

>>> I.x() # Instance doesn't see meta names

AttributeError: 'B' object has no attribute 'x'

Metaclass Methods Versus Class Methods

Though they differ in inheritance visibility, much like class methods, metaclass methods are designed to manage class-level data. In fact, their roles can overlap—much as metaclasses do in general with class decorators—but metaclass methods are not accessible except through the class, and do not require an explicit classmethod class-level data declaration in order to be bound with the class. In other words, metaclass methods can be thought of as implicit class methods, with limited visibility:

>>> class A(type):

def a(cls): # Metaclass method: gets class

cls.x = cls.y + cls.z

>>> class B(metaclass=A):

y, z = 11, 22

@classmethod # Class method: gets class

def b(cls):

return cls.x

>>> B.a() # Call metaclass method; visible to class only

>>> B.x # Creates class data on B, accessible to normal instances

33

>>> I = B()

>>> I.x, I.y, I.z

(33, 11, 22)

>>> I.b() # Class method: sends class, not instance; visible to instance

33

>>> I.a() # Metaclass methods: accessible through class only

AttributeError: 'B' object has no attribute 'a'

Operator Overloading in Metaclass Methods

Just like normal classes, metaclasses may also employ operator overloading to make built-in operations applicable to their instance classes. The __getitem__ indexing method in the following metaclass, for example, is a metaclass method designed to process classes themselves—the classes that are instances of the metaclass, not those classes’ own later instances. In fact, per the inheritance algorithms sketched earlier, normal class instances don’t inherit names acquired via the metaclass instance relationship at all, though they can access names present on their own classes:

>>> class A(type):

def __getitem__(cls, i): # Meta method for processing classes:

return cls.data[i] # Built-ins skip class, use meta

# Explicit names search class + meta

>>> class B(metaclass=A): # Data descriptors in meta used first

data = 'spam'

>>> B[0] # Metaclass instance names: visible to class only

's'

>>> B.__getitem__

<bound method A.__getitem__ of <class '__main__.B'>>

>>> I = B()

>>> I.data, B.data # Normal inheritance names: visible to instance and class

('spam', 'spam')

>>> I[0]

TypeError: 'B' object does not support indexing

It’s possible to define a __getattr__ on a metaclass too, but it can be used to process its instance classes only, not their normal instances—as usual, it’s not even acquired by a class’s instances:

>>> class A(type):

def __getattr__(cls, name): # Acquired by class B getitem

return getattr(cls.data, name) # But not run same by built-ins

>>> class B(metaclass=A):

data = 'spam'

>>> B.upper()

'SPAM'

>>> B.upper

<built-in method upper of str object at 0x029E7420>

>>> B.__getattr__

<bound method A.__getattr__ of <class '__main__.B'>>

>>> I = B()

>>> I.upper

AttributeError: 'B' object has no attribute 'upper'

>>> I.__getattr__

AttributeError: 'B' object has no attribute '__getattr__'

Moving the __getattr__ to a metaclass doesn’t help with its built-in interception shortcomings, though. In the following continuation, explicit attributes are routed to the metaclass’s __getattr__, but built-ins are not, despite that fact the indexing is routed to a metaclass’s __getitem__in the first example of the section—strongly suggesting that new-style __getattr__ is a special case of a special case, and further recommending code simplicity that avoids dependence on such boundary cases:

>>> B.data = [1, 2, 3]

>>> B.append(4) # Explicit normal names routed to meta's getattr

>>> B.data

[1, 2, 3, 4]

>>> B.__getitem__(0) # Explicit special names routed to meta's gettarr

1

>>> B[0] # But built-ins skip meta's gettatr too?!

TypeError: 'A' object does not support indexing

As you can probably tell, metaclasses are interesting to explore, but it’s easy to lose track of their big picture. In the interest of space, we’ll omit additional fine points here. For the purposes of this chapter, it’s more important to show why you’d care to use such a tool in the first place. Let’s move on to some larger examples to sample the roles of metaclasses in action. As we’ll find, like so many tools in Python, metaclasses are first and foremost about easing maintenance work by eliminating redundancy.

Example: Adding Methods to Classes

In this and the following section, we’re going to study examples of two common use cases for metaclasses: adding methods to a class, and decorating all methods automatically. These are just two of the many metaclass roles, which unfortunately will consume the space we have left for this chapter; again, you should consult the Web for more advanced applications. These examples are representative of metaclasses in action, though, and they suffice to illustrate their application.

Moreover, both give us an opportunity to contrast class decorators and metaclasses—our first example compares metaclass- and decorator-based implementations of class augmentation and instance wrapping, and the second applies a decorator with a metaclass first and then with another decorator. As you’ll see, the two tools are often interchangeable, and even complementary.

Manual Augmentation

Earlier in this chapter, we looked at skeleton code that augmented classes by adding methods to them in various ways. As we saw, simple class-based inheritance suffices if the extra methods are statically known when the class is coded. Composition via object embedding can often achieve the same effect too. For more dynamic scenarios, though, other techniques are sometimes required—helper functions can usually suffice, but metaclasses provide an explicit structure and minimize the maintenance costs of changes in the future.

Let’s put these ideas in action here with working code. Consider the following example of manual class augmentation—it adds two methods to two classes, after they have been created:

# Extend manually - adding new methods to classes

class Client1:

def __init__(self, value):

self.value = value

def spam(self):

return self.value * 2

class Client2:

value = 'ni?'

def eggsfunc(obj):

return obj.value * 4

def hamfunc(obj, value):

return value + 'ham'

Client1.eggs = eggsfunc

Client1.ham = hamfunc

Client2.eggs = eggsfunc

Client2.ham = hamfunc

X = Client1('Ni!')

print(X.spam())

print(X.eggs())

print(X.ham('bacon'))

Y = Client2()

print(Y.eggs())

print(Y.ham('bacon'))

This works because methods can always be assigned to a class after it’s been created, as long as the methods assigned are functions with an extra first argument to receive the subject self instance—this argument can be used to access state information accessible from the class instance, even though the function is defined independently of the class.

When this code runs, we receive the output of a method coded inside the first class, as well as the two methods added to the classes after the fact:

c:\code> py −3 extend-manual.py

Ni!Ni!

Ni!Ni!Ni!Ni!

baconham

ni?ni?ni?ni?

baconham

This scheme works well in isolated cases and can be used to fill out a class arbitrarily at runtime. It suffers from a potentially major downside, though: we have to repeat the augmentation code for every class that needs these methods. In our case, it wasn’t too onerous to add the two methods to both classes, but in more complex scenarios this approach can be time-consuming and error-prone. If we ever forget to do this consistently, or we ever need to change the augmentation, we can run into problems.

Metaclass-Based Augmentation

Although manual augmentation works, in larger programs it would be better if we could apply such changes to an entire set of classes automatically. That way, we’d avoid the chance of the augmentation being botched for any given class. Moreover, coding the augmentation in a single location better supports future changes—all classes in the set will pick up changes automatically.

One way to meet this goal is to use metaclasses. If we code the augmentation in a metaclass, every class that declares that metaclass will be augmented uniformly and correctly and will automatically pick up any changes made in the future. The following code demonstrates:

# Extend with a metaclass - supports future changes better

def eggsfunc(obj):

return obj.value * 4

def hamfunc(obj, value):

return value + 'ham'

class Extender(type):

def __new__(meta, classname, supers, classdict):

classdict['eggs'] = eggsfunc

classdict['ham'] = hamfunc

return type.__new__(meta, classname, supers, classdict)

class Client1(metaclass=Extender):

def __init__(self, value):

self.value = value

def spam(self):

return self.value * 2

class Client2(metaclass=Extender):

value = 'ni?'

X = Client1('Ni!')

print(X.spam())

print(X.eggs())

print(X.ham('bacon'))

Y = Client2()

print(Y.eggs())

print(Y.ham('bacon'))

This time, both of the client classes are extended with the new methods because they are instances of a metaclass that performs the augmentation. When run, this version’s output is the same as before—we haven’t changed what the code does, we’ve just refactored it to encapsulate the augmentation more cleanly:

c:\code> py −3 extend-meta.py

Ni!Ni!

Ni!Ni!Ni!Ni!

baconham

ni?ni?ni?ni?

baconham

Notice that the metaclass in this example still performs a fairly static task: adding two known methods to every class that declares it. In fact, if all we need to do is always add the same two methods to a set of classes, we might as well code them in a normal superclass and inherit in subclasses. In practice, though, the metaclass structure supports much more dynamic behavior. For instance, the subject class might also be configured based upon arbitrary logic at runtime:

# Can also configure class based on runtime tests

class MetaExtend(type):

def __new__(meta, classname, supers, classdict):

if sometest():

classdict['eggs'] = eggsfunc1

else:

classdict['eggs'] = eggsfunc2

if someothertest():

classdict['ham'] = hamfunc

else:

classdict['ham'] = lambda *args: 'Not supported'

return type.__new__(meta, classname, supers, classdict)

Metaclasses Versus Class Decorators: Round 2

Keep in mind again that the prior chapter’s class decorators often overlap with this chapter’s metaclasses in terms of functionality. This derives from the fact that:

§ Class decorators rebind class names to the result of a function at the end of a class statement, after the new class has been created.

§ Metaclasses work by routing class object creation through an object at the end of a class statement, in order to create the new class.

Although these are slightly different models, in practice they can often achieve the same goals, albeit in different ways. As you’ve now seen, class decorators correspond directly to metaclass __init__ methods called to initialize newly created classes. Decorators have no direct analog to the metaclass __new__ (called to make classes in the first place) or to metaclass methods (used to process instance classes), but many or most use cases for these tools do not require these extra steps.

Because of this, both tools in principle can be used to manage both instances of a class and the class itself. In practice, though, metaclasses incur extra steps to manage instances, and decorators incur extra steps to create new classes. Hence, while their roles often overlap, metaclasses are probably best used for class object management. Let’s translate these ideas to code.

Decorator-based augmentation

In pure augmentation cases, decorators can often stand in for metaclasses. For example, the prior section’s metaclass example, which adds methods to a class on creation, can also be coded as a class decorator; in this mode, decorators roughly correspond to the __init__ method of metaclasses, since the class object has already been created by the time the decorator is invoked. Also as for metaclasses, the original class type is retained, since no wrapper object layer is inserted. The output of the following, file extend-deco.py, is the same as that of the prior metaclass code:

# Extend with a decorator: same as providing __init__ in a metaclass

def eggsfunc(obj):

return obj.value * 4

def hamfunc(obj, value):

return value + 'ham'

def Extender(aClass):

aClass.eggs = eggsfunc # Manages class, not instance

aClass.ham = hamfunc # Equiv to metaclass __init__

return aClass

@Extender

class Client1: # Client1 = Extender(Client1)

def __init__(self, value): # Rebound at end of class stmt

self.value = value

def spam(self):

return self.value * 2

@Extender

class Client2:

value = 'ni?'

X = Client1('Ni!') # X is a Client1 instance

print(X.spam())

print(X.eggs())

print(X.ham('bacon'))

Y = Client2()

print(Y.eggs())

print(Y.ham('bacon'))

In other words, at least in certain cases, decorators can manage classes as easily as metaclasses. The converse isn’t quite so straightforward, though; metaclasses can be used to manage instances, but only with a certain amount of extra magic. The next section demonstrates.

Managing instances instead of classes

As we’ve just seen, class decorators can often serve the same class-management role as metaclasses. Metaclasses can often serve the same instance-management role as decorators, too, but this requires extra code and may seem less natural. That is:

§ Class decorators can manage both classes and instances, but don’t create classes normally.

§ Metaclasses can manage both classes and instances, but instances require extra work.

That said, certain applications may be better coded in one or the other. For example, consider the following class decorator example from the prior chapter; it’s used to print a trace message whenever any normally named attribute of a class instance is fetched:

# Class decorator to trace external instance attribute fetches

def Tracer(aClass): # On @ decorator

class Wrapper:

def __init__(self, *args, **kargs): # On instance creation

self.wrapped = aClass(*args, **kargs) # Use enclosing scope name

def __getattr__(self, attrname):

print('Trace:', attrname) # Catches all but .wrapped

return getattr(self.wrapped, attrname) # Delegate to wrapped object

return Wrapper

@Tracer

class Person: # Person = Tracer(Person)

def __init__(self, name, hours, rate): # Wrapper remembers Person

self.name = name

self.hours = hours

self.rate = rate # In-method fetch not traced

def pay(self):

return self.hours * self.rate

bob = Person('Bob', 40, 50) # bob is really a Wrapper

print(bob.name) # Wrapper embeds a Person

print(bob.pay()) # Triggers __getattr__

When this code is run, the decorator uses class name rebinding to wrap instance objects in an object that produces the trace lines in the following output:

c:\code> py −3 manage-inst-deco.py

Trace: name

Bob

Trace: pay

2000

Although it’s possible for a metaclass to achieve the same effect, it seems less straightforward conceptually. Metaclasses are designed explicitly to manage class object creation, and they have an interface tailored for this purpose. To use a metaclass just to manage instances, we have to also take on responsibility for creating the class too—an extra step if normal class creation would otherwise suffice. The following metaclass, in file manage-inst-meta.py, has the same effect as the prior decorator:

# Manage instances like the prior example, but with a metaclass

def Tracer(classname, supers, classdict): # On class creation call

aClass = type(classname, supers, classdict) # Make client class

class Wrapper:

def __init__(self, *args, **kargs): # On instance creation

self.wrapped = aClass(*args, **kargs)

def __getattr__(self, attrname):

print('Trace:', attrname) # Catches all but .wrapped

return getattr(self.wrapped, attrname) # Delegate to wrapped object

return Wrapper

class Person(metaclass=Tracer): # Make Person with Tracer

def __init__(self, name, hours, rate): # Wrapper remembers Person

self.name = name

self.hours = hours

self.rate = rate # In-method fetch not traced

def pay(self):

return self.hours * self.rate

bob = Person('Bob', 40, 50) # bob is really a Wrapper

print(bob.name) # Wrapper embeds a Person

print(bob.pay()) # Triggers __getattr__

This works, but it relies on two tricks. First, it must use a simple function instead of a class, because type subclasses must adhere to object creation protocols. Second, it must manually create the subject class by calling type manually; it needs to return an instance wrapper, but metaclasses are also responsible for creating and returning the subject class. Really, we’re using the metaclass protocol to imitate decorators in this example, rather than vice versa; because both run at the conclusion of a class statement, in many roles they are just variations on a theme. This metaclass version produces the same output as the decorator when run live:

c:\code> py −3 manage-inst-meta.py

Trace: name

Bob

Trace: pay

2000

You should study both versions of these examples for yourself to weigh their tradeoffs. In general, though, metaclasses are probably best suited to class management, due to their design; class decorators can manage either instances or classes, though they may not be the best option for more advanced metaclass roles that we don’t have space to cover in this book. See the Web for more metaclass examples, but keep in mind that some are more appropriate than others (and some of their authors may know less of Python than you do!).

Metaclass and class decorator equivalence?

The preceding section illustrated that metaclasses incur an extra step to create the class when used in instance management roles, and hence can’t quite subsume decorators in all use cases. But what about the inverse—are decorators a replacement for metaclasses?

Just in case this chapter has not yet managed to make your head explode, consider the following metaclass coding alternative too—a class decorator that returns a metaclass instance:

# A decorator can call a metaclass, though not vice versa without type()

>>> class Metaclass(type):

def __new__(meta, clsname, supers, attrdict):

print('In M.__new__:')

print([clsname, supers, list(attrdict.keys())])

return type.__new__(meta, clsname, supers, attrdict)

>>> def decorator(cls):

return Metaclass(cls.__name__, cls.__bases__, dict(cls.__dict__))

>>> class A:

x = 1

>>> @decorator

class B(A):

y = 2

def m(self): return self.x + self.y

In M.__new__:

['B', (<class '__main__.A'>,), ['__qualname__', '__doc__', 'm', 'y', '__module__']]

>>> B.x, B.y

(1, 2)

>>> I = B()

>>> I.x, I.y, I.m()

(1, 2, 3)

This nearly proves the equivalence of the two tools, but really just in terms of dispatch at class construction time. Again, decorators essentially serve the same role as metaclass __init__ methods. Because this decorator returns a metaclass instance, metaclasses—or at least their typesuperclass—are still assumed here. Moreover, this winds up triggering an additional metaclass call after the class is created, and isn’t an ideal scheme in real code—you might as well move this metaclass to the first creation step:

>>> class B(A, metaclass=Metaclass): ... # Same effect, but makes just one class

Still, there is some tool redundancy here, and decorator and metaclass roles often overlap in practice. And although decorators don’t directly support the notion of class-level methods in metaclasses discussed earlier, methods and state in proxy objects created by decorators can achieve similar effects, though for space we’ll leave this last observation in the suggested explorations column.

The inverse may not seem applicable—a metaclass can’t generally defer to a nonmetaclass decorator, because the class doesn’t yet exist until the metaclass call completes—although a metaclass can take the form of a simple callable that invokes type to create the class directly and passes it on to the decorator. In other words, the crucial hook in the model is the type call issued for class construction. Given that, metaclasses and class decorators are often functionally equivalent, with varying dispatch protocol models:

>>> def Metaclass(clsname, supers, attrdict):

return decorator(type(clsname, supers, attrdict))

>>> def decorator(cls): ...

>>> class B(A, metaclass=Metaclass): ... # Metas can call decos and vice versa

In fact, metaclasses need not necessarily return a type instance either—any object compatible with the class coder’s expectations will do—and this further blurs the decorator/metaclass distinction:

>>> def func(name, supers, attrs):

return 'spam'

>>> class C(metaclass=func): # A class whose metaclass makes it a string!

attr = 'huh?'

>>> C, C.upper()

('spam', 'SPAM')

>>> def func(cls):

return 'spam'

>>> @func

class C: # A class whose decorator makes it a string!

attr = 'huh?'

>>> C, C.upper()

('spam', 'SPAM')

Odd metaclass and decorator tricks like these aside, timing often determines roles in practice, as stated earlier:

§ Because decorators run after a class is created, they incur an extra runtime step in class creation roles.

§ Because metaclasses must create classes, they incur an extra coding step in instance management roles.

In other words, neither completely subsumes the other. Strictly speaking, metaclasses might be a functional superset, as they can call decorators during class creation; but metaclasses can also be substantially heavier to understand and code, and many roles intersect completely. In practice, the need to take over class creation entirely is probably much less important than tapping into the process in general.

Rather than follow this rabbit hole further, though, let’s move on to explore metaclass roles that may be a bit more typical and practical. The next section concludes this chapter with one more common use case—applying operations to a class’s methods automatically at class creation time.

Example: Applying Decorators to Methods

As we saw in the prior section, because they are both run at the end of a class statement, metaclasses and decorators can often be used interchangeably, albeit with different syntax. The choice between the two is arbitrary in many contexts. It’s also possible to use them in combination, as complementary tools. In this section, we’ll explore an example of just such a combination—applying a function decorator to all the methods of a class.

Tracing with Decoration Manually

In the prior chapter we coded two function decorators, one that traced and counted all calls made to a decorated function and another that timed such calls. They took various forms there, some of which were applicable to both functions and methods and some of which were not. The following collects both decorators’ final forms into a module file for reuse and reference here:

# File decotools.py: assorted decorator tools

import time

def tracer(func): # Use function, not class with __call__

calls = 0 # Else self is decorator instance only

def onCall(*args, **kwargs):

nonlocal calls

calls += 1

print('call %s to %s' % (calls, func.__name__))

return func(*args, **kwargs)

return onCall

def timer(label='', trace=True): # On decorator args: retain args

def onDecorator(func): # On @: retain decorated func

def onCall(*args, **kargs): # On calls: call original

start = time.clock() # State is scopes + func attr

result = func(*args, **kargs)

elapsed = time.clock() - start

onCall.alltime += elapsed

if trace:

format = '%s%s: %.5f, %.5f'

values = (label, func.__name__, elapsed, onCall.alltime)

print(format % values)

return result

onCall.alltime = 0

return onCall

return onDecorator

As we learned in the prior chapter, to use these decorators manually, we simply import them from the module and code the decoration @ syntax before each method we wish to trace or time:

from decotools import tracer

class Person:

@tracer

def __init__(self, name, pay):

self.name = name

self.pay = pay

@tracer

def giveRaise(self, percent): # giveRaise = tracer(giverRaise)

self.pay *= (1.0 + percent) # onCall remembers giveRaise

@tracer

def lastName(self): # lastName = tracer(lastName)

return self.name.split()[-1]

bob = Person('Bob Smith', 50000)

sue = Person('Sue Jones', 100000)

print(bob.name, sue.name)

sue.giveRaise(.10) # Runs onCall(sue, .10)

print('%.2f' % sue.pay)

print(bob.lastName(), sue.lastName()) # Runs onCall(bob), remembers lastName

When this code is run, we get the following output—calls to decorated methods are routed to logic that intercepts and then delegates the call, because the original method names have been bound to the decorator:

c:\code> py −3 decoall-manual.py

call 1 to __init__

call 2 to __init__

Bob Smith Sue Jones

call 1 to giveRaise

110000.00

call 1 to lastName

call 2 to lastName

Smith Jones

Tracing with Metaclasses and Decorators

The manual decoration scheme of the prior section works, but it requires us to add decoration syntax before each method we wish to trace and to later remove that syntax when we no longer desire tracing. If we want to trace every method of a class, this can become tedious in larger programs. In more dynamic contexts where augmentations depend upon runtime parameters, it may not be possible at all. It would be better if we could somehow apply the tracer decorator to all of a class’s methods automatically.

With metaclasses, we can do exactly that—because they are run when a class is constructed, they are a natural place to add decoration wrappers to a class’s methods. By scanning the class’s attribute dictionary and testing for function objects there, we can automatically run methods through the decorator and rebind the original names to the results. The effect is the same as the automatic method name rebinding of decorators, but we can apply it more globally:

# Metaclass that adds tracing decorator to every method of a client class

from types import FunctionType

from decotools import tracer

class MetaTrace(type):

def __new__(meta, classname, supers, classdict):

for attr, attrval in classdict.items():

if type(attrval) is FunctionType: # Method?

classdict[attr] = tracer(attrval) # Decorate it

return type.__new__(meta, classname, supers, classdict) # Make class

class Person(metaclass=MetaTrace):

def __init__(self, name, pay):

self.name = name

self.pay = pay

def giveRaise(self, percent):

self.pay *= (1.0 + percent)

def lastName(self):

return self.name.split()[-1]

bob = Person('Bob Smith', 50000)

sue = Person('Sue Jones', 100000)

print(bob.name, sue.name)

sue.giveRaise(.10)

print('%.2f' % sue.pay)

print(bob.lastName(), sue.lastName())

When this code is run, the results are the same as before—calls to methods are routed to the tracing decorator first for tracing, and then propagated on to the original method:

c:\code> py −3 decoall-meta.py

call 1 to __init__

call 2 to __init__

Bob Smith Sue Jones

call 1 to giveRaise

110000.00

call 1 to lastName

call 2 to lastName

Smith Jones

The result you see here is a combination of decorator and metaclass work—the metaclass automatically applies the function decorator to every method at class creation time, and the function decorator automatically intercepts method calls in order to print the trace messages in this output. The combination “just works,” thanks to the generality of both tools.

Applying Any Decorator to Methods

The prior metaclass example works for just one specific function decorator—tracing. However, it’s trivial to generalize this to apply any decorator to all the methods of a class. All we have to do is add an outer scope layer to retain the desired decorator, much like we did for decorators in the prior chapter. The following, for example, codes such a generalization and then uses it to apply the tracer decorator again:

# Metaclass factory: apply any decorator to all methods of a class

from types import FunctionType

from decotools import tracer, timer

def decorateAll(decorator):

class MetaDecorate(type):

def __new__(meta, classname, supers, classdict):

for attr, attrval in classdict.items():

if type(attrval) is FunctionType:

classdict[attr] = decorator(attrval)

return type.__new__(meta, classname, supers, classdict)

return MetaDecorate

class Person(metaclass=decorateAll(tracer)): # Apply a decorator to all

def __init__(self, name, pay):

self.name = name

self.pay = pay

def giveRaise(self, percent):

self.pay *= (1.0 + percent)

def lastName(self):

return self.name.split()[-1]

bob = Person('Bob Smith', 50000)

sue = Person('Sue Jones', 100000)

print(bob.name, sue.name)

sue.giveRaise(.10)

print('%.2f' % sue.pay)

print(bob.lastName(), sue.lastName())

When this code is run as it is, the output is again the same as that of the previous examples—we’re still ultimately decorating every method in a client class with the tracer function decorator, but we’re doing so in a more generic fashion:

c:\code> py −3 decoall-meta-any.py

call 1 to __init__

call 2 to __init__

Bob Smith Sue Jones

call 1 to giveRaise

110000.00

call 1 to lastName

call 2 to lastName

Smith Jones

Now, to apply a different decorator to the methods, we can simply replace the decorator name in the class header line. To use the timer function decorator shown earlier, for example, we could use either of the last two header lines in the following when defining our class—the first accepts the timer’s default arguments, and the second specifies label text:

class Person(metaclass=decorateAll(tracer)): # Apply tracer

class Person(metaclass=decorateAll(timer())): # Apply timer, defaults

class Person(metaclass=decorateAll(timer(label='**'))): # Decorator arguments

Notice that this scheme cannot support nondefault decorator arguments differing per method in the client class, but it can pass in decorator arguments that apply to all such methods, as done here. To test, use the last of these metaclass declarations to apply the timer, and add the following lines at the end of the script to see the timer’s extra informational attributes:

# If using timer: total time per method

print('-'*40)

print('%.5f' % Person.__init__.alltime)

print('%.5f' % Person.giveRaise.alltime)

print('%.5f' % Person.lastName.alltime)

The new output is as follows—the metaclass wraps methods in timer decorators now, so we can tell how long each and every call takes, for every method of the class:

c:\code> py −3 decoall-meta-any2.py

**__init__: 0.00001, 0.00001

**__init__: 0.00001, 0.00001

Bob Smith Sue Jones

**giveRaise: 0.00002, 0.00002

110000.00

**lastName: 0.00002, 0.00002

**lastName: 0.00002, 0.00004

Smith Jones

----------------------------------------

0.00001

0.00002

0.00004

Metaclasses Versus Class Decorators: Round 3 (and Last)

As you might expect, class decorators intersect with metaclasses here, too. The following version replaces the preceding example’s metaclass with a class decorator. That is, it defines and uses a class decorator that applies a function decorator to all methods of a class. Although the prior sentence may sound more like a Zen statement than a technical description, this all works quite naturally—Python’s decorators support arbitrary nesting and combinations:

# Class decorator factory: apply any decorator to all methods of a class

from types import FunctionType

from decotools import tracer, timer

def decorateAll(decorator):

def DecoDecorate(aClass):

for attr, attrval in aClass.__dict__.items():

if type(attrval) is FunctionType:

setattr(aClass, attr, decorator(attrval)) # Not __dict__

return aClass

return DecoDecorate

@decorateAll(tracer) # Use a class decorator

class Person: # Applies func decorator to methods

def __init__(self, name, pay): # Person = decorateAll(..)(Person)

self.name = name # Person = DecoDecorate(Person)

self.pay = pay

def giveRaise(self, percent):

self.pay *= (1.0 + percent)

def lastName(self):

return self.name.split()[-1]

bob = Person('Bob Smith', 50000)

sue = Person('Sue Jones', 100000)

print(bob.name, sue.name)

sue.giveRaise(.10)

print('%.2f' % sue.pay)

print(bob.lastName(), sue.lastName())

When this code is run as it is, the class decorator applies the tracer function decorator to every method and produces a trace message on calls (the output is the same as that of the preceding metaclass version of this example):

c:\code> py −3 decoall-deco-any.py

call 1 to __init__

call 2 to __init__

Bob Smith Sue Jones

call 1 to giveRaise

110000.00

call 1 to lastName

call 2 to lastName

Smith Jones

Notice that the class decorator returns the original, augmented class, not a wrapper layer for it (as is common when wrapping instance objects instead). As for the metaclass version, we retain the type of the original class—an instance of Person is an instance of Person, not of some wrapper class. In fact, this class decorator deals with class creation only; instance creation calls are not intercepted at all.

This distinction can matter in programs that require type testing for instances to yield the original class, not a wrapper. When augmenting a class instead of an instance, class decorators can retain the original class type. The class’s methods are not their original functions because they are rebound to decorators, but this is likely less important in practice, and it’s true in the metaclass alternative as well.

Also note that, like the metaclass version, this structure cannot support function decorator arguments that differ per method in the decorated class, but it can handle such arguments if they apply to all such methods. To use this scheme to apply the timer decorator, for example, either of the last two decoration lines in the following will suffice if coded just before our class definition—the first uses decorator argument defaults, and the second provides one explicitly:

@decorateAll(tracer) # Decorate all with tracer

@decorateAll(timer()) # Decorate all with timer, defaults

@decorateAll(timer(label='@@')) # Same but pass a decorator argument

As before, let’s use the last of these decorator lines and add the following at the end of the script to test our example with a different decorator (better schemes are possible on both the testing and timing fronts here, of course, but we’re at chapter end; improve as desired):

# If using timer: total time per method

print('-'*40)

print('%.5f' % Person.__init__.alltime)

print('%.5f' % Person.giveRaise.alltime)

print('%.5f' % Person.lastName.alltime)

The same sort of output appears—for every method we get timing data for each and all calls, but we’ve passed a different label argument to the timer decorator:

c:\code> py −3 decoall-deco-any2.py

@@__init__: 0.00001, 0.00001

@@__init__: 0.00001, 0.00001

Bob Smith Sue Jones

@@giveRaise: 0.00002, 0.00002

110000.00

@@lastName: 0.00002, 0.00002

@@lastName: 0.00002, 0.00004

Smith Jones

----------------------------------------

0.00001

0.00002

0.00004

Finally, it’s possible to combine decorators such that each runs per method call, but it will likely require changes to those we’ve coded here. As is, nesting calls to them directly winds up tracing or timing the other’s creation-time application, listing the two on separate lines results in tracing or timing the other’s wrapper before running the original method, and metaclasses seem to fare no better on this front:

@decorateAll(tracer(timer(label='@@'))) # Traces applying the timer

class Person:

@decorateAll(tracer) # Traces onCall wrapper, times methods

@decorateAll(timer(label='@@'))

class Person:

@decorateAll(timer(label='@@'))

@decorateAll(tracer) # Times onCall wrapper, traces methods

class Person:

Pondering this further will have to remain suggested study—both because we’re out of space and time, and because this may quite possibly be illegal in some states!

As you can see, metaclasses and class decorators are not only often interchangeable, but also commonly complementary. Both provide advanced but powerful ways to customize and manage both class and instance objects, because both ultimately allow you to insert code into the class creation process. Although some more advanced applications may be better coded with one or the other, the way you choose or combine these two tools in many cases is largely up to you.

Chapter Summary

In this chapter, we studied metaclasses and explored examples of them in action. Metaclasses allow us to tap into the class creation protocol of Python, in order to manage or augment user-defined classes. Because they automate this process, they may provide better solutions for API writers than manual code or helper functions; because they encapsulate such code, they may minimize maintenance costs better than some other approaches.

Along the way, we also saw how the roles of class decorators and metaclasses often intersect: because both run at the conclusion of a class statement, they can sometimes be used interchangeably. Class decorators and metaclasses can both be used to manage both class and instance objects, though each tool may present tradeoffs in some use cases.

Since this chapter covered an advanced topic, we’ll work through just a few quiz questions to review the basics (candidly, if you’ve made it this far in a chapter on metaclasses, you probably already deserve extra credit!). Because this is the last part of the book, we’ll forgo the end-of-part exercises. Be sure to see the appendixes that follow for Python changes, the solutions to the prior parts’ exercises, and more; the last of these includes a sampling of typical application-level programs for self-study.

Once you finish the quiz, you’ve officially reached the end of this book’s technical material. The next and final chapter offers some brief closing thoughts to wrap up the book at large. I’ll see you there in the Python benediction after you work through this final quiz.

Test Your Knowledge: Quiz

1. What is a metaclass?

2. How do you declare the metaclass of a class?

3. How do class decorators overlap with metaclasses for managing classes?

4. How do class decorators overlap with metaclasses for managing instances?

5. Would you rather count decorators or metaclasses amongst your weaponry? (And please phrase your answer in terms of a popular Monty Python skit.)

Test Your Knowledge: Answers

1. A metaclass is a class used to create a class. Normal new-style classes are instances of the type class by default. Metaclasses are usually subclasses of the type class, which redefines class creation protocol methods in order to customize the class creation call issued at the end of aclass statement; they typically redefine the methods __new__ and __init__ to tap into the class creation protocol. Metaclasses can also be coded other ways—as simple functions, for example—but they are always responsible for making and returning an object for the new class. Metaclasses may have methods and data to provide behavior for their classes too—and constitute a secondary pathway for inheritance search—but their attributes are accessible only to their class instances, not to their instance’s instances.

2. In Python 3.X, use a keyword argument in the class header line: class C(metaclass=M). In Python 2.X, use a class attribute instead: __metaclass__ = M. In 3.X, the class header line can also name normal superclasses before the metaclass keyword argument; in 2.X you generally should derive from object too, though this is sometimes optional.

3. Because both are automatically triggered at the end of a class statement, class decorators and metaclasses can both be used to manage classes. Decorators rebind a class name to a callable’s result and metaclasses route class creation through a callable, but both hooks can be used for similar purposes. To manage classes, decorators simply augment and return the original class objects. Metaclasses augment a class after they create it. Decorators may have a slight disadvantage in this role if a new class must be defined, because the original class has already been created.

4. Because both are automatically triggered at the end of a class statement, we can use both class decorators and metaclasses to manage class instances, by inserting a wrapper (proxy) object to catch instance creation calls. Decorators may rebind the class name to a callable run on instance creation that retains the original class object. Metaclasses can do the same, but may have a slight disadvantage in this role, because they must also create the class object.

5. Our chief weapon is decorators...decorators and metaclasses...metaclasses and decorators... Our two weapons are metaclasses and decorators...and ruthless efficiency... Our three weapons are metaclasses, decorators, and ruthless efficiency...and an almost fanatical devotion to Python... Our four...no... Amongst our weapons... Amongst our weaponry...are such elements as metaclasses, decorators... I’ll come in again...