Cython (2015)

Chapter 5. Cython and Extension Types

Make everything as simple as possible, but not simpler.

— A. Einstein

In Chapter 3, we covered the fundamentals of what Cython adds to the Python language, and the power and control those additions provide. That chapter focused on basic data types and functions. Cython can enhance Python classes as well. Before we learn the specifics, we must first review the difference between Python classes and extension types, which will help us understand the what and why of Cython’s approach.

Comparing Python Classes and Extension Types

In Python everything is an object. What does that mean, specifically? At its most basic level, an object has three things: identity, value, and type. An object’s identity distinguishes it from all others and is provided by the id built-in function. An object’s value is simply the data associated with it, accessible via dot notation. Typically Python places an object’s data inside an internal instance dictionary named __dict__. The third essential attribute of any object is its type, which specifies the behaviors that an object of that type exhibits. These behaviors are accessible via special functions, called methods. A type is responsible for creating and destroying its objects, initializing them, and updating their values when methods are called on the object. Python allows us to create new types, in Python code, with the class statement.

We will see in this chapter how Cython allows low-level C access to an object’s data and methods, and what benefits that access provides.

The built-in types—object, list, dict, file, int, float, and so on—are implemented at the C level via the Python/C API and are incorporated into the Python runtime. Usage-wise, built-in types behave just like regular Python classes defined with the class statement, and the Python type system treats built-in types just like regular classes.

We can also create our own types at the C level directly using the Python/C API; these are known as extension types. They fold into the type system along with regular Python classes and built-in types, and are therefore transparent to the end user. When we call methods on extension type instances, we are running compiled and statically typed code. In particular, the extension type has fast C-level access to the type’s methods and the instance’s data. As discussed in Chapter 3, this fast C-level access can lead to significant performance improvements. Implementation-wise, defining an extension type’s methods and working with a type’s instances is very different from defining new classes in pure Python. Implementing an extension type directly in C requires expertise in the Python/C API and is not for the uninitiated.

This is where Cython comes in: Cython makes creating and using extension types as straightforward as working with pure-Python classes. Extension types are created in Cython with the cdef class statement, and have much in common with regular Python classes.

Despite the syntactic similarities, it is important to remember that a cdef class has fast C-level access to all methods and data. This feature is the most significant difference between an extension type and a plain Python class defined in a .py module.

Let’s see an example.

Extension Types in Cython

Consider a simple class meant to model particles. Each particle has a mass, an x position, and a velocity. A simple Particle class in Python would look something like:^[8^]

class Particle(object):

"""Simple Particle type."""

def __init__(self, m, p, v):

self.mass = m

self.position = p

self.velocity = v

def get_momentum(self):

return self.mass * self.velocity

This class can be defined in pure Python at the interpreted level, or it can be compiled by Cython. In both cases, the result is essentially the same. An instance of Particle has a mass, a position, and a velocity, and users can call its get_momentum method. All attributes are readable and writeable, and users are free to assign other attributes to Particle objects outside the class body.

When we compile the Particle class to C with cython, the resulting class is just a regular Python class, not an extension type. When Cython compiles it to C, it is still implemented with general Python objects using dynamic dispatch for all operations. The generated code uses the Python/C API heavily and makes the same calls that the interpreter would if this class were defined in pure Python. Because the interpreter overhead is removed, the Cython version of Particle will have a small performance boost. But it does not benefit from any static typing, so the Cython code still has to fall back on dynamic dispatch to resolve types at runtime.

It is trivial to convert the Particle class into an extension type:

cdef class Particle:

"""Simple Particle extension type."""

cdef double mass, position, velocity

# ...

There are two additions: cdef is added before the class statement, and static cdef declarations are added in the class body after the docstring, one for each instance attribute assigned to in __init__. The __init__ and get_momentum methods remain unchanged.

The cdef class statement tells Cython to make an extension type rather than a regular Python class. The cdef type declarations in the class body are not, despite appearances, class-level attributes. They are C-level instance attributes; this style of attribute declaration is similar to languages like C++ and Java. All instance attributes must be declared with cdef at the class level in this way for extension types. If we did not declare all three of mass, position, and velocity in our Particle extension type, we would get a runtime exception inside __init__ when we tried to assign to an undeclared attribute.

Let’s kick the tires. We’ll put our cdef class Particle in a file cython_particle.pyx, and the regular class Particle in a file python_particle.py. Then, from IPython:

In [1]: import pyximport; pyximport.install()

Out[1]: (None, <pyximport.pyximport.PyxImporter at 0x101c64290>)

In [2]: import cython_particle

In [3]: import python_particle

Here we use pyximport to compile the cython_particle.pyx file automatically at import time. We can inspect the two Particle types:

In [4]: python_particle.Particle?

Type: type

String Form:<class 'python_particle.Particle'>

File: [...]/python_particle.py

Docstring: Simple Particle type.

Constructor information:

Definition:python_particle.Particle(self, m, p, v)

In [5]: cython_particle.Particle?

Type: type

String Form:<type 'cython_particle.Particle'>

File: [...]/cython_particle.so

Docstring: Simple Particle extension type.

And we see that, besides the fact that the Cython version comes from a compiled library, they are very similar.

The two types have identical initializers, so creation is the same:

In [6]: py_particle = python_particle.Particle(1.0, 2.0, 3.0)

In [7]: cy_particle = cython_particle.Particle(1.0, 2.0, 3.0)

Calling their get_momentum methods is as we would expect:

In [8]: py_particle.get_momentum()

Out[8]: 3.0

In [9]: cy_particle.get_momentum()

Out[9]: 3.0

We can access all of the py_particle’s attributes:

In [10]: py_particle.mass, py_particle.position, py_particle.velocity

Out[10]: (1.0, 2.0, 3.0)

but none of cy_particle’s:

In [11]: cy_particle.mass, cy_particle.position, cy_particle.velocity

Traceback (most recent call last)

[...]

AttributeError: 'cython_particle.Particle' object has no attribute 'mass'

Furthermore, we can add new attributes to py_particle on the fly, but cy_particle is locked down:

In [13]: py_particle.charge = 12.0

In [14]: cy_particle.charge = 12.0

Traceback (most recent call last)

[...]

AttributeError: 'cython_particle.Particle' object has no attribute 'charge'

This seems strange—why are the instance attributes in the extension type not accessible from Python? Why can we add new attributes for py_particle and not cy_particle? And why do we have to declare them with cdef in the first place?

When an extension type like cython_particle.Particle is instantiated, a C struct is allocated and initialized. These steps require that the size and fields of that struct be known at compile time, hence the need to declare all attributes with cdef.

In contrast, when python_particle.Particle is instantiated, a Python dictionary is created and assigned to the instance’s __dict__ attribute, and all other attributes are stored here with their associated values:

In [15]: py_particle.__dict__

Out[15]: {'charge': 12.0, 'mass': 1.0, 'position': 2.0, 'velocity': 3.0}

C structs are fixed and not open to new members, so no new attributes can be set on an extension type instance. For an object of a regular Python class, its underlying dictionary is modifiable and open to new key/value pairs, as we can see with the "charge": 12.0 key/value pair in the preceding IPython output.

Extension type attributes are private by default, and are accessible by the methods of the class. We saw how get_momentum was able to return the right value in both cases. An instance of a regular class is wide open—anything can access and modify its attributes.

Type Attributes and Access Control

In the pure-Python Particle class, attribute access like self.mass goes through a general lookup process that works for any attribute, whether it is an instance attribute, a method, or a method or data attribute inside a base class. In our example the process will eventually find the masskey inside the instance’s __dict__ and return its associated value without much effort. But it is possible for the attribute lookup machinery to go through several levels of indirection to find its target. As always, this generality comes with performance overhead.

Methods defined in cdef class extension types have full access to all instance attributes. Furthermore, cython will translate any accesses like self.mass or self.velocity into low-level accesses to C-struct fields. This bypasses the general lookup process for pure-Python classes, and can lead to significant performance improvements.

But what if we want to be able to access instance attributes of extension types? It is straightforward to have Cython make instance attributes read-only, or readable and writeable.

First, let’s see an example with read-only attributes. We include the readonly declaration along with the instance attributes, like this:

cdef class Particle:

"""Simple Particle extension type."""

cdef readonly double mass, position, velocity

# ...

If we wanted just the mass attribute to be accessible from Python, but position and velocity to remain private, we would say:

cdef class Particle:

"""Simple Particle extension type."""

cdef readonly double mass

cdef double position, velocity

# ...

After making these changes, we have to recompile the extension module, which means reimporting it from a new interpreter session with pyximport:

In [1]: import pyximport; pyximport.install()

Out[1]: (None, <pyximport.pyximport.PyxImporter at 0x101c64290>)

In [2]: import cython_particle

The mass attribute is now accessible from Python:

In [3]: cy_particle = cython_particle.Particle(1.0, 2.0, 3.0)

In [4]: cy_particle.mass

Out[4]: 1.0

But it is not modifiable:

In [5]: cy_particle.mass = -3.0

Traceback (most recent call last)

[...]

AttributeError: attribute 'mass' of 'cython_particle.Particle'

objects is not writable

If we want to make an attribute both readable and writeable from Python, we can use the public attribute:

cdef class Particle:

"""Simple Particle extension type."""

cdef public double mass

cdef readonly double position

cdef double velocity

# ...

Here we have made mass readable and writeable with public, position read-only, and velocity private.

After recompiling via pyximport, we see that we can now access both the mass and position attributes:

In [3]: cy_particle = cython_particle.Particle(1.0, 2.0, 3.0)

In [4]: cy_particle.mass

Out[4]: 1.0

In [5]: cy_particle.mass, cy_particle.position

Out[5]: (1.0, 2.0)

and we can modify the mass as well:

In [6]: cy_particle.mass = 1e-6

When calling the get_momentum method, Cython still uses fast C-level direct access, and extension type methods essentially ignore the readonly and public declarations. These exist only to allow and control access from Python.

C-Level Initialization and Finalization

The fact that we have a C struct behind every extension type instance has other implications, particularly for object creation and initialization. When Python calls __init__, the self argument is required to be a valid instance of that extension type. When __init__ is called, it typically initializes the attributes on the self argument. At the C level, before __init__ is called, the instance’s struct must be allocated, and all struct fields must be in a valid state, ready to accept initial values.

Cython adds a special method named __cinit__ whose responsibility is to perform C-level allocation and initialization. For the Particle extension type declared earlier, __init__ can take on this role, because the fields are all double scalars and require no C-level allocations. But it is possible, depending on how an extension type is subclassed or if there are alternative constructors, for __init__ to be called multiple times during object creation, and there are other situations where __init__ is bypassed entirely. Cython guarantees that __cinit__ is called exactly once and that it is called before __init__, __new__, or alternative Python-level constructors (e.g., classmethod constructors). Cython passes any initialization arguments into __cinit__.

For example, say we have an extension type whose instances have an internal C array, dynamically allocated:

cdef class Matrix:

cdef:

unsigned int nrows, ncols

double *_matrix

The correct place to put self._matrix’s dynamic allocation is in a __cinit__ method:

cdef class Matrix:

cdef:

unsigned int nrows, ncols

double *_matrix

def __cinit__(self, nr, nc):

self.nrows = nr

self.ncols = nc

self._matrix = <double*>malloc(nr * nc * sizeof(double))

if self._matrix == NULL:

raise MemoryError()

If self._matrix were allocated inside __init__ instead, and __init__ were never called—which can occur with an alternate classmethod constructor, for instance—then any method using self._matrix would lead to ugly segmentation faults. Conversely, if __init__ were called twice—perhaps due to inconsistent use of super in a class hierarchy—then a memory leak would result (and would be particularly difficult to track down).

What about cleanup? Cython also supports C-level finalization through the __dealloc__ special method. This method’s responsibility is to undo what __cinit__ did during creation. For our Matrix extension type, we should add a __dealloc__ that frees the self._matrix array:

cdef class Matrix:

cdef:

unsigned int nrows, ncols

double *_matrix

def __cinit__(self, nr, nc):

self.nrows = nr

self.ncols = nc

self._matrix = <double*>malloc(nr * nc * sizeof(double))

if self._matrix == NULL:

raise MemoryError()

def __dealloc__(self):

if self._matrix != NULL:

free(self._matrix)

If defined, Cython ensures that __dealloc__ is called once during finalization. In this example __dealloc__ need only check that self._matrix is non-null and free it to ensure no memory leaks.

Now that we have covered the essential pieces for creation and finalization of extension type instances, let’s focus on extension type methods. Cython’s cdef and cpdef declarations work there as well.

cdef and cpdef Methods

The concepts we learned in Chapter 3 about def, cdef, and cpdef functions also apply to extension type methods. Note that we cannot use cdef and cpdef to define methods on non-cdef classes; doing so is a compile-time error.

A cdef method has C calling semantics, just as cdef functions do: all arguments are passed in as is, so no type mapping from Python to C occurs. This provides cdef methods with a performance boost over their def counterparts, which always have to accept and return Python objects of one type or another. This also means that a cdef method is accessible only from other Cython code and cannot be called from Python.

A cpdef method is particularly useful. As we can infer from what we know about cpdef functions, a cpdef method is callable both from external Python code and from other Cython code. When it is called from Cython, no marshalling to and from Python objects takes place, so it is as efficient as can be. However, the argument and return types have to be automatically convertible from and to Python objects, respectively, which restricts the allowed types somewhat (no pointer types, for example).

For example, we can declare the get_momentum method on the Particle extension type to be a cpdef method instead:

cdef class Particle:

"""Simple Particle extension type."""

cdef double mass, position, velocity

# ...

cpdef double get_momentum(self):

return self.mass * self.velocity

Say we have a function add_momentums:

def add_momentums(particles):

"""Returns the sum of the particle momentums."""

total_mom = 0.0

for particle inparticles:

total_mom += particle.get_momentum()

return total_mom

This could be defined in interpreted Python, or it could be compiled and run by Cython— in either case, the call to get_momentum is a fully general Python attribute lookup and call, because Cython does not know that particles is a list of Particle objects.

Calling add_momentums in the preceding example on a list of 1,000 Particle objects takes approximately 65 microseconds.

When Python calls get_momentum on a Particle object, the get_momentum Python wrapper is used, and the correct packing and unpacking from Python object to underlying Particle struct occurs automatically.

If we add typing information, then Cython will be able to generate faster code:

def add_momentums_typed(list particles):

"""Returns the sum of the particle momentums."""

cdef:

double total_mom = 0.0

Particle particle

for particle inparticles:

total_mom += particle.get_momentum()

return total_mom

Note that we typed the particles argument as a list, total_mom as a double, and, crucially, the loop indexing variable particle as a Particle.

Because particle is a statically typed Particle and get_momentum is a cpdef method, when get_momentum is called in add_momentums_typed, no Python objects are involved. Even the in-place sum is a C-only operation, because total_mom is a statically typed C double.

This typed version takes about 7 microseconds to run on the same list as before, indicating a tenfold speedup over the untyped version. To see the effect of the cpdef over the def method, we can remove the Particle particle declaration, forcing Cython to use Python calling semantics on particle.get_momentum(). The result isn’t pretty: 71 microseconds, which is slower than the all-Python version! Typing the particle loop variable here yields the most significant performance improvement; typing particles and total_mom has less of an effect.

There is one last comparison to make: what if we make get_momentum a cdef method? To keep things separate, we will define another method, get_momentum_c:

cdef class Particle:

"""Simple Particle extension type."""

cdef double mass, position, velocity

# ...

cpdef double get_momentum(self):

return self.mass * self.velocity

cdef double get_momentum_c(self):

return self.mass * self.velocity

We will have to modify add_momentums_typed as well; we will call the new version add_momentums_typed_c for clarity:

def add_momentums_typed_c(list particles):

"""Returns the sum of the particle momentums."""

cdef:

double total_mom = 0.0

Particle particle

for particle inparticles:

total_mom += particle.get_momentum_c()

return total_mom

This version has the best performance: approximately 4.6 microseconds, another 40 percent boost over the add_momentums_typed version. The downside is that get_momentum_c is not callable from Python code, only Cython.^[9^]

What explains this additional performance improvement? To answer that, we will have to understand the basics of inheritance, subclassing, and polymorphism with extension types.

Inheritance and Subclassing

An extension type can subclass a single base type, and that base type must itself be a type implemented in C—either a built-in type or another extension type. If the base type is a regular Python class, or if the extension type attempts to inherit from multiple base types, a cython compile-time error will result.

For example, consider a subclass of Particle, called CParticle, that stores the particle’s momentum rather than computing it on the fly. We do not want to duplicate work done in Particle, so we subclass it:^[10^]

cdef class CParticle(Particle):

cdef double momentum

def __init__(self, m, p, v):

super(CParticle, self).__init__(m, p, v)

self.momentum = self.mass * self.velocity

cpdef double get_momentum(self):

return self.momentum

Because a CParticle is a (more specific) Particle, everywhere we use a Particle, we should be able to substitute in a CParticle without any modification to the code, all while we revel in the Platonic beauty of polymorphism. In our add_momentums or add_momentums_typedfunctions defined in the preceding examples, we can pass in a list of CParticles instead. The add_momentums function does everything with dynamic Python variables, so everything follows Python semantics there. But add_momentums_typed expects the elements of the list to beParticle instances. When CParticles are passed in, the right version of get_momentum is resolved, bypassing the Python/C API.

We can subclass Particle in pure Python as well. Consider PyParticle:

class PyParticle(Particle):

def __init__(self, m, p, v):

super(PyParticle, self).__init__(m, p, v)

def get_momentum(self):

return super(PyParticle, self).get_momentum()

The PyParticle class cannot access any private C-level attributes or cdef methods. It can override def and cpdef methods defined on Particle, as we have done with get_momentum. We can pass add_momentums_typed a list of PyParticles as well; doing so takes about 340 microseconds per call, making it about five times slower than using Particle objects. Crossing the Cython/Python language boundary polymorphically is nice, but it does have overhead.

Because a cdef method is not accessible or overrideable from Python, it does not have to cross the language boundary, so it has less call overhead than a cpdef equivalent. This is a relevant concern only for small functions where call overhead is non-negligible. For methods that perform significant calculations, the performance difference between cdef and cpdef is less a concern.

Casting and Subclasses

When working with a dynamically typed object, Cython cannot access any C-level data or methods on it. All attribute lookup must be done via the Python/C API, which is slow. If we know the dynamic variable is or may possibly be an instance of a built-in type or an extension type, then it is worth casting to the static type. Doing so allows Cython to access C-level attributes and methods, and it can do so more efficiently. Further, Cython can also access Python-level attributes and cpdef methods directly without going through the Python/C API.

There are two ways to perform this casting: either by creating a statically typed variable of the desired type and assigning the dynamic variable to it, or by using Cython’s casting operator, covered briefly in Chapter 3.

For example, say we are working with an object p that might be an instance of Particle or one of its subclasses. All Cython knows about p is that it is a Python object. We can call get_momentum on it, which will work if p has such a method and fail with an AttributeError otherwise. Because p is a dynamic variable, Cython will access get_momentum by looking it up in a Python dictionary, and if successful, PyObject_Call will execute the method. But if we cast it to a Particle explicitly, the call to get_momentum will be much faster:

cdef Particle static_p = p

print static_p.get_momentum()

print static_p.velocity

The assignment to static_p will raise a TypeError exception if p is not an instance of Particle or its subclasses, so this is safe. The call static_p.get_momentum will use direct access to the get_momentum cpdef method. It also allows access to the private velocity attribute, which is not available via p.

TIP

Cython uses general Python method lookups on dynamically typed objects. This will fail with an AttributeError if the method is declared cdef. To ensure fast access to cpdef methods, or to allow any access to cdef methods, we must provide static type information for the object.

Cython also supports the casting operator, and we can use it to achieve the same result:

print (<Particle>p).get_momentum()

print (<Particle>p).velocity

This removes the need to create a temporary variable as in the previous example. The cast is enclosed in parentheses due to Cython’s precedence rules. Because we use a raw cast to a Particle object in this example, no type checking is performed for performance reasons. It is unsafe if p is not an instance of Particle, which may lead to a segmentation fault. If there is a possibility that p is not a Particle, then using the checked cast is safer:

print (<Particle?>p).get_momentum()

print (<Particle?>p).velocity

If p is not a Particle, this example will raise a TypeError. The tradeoff is that a checked cast calls into the Python/C API and incurs runtime overhead, trading performance for safety.

Extension Type Objects and None

Consider a simple function dispatch:

def dispatch(Particle p):

print p.get_momentum()

print p.velocity

If we call dispatch and pass a non-Particle object, then we would expect to get a TypeError. Usually, this is the case:

dispatch(Particle(1, 2, 3)) # OK

dispatch(CParticle(1, 2, 3)) # OK

dispatch(PyParticle(1, 2, 3)) # OK

dispatch(object()) # TypeError

However, Cython treats None specially—even though it is not an instance of Particle, Cython allows it to be passed in as if it were. This is analogous to the NULL pointer in C: it is allowed wherever a C pointer is expected, but doing anything other than checking whether it is NULL will result in a segmentation fault or worse.

Calling dispatch with None does not result in a TypeError:

dispatch(None) # Segmentation fault!

The reason for the segmentation fault when None is passed to dispatch is because dispatch (unsafely) accesses the cpdef function get_momentum and the private attribute velocity, both of which are part of Particle’s C interface. Python’s None object essentially has no C interface, so trying to call a method on it or access an attribute is not valid. To make these operations safe, dispatch could check if p is None first:

def dispatch(Particle p):

if p isNone:

raise TypeError("...")

print p.get_momentum()

print p.velocity

This is such a common operation that Cython provides special syntax for it:

def dispatch(Particle p notNone):

print p.get_momentum()

print p.velocity

This version of dispatch will do the right thing when passed None, at the expense of some up-front type checking. If there is any possibility that a function or method argument might be None, then it is our responsibility to guard against it if accessing any C-level attributes or methods on the object. Not doing so will result in ugly segmentation faults or data corruption. If we access only Python-level methods (i.e., def methods) and Python-level attributes (public or readonly attributes, for example) on the object, then an exception will be raised, as the Python/C API will handle things for us.

Many see the need for the not None clause as inconvenient; this feature of Cython is often debated. Fortunately, it is straightforward to write None-safe code with the not None clause in the function’s argument declaration.

Cython also provides a nonecheck compiler directive—off by default for performance reasons—that makes all function and method calls None-safe. To enable None checking globally for an extension module, we can either place a directive comment toward the beginning of the file:

# cython: nonecheck=True

or set nonecheck to True from the command line during compilation:

$ cython --directive nonecheck=True source.pyx

Extension Type Properties in Cython

Python properties are handy and very powerful, allowing precise control over attribute access and on-the-fly computation.

All this time, the Particle extension type has had a get_momentum method, but any Python programmer would berate us for having a getter method like that; the right way to do it is to either expose momentum directly or make a property instead. Doing so in pure Python is simple with the property built-in function:

class Particle(object):

# ...

def _get_momentum(self):

return self.mass * self.velocity

momentum = property(_get_momentum)

Accessing p.momentum (no parentheses!) on a Particle instance p calls _get_momentum automatically. It is not possible to set or delete p.momentum because no setter or deleter was passed to property when the momentum property was defined.

Cython has different syntax for extension type properties, but it achieves the same end:

cdef class Particle:

"""Simple Particle extension type."""

cdef double mass, position, velocity

# ...

property momentum:

"""The momentum Particle property."""

__get__(self):

"""momentum's getter"""

return self.mass * self.velocity

We can now access p.momentum from either Python code or Cython code; doing so calls the underlying __get__() momentum getter. The property and __get__ docstrings are optional; if present, they can be extracted by automatic documentation generators, and are equivalent to passing in a doc argument to the Python property built-in function. If Cython knows the static type of the object in question, the property access will be efficient and bypass the Python/C API. Like the pure-Python property in the initial example, this is a read-only property.

For the sake of this example, suppose we want to be able to get and set a Particle’s momentum. We can add a __set__ property method to do so:

cdef class Particle:

"""Simple Particle extension type."""

# ...

property momentum:

"""The momentum Particle property."""

def __get__(self):

"""momentum's getter"""

return self.mass * self.velocity

def __set__(self, m):

"""momentum's setter"""

self.velocity = m / self.mass

We arbitrarily decide that setting the momentum will modify the velocity and leave the mass unchanged. This allows p.momentum to be assigned to:

In [3]: p = cython_particle.Particle(1, 2, 3)

In [4]: p.momentum

Out[4]: 3.0

In [5]: p.momentum = 4.0

In [6]: p.momentum

Out[6]: 4.0

If it makes sense to do so, we can also define a __del__ property method, which controls property deletion. If any one of __get__, __set__, or __del__ is not defined, then that operation is not allowed.

To finish our treatment of extension types in Cython, we should cover how extension type special methods are different from their pure-Python counterparts.

Special Methods Are Even More Special

When providing support for operator overloading with a Cython extension type, we have to define a special method; that is, a method of a specific name with leading and trailing double underscores. We previously covered the __cinit__, __init__, and __dealloc__ special methods and saw how they handle C-level initialization, Python-level initialization, and finalization, respectively. Extension types do not support the __del__ special method; that is the role of __dealloc__.

Arithmetic Methods

To support the in-place + operator for a pure-Python class C, we define an __add__(self, other) method. The operation c + d is transformed into C.__add__(c, d) when c is an instance of the C class. If C does not know how to add itself to the other argument, then it returnsNotImplemented. In this case, the Python interpreter then calls type(d).__radd__(d, c) to give d’s class a chance to add itself to a C instance.

For extension types, the situation is different.^[11^] Extension types do not support __radd__; instead, they (effectively) overload __add__ to do the job of both the regular __add__ and __radd__ in one special method. This means that, for a Cython-defined extension type E, __add__ will be called when the expression e + f is evaluated and e is an E instance. In this case, the arguments to __add__ are e and f, in that order. The __add__ method will also be called when the expression f + e is evaluated and f’s __add__ method returns NotImplemented, indicating thatf cannot handle an E instance. In this case, E.__add__ is called with f and e as arguments, in that order! So __add__ may be called with an arbitrary type as the first argument, not an instance of the E class; because of this possibility, it is misleading to name its first argument self.

Here is the proper implementation of __add__ for a simple Cython extension type that can be added to integers:

cdef class E:

"""Extension type that supports addition."""

cdef int data

def __init__(self, d):

self.data = d

def __add__(x, y):

# Regular __add__ behavior

if isinstance(x, E):

if isinstance(y, int):

return (<E>x).data + y

# __radd__ behavior

elif isinstance(y, E):

if isinstance(x, int):

return (<E>y).data + x

else:

return NotImplemented

Cython does not automatically type either argument to __add__, making the isinstance check and cast necessary to access each E instance’s internal .data attribute.

Let’s place the preceding code block in special_methods.pyx and try it out from IPython:

In [1]: import pyximport; pyximport.install()

Out[1]: (None, <pyximport.pyximport.PyxImporter at 0x101c65290>)

In [2]: import special_methods

In [3]: e = special_methods.E(100)

In [4]: e + 1

Out[4]: 101

In [5]: 1 + e

Out[5]: 101

The first addition takes the first branch of E.__add__, and the second addition takes the second branch. What about the error cases?

In [6]: e + 1.0

Traceback (most recent call last):

[...]

TypeError: unsupported operand type(s) for +:

'special_methods.E' and 'float'

For this case, E.__add__ returns NotImplemented, and the built-in float type tries to do an __radd__ with an E instance as the left argument. Not knowing how to add itself to an E object, it again returns NotImplemented, and Python then raises a TypeError.

One more case to consider:

In [7]: 1.0 + e

Traceback (most recent call last):

[...]

TypeError: unsupported operand type(s) for +:

'float' and 'special_methods.E'

For this case, float’s __add__ was called, realized it did not know how to handle E instances, and returned NotImplemented. Python then called E.__add__(1.0, e) (or the equivalent), which also returned NotImplemented, causing Python to raise the TypeError.

Phew. That rounds it out for __add__. Cython follows the same pattern for all arithmetic special methods, so what we have learned about __add__ here applies elsewhere.

The in-place operations like __iadd__ always take an instance of the class as the first argument, so self is an appropriate name in these cases. The exception to this is __ipow__, which may be called with a different order of arguments, like __add__.

Rich Comparisons

Cython extension types do not support the individual comparison special methods like __eq__, __lt__, and __le__. Instead, Cython provides a single (some would say cryptic) method, __richcmp__(x, y, op), that takes an integer third argument to specify which comparison operation to perform. The correspondence between integer argument and comparison operation is detailed in Table 5-1.

Table 5-1. richcmp comparison operations

In Table 5-1, the integer arguments are compile-time constants declared in the Python runtime object.h header. We can access these constants via a cimport statement, the details of which are covered in Chapter 6.

For example, to support comparisons with an extension type, we would do the following:

from cpython.object cimport Py_LT, Py_LE, Py_EQ, Py_GE, Py_GT, Py_NE

cdef class R:

"""Extension type that supports rich comparisons."""

cdef double data

def __init__(self, d):

self.data = d

def __richcmp__(x, y, int op):

cdef:

R r

double data

# Make r always refer to the R instance.

r, y = (x, y) if isinstance(x, R) else (y, x)

data = r.data

if op == Py_LT:

return data < y

elif op == Py_LE:

return data <= y

elif op == Py_EQ:

return data == y

elif op == Py_NE:

return data != y

elif op == Py_GT:

return data > y

elif op == Py_GE:

return data >= y

else:

assert False

The behavior is as expected:

In [1]: import pyximport; pyximport.install()

Out[1]: (None, <pyximport.pyximport.PyxImporter at 0x101c7d290>)

In [2]: from special_methods import R

In [3]: r = R(10)

In [4]: r < 20 and 20 > r

Out[4]: True

In [5]: r > 20 and 20 < r

Out[5]: False

In [6]: 0 <= r <= 100

Out[6]: True

In [7]: r == 10

Out[7]: True

In [8]: r != 10

Out[8]: False

In [9]: r == 20

Out[9]: False

In [10]: 20 == r

Out[10]: False

Note that if a type supports rich comparisons, then chained comparisons like 0 <= r <= 100 are automatically supported as well.

One last major difference between regular Python and Cython extension types is iterator support.

Iterator Support

To make an extension type iterable, we define __iter__ on it, just as in regular Python. To make an extension type an iterator, we define a __next__ special method on it, as we would in Python 3. This is different from a pure-Python object, where we would define a next method instead. Cython will expose __next__ as next to Python.

A (perhaps contrived) example:

cdef class I:

cdef:

list data

int i

def __init__(self):

self.data = range(100)

self.i = 0

def __iter__(self):

return self

def __next__(self):

if self.i >= len(self.data):

raise StopIteration()

ret = self.data[self.i]

self.i += 1

return ret

Because I defines __iter__, instances of I can be used in for loops:

In [1]: import pyximport; pyximport.install()

Out[1]: (None, <pyximport.pyximport.PyxImporter at 0x101c7e290>)

In [2]: from special_methods import I

In [3]: i = I()

In [4]: s = 0

In [5]: for x in i:

...: s += x

...:

In [6]: s

Out[6]: 4950

Because I defines __next__, instances can be used where an iterator is required:

In [15]: it = iter(I())

In [16]: it.next()

Out[16]: 0

In [17]: next(it)

Out[17]: 1

This covers the primary differences between Cython special methods and their usual semantics in Python. For a full list of special methods, please refer to the relevant sections in Cython’s online documentation.

Summary

The easiest way to create Python extension types, without exception, is through Cython. Trying to do so in straight C via the Python/C API is a useful exercise, but it requires a certain facility with the Python object model and C API that is hard to come by.

Extension types are another instance where Cython melds C-level performance with a Python-like look and feel. A Cython-defined extension type

§ allows easy and efficient access to an instance’s C-level data and methods;

§ is memory efficient;

§ allows control over attribute visibility;

§ can be subclassed from Python;

§ works with existing built-in types and other extension types.

In future chapters we will make use of extension types liberally. In particular, we will cover in Chapters 7 and 8 how to use extension types to wrap C structs, functions, and C++ classes to provide nice object-oriented interfaces to external libraries.