Wrapping C++ Libraries with Cython - Cython (2015)

Cython (2015)

Chapter 8. Wrapping C++ Libraries with Cython

There are only two kinds of languages: the ones people complain about and the ones nobody uses.

— B. Stroustrup

Using Cython to wrap C++ has much in common with using it to wrap C: we must declare the C or C++ interface we want to wrap in an extern block; we must define Python-accessible functions and extension types that wrap the library; and we must convert Python types to and from C or C++ types when Cython cannot apply automatic conversions.

But C++ is a much larger and more complex language than C. To deal with this added complexity and the additional language constructs, Cython has C++-specific syntax to help.

In this chapter, we will cover all of Cython’s C++ wrapping features. Using them, we will learn how to wrap most C++ constructs in Python.

To get an overview, let’s wrap a simple C++ class from end to end.

Simple Example: MT_RNG Class

To extend our example in Chapter 7, suppose we reimplement our random-number generator in a simple C++ class with the following interface:[14]

namespace mtrandom {

const static unsigned int N = 624;

class MT_RNG {

public:

MT_RNG();

MT_RNG(unsigned long s);

MT_RNG(unsigned long init_key[], int key_length);

// initializes RNG state, called by constructors

void init_genrand(unsigned long s);

// generates a random number on [0,0xffffffff]-interval

unsigned long genrand_int32();

// generates a random number on [0,1]-real-interval

double genrand_real1();

private:

unsigned long mt[N];

int mti;

}; // class MT_RNG

} // namespace mtrandom

Cython can only wrap public methods and members; any private or protected methods or members are not accessible, and thus not wrappable.

To declare this class interface for use in Cython, we use an extern block as before. This extern block requires three additional elements to handle C++-isms:

§ Declaring the C++ namespace with the Cython namespace clause

§ Using the cppclass keyword to declare a C++ class interface block

§ Declaring the class’s interface in this block

Because MT_RNG is declared in the mtrandom namespace, we must declare the namespace to Cython in a namepace clause with the cdef extern statement:

cdef extern from "mt19937.h" namespace "mtrandom":

# ...

Inside the extern block, we declare the namespace-level constant integer N, and we use the cppclass keyword to declare the MT_RNG C++ class:

cdef extern from "mt19937.h" namespace "mtrandom":

unsigned int N

cdef cppclass MT_RNG:

# ...

Lastly, inside the MT_RNG class’s declaration we place all public constructors, methods, and data that we wish to access from Cython:

# ...

cdef cppclass MT_RNG:

MT_RNG(unsigned long s)

MT_RNG(unsigned long init_key[], int key_length)

void init_genrand(unsigned long s)

unsigned long genrand_int32()

double genrand_real1()

If there is no namespace, the namespace clause can be omitted. If there are several nested namespaces, we can declare them to Cython as namespace "ns_outer::ns_inner".

NOTE

There can be many cdef extern blocks for each C++ namespace, but only one C++ namespace per cdef extern block. All C++ constructs inside a cdef extern block with a namespace clause must be declared inside that C++ namespace. The namespace clause is required to ensure that Cython generates the proper fully qualified names in the extension module. We do not use the C++ namespace in Cython code.

This suffices to declare the MT_RNG class, allowing us to instantiate it and call its methods from Cython code. To access it from Python, we still need to write Python-accessible functions and extension types that wrap MT_RNG.

The Wrapper Extension Type

The conventional way to wrap a C++ class in Cython is with an extension type. We name it RNG to avoid clashing with the MT_RNG name, although there are ways to allow them to have the same name (see Chapter 6). Typically, a wrapper extension type has a pointer to a heap-allocated instance of the C++ class it is wrapping:

cdef class RNG:

cdef MT_RNG *_thisptr

# ...

NOTE

Storing a pointer to a heap-allocated C++ object in an extension type works in all instances. If the C++ class provides a nullary (no-argument) constructor, we can store a stack-allocated object directly—that is, no pointer indirection required. This removes the need to allocate and delete the instance, and there are efficiency gains as well.

In order for the RNG object to be in a valid state, we need to create and initialize a valid MT_RNG object, requiring a __cinit__ method. Inside it, we use the new operator to create a heap-allocated MT_RNG object:

cdef class RNG:

cdef MT_RNG *_thisptr

def __cinit__(self, unsigned long s):

self._thisptr = new MT_RNG(s)

Cython passes the new operator through to the generated C++ code. The new operator can be used only with C++ classes; the cython compiler will issue a compile-time error if it’s used incorrectly. (We could check for a NULL result, but Cython can automatically convert C++ exceptions; see C++ Exceptions.) The __cinit__ call here uses the first overloaded MT_RNG constructor.

Because every call to new must be matched by a call to delete, we need a __dealloc__ method. Inside it, we call del on self._thisptr, which Cython translates to the C++ delete operator in the generated code:

cdef class RNG:

# ...

def __dealloc__(self):

if self._thisptr != NULL:

del self._thisptr

As we learned in Chapters 5 and 7, __dealloc__ is called once at finalization, when no more references to an RNG instance remain.

That takes care of basic creation, initialization, and finalization. To generate random numbers from Python, we can create simple forwarding cpdef methods for the genrand_int32 and genrand_real1 methods:

cdef class RNG:

# ...

cpdef unsigned long randint(self):

return self._thisptr.genrand_int32()

cpdef double rand(self):

return self._thisptr.genrand_real1()

With these in place, our basic wrapper class is complete.

Compiling with C++

When compiling a C++ project, we need to specify that we are using C++ rather than C, and we need to include all C++ source files for compilation. To do this with a distutils script, we:

§ Add a language = "c++" argument to the Extension instance.

§ Include all C++ source files in the sources list argument.

For example, a minimal setup.py distutils script to compile our RNG.pyx example would look like:

from distutils.core import setup, Extension

from Cython.Build import cythonize

ext = Extension("RNG",

sources=["RNG.pyx", "mt19937.cpp"],

language="c++")

setup(name="RNG",

ext_modules=cythonize(ext))

If we use compiler directives inside RNG.pyx (see Chapter 2), we can simplify the distutils script. At the top of RNG.pyx, we add the following directive comments:

# distutils: language = c++

# distutils: sources = mt19937.cpp

With these directives in place, the cythonize command can extract the necessary information automatically to correctly build the extension. The setup.py script then simplifies to:

from distutils.core import setup

from Cython.Build import cythonize

setup(name="RNG",

ext_modules=cythonize("RNG.pyx"))

To compile our extension, we can use the usual command-line invocation:

$ python setup.py build_ext -i

See Chapter 2 for platform-specific details when invoking the compilation step.

We can also use pyximport to compile this extension module. It necessitates creating an RNG.pyxbld file—not shown here—to instruct pyximport that we are compiling for C++ and tell it which C++ source files to include.

After compiling, we can try out our RNG class from Python.

Using Our Wrapper from Python

We can import the RNG extension module from the default Python interpreter or from IPython:

In [1]: from RNG import RNG

and we can instantiate the RNG class and use its methods:

In [2]: r = RNG(42)

In [3]: r.randint()

Out[3]: 1608637542L

In [4]: r.randint()

Out[4]: 3421126067L

In [5]: r.rand()

Out[5]: 0.9507143117838339

In [6]: r.rand()

Out[6]: 0.1834347877147223

We see that using our random-number generator is high level and straightforward. Using __cinit__ and __dealloc__ in our RNG extension type allows Cython to properly tie allocation and finalization to Python’s reference counting.

This covers the basics of wrapping our Mersenne twister C++ class in Cython. Going deeper, we can also wrap C++-specific features with Cython, starting with function overloading.

Overloaded Methods and Functions

The MT_RNG class has an alternate constructor that takes an array of unsigned longs to initialize the random-number generator’s state. How can we call this from Python?

Because Python does not support overloading methods, it is up to us to emulate overloading by checking argument types and dispatching to the proper C++ constructor inside __cinit__. To call MT_RNG’s alternate constructor, we need to supply an array of unsigned longs and its length. To help with this, we can use the array built-in type from the Python standard library. An array instance has a similar interface to a list, but it requires that all contained elements have the same scalar C type. Cython knows how to work with array objects at both the Python and the C level. In particular, we can grab a pointer to an array’s underlying C array to pass to our C++ MT_RNG class constructor.[15]

To access the built-in array type at the C level, we must use the cimport statement, which is covered in depth in Chapter 6. We first need to add the proper cimport to RNG.pyx:

from cpython.array cimport array

We then modify RNG’s __cinit__ to take either a Python integer or a Python sequence. If the user creates an RNG with an integer argument, we want __cinit__ to call the original constructor:

# ...

def __cinit__(self, seed_or_state):

if isinstance(seed_or_state, int):

self._thisptr = new MT_RNG(seed_or_state)

If a sequence is passed instead, we want to call the second constructor. Before doing so, we must convert the argument to an array:

# ...

def __cinit__(self, seed_or_state):

cdef array state_arr

if isinstance(seed_or_state, int):

self._thisptr = new MT_RNG(seed_or_state)

else:

state_arr = array("L", seed_or_state)

This converts the seed_or_state argument into a Python array of unsigned longs and fails with a runtime exception if the conversion is not possible.

Because we have C-level access to the array object, we can extract its underlying C array of unsigned long integers by using state_arr.data.as_ulongs. Putting it all together, this allows us to dispatch to the second constructor:

# ...

def __cinit__(self, seed_or_state):

# ...

else:

state_arr = array("L", seed_or_state)

self._thisptr = new MT_RNG(state_arr.data.as_ulongs,

len(state_arr))

After recompiling with this improved __cinit__, we can now create an RNG object by passing in either an integer or a sequence of integers:

In [36]: from RNG import RNG

In [37]: r = RNG(42)

In [38]: r.rand()

Out[38]: 0.37454011439684315

In [39]: r2 = RNG(range(30, 40))

In [40]: r2.rand()

Out[40]: 0.04691027990703245

In [41]: r2.randint()

Out[41]: 2626217183L

To wrap overloaded C++ functions, we use a similar pattern. Either we can provide several differently named functions in Python, each calling a different version of the overloaded C++ function, or we can provide a single Python function that does the dispatching, as we did with__cinit__.

The other form of overloading, operator overloading, is also supported by Cython. Because Python also supports overloaded operators, exposing them to Python is much more straightforward.

Operator Overloading

Cython supports most C++ operator overloads. This includes the binary and unary arithmetic operators, the bitwise operators, the Boolean comparison operators, the pre- and post-increment and -decrement operators, the indexing operator (square brackets), and the function call operator (parentheses). Currently, the in-place operators (+=, -=, etc.) are not supported. Some operators are incompatible with Python’s syntax, so Cython provides a special cython.operators magic module to allow Python-compatible access. Table 8-1 gives the full details.

Table 8-1. C++ operators

Operator type

C++ syntax

Notes

Unary and binary arithmetic operators

operator+

operator-

operator*

operator/

operator%

Unary form takes no arguments; binary form takes an rhs. In-place operators not currently supported.

Pre- and post-increment, pre- and post-decrement

operator++()

operator--()

operator++(int)

operator--(int)

No arg indicates pre, int arg indicates post. Must use cython.operator.preincrement to call.

Bitwise operators

operator|

operator&

operator^

operator~

operator<<

operator>>

Bitshift operators often overloaded for input/output.

Dereferencing, comma operators

operator,

operator*()

Must use cython.operator.comma and cython.operator.dereference to access.

Boolean operators

operator==

operator!

operator!=

operator>=

operator<=

operator>

operator<

Indexing, call operators

operator[]

operator()

Cython provides no way to declare the assignment operator operator=; assignment by value is assumed.

Suppose our MT_RNG class implements the function call operator, operator. By calling an MT_RNG instance we get back a random double on the closed [0,1] interval, essentially forwarding to the genrand_real1 method.

We only have to add a single declaration to our cppclass block for MT_RNG:

# ...

cdef cppclass MT_RNG:

# ...

double operator()()

Python, of course, has its own operator overloading syntax. To support calling RNG instances in Python, we implement the __call__ magic method on our RNG extension type:

cdef class RNG:

# ...

def __call__(self):

return self._thisptr[0]()

We cannot say self._thisptr directly, as _thisptr is, of course, a pointer to an MT_RNG object. Cython allows us to use the dot operator on a C or C++ pointer and will automatically convert it to the indirection or arrow operator, ->. Not so for operators: we first dereference the pointer using Cython’s Python-compatible pointer-dereferencing-by-indexing-at-zero [0] syntax, which allows us to then apply operator on it.

Alternatively, we can use the dereference Cython operator from the special cython.operator module (Chapter 3):

from cython.operator cimport dereference as deref

cdef class RNG:

# ...

def __call__(self):

return deref(self._thisptr)()

Using either self._thisptr[0] or deref(self._thisptr) has equivalent semantics when _thisptr is a raw pointer.

After recompiling, we can now use our new operator from Python:

In [1]: from RNG import RNG

In [2]: r = RNG(10)

In [3]: r()

Out[3]: 0.7713206433158649

In [4]: [r() for i in range(3)]

Out[4]: [0.02075194661057367, 0.49458992841993227, 0.6336482317730897]

In some cases C++ operators are implemented as external functions rather than member methods. For instance, suppose the binary + operator for a C++ class C is implemented as:

inline C operator+(C lhs, const C& rhs) {

// ...

}

Cython does not support nonmember operators, but we can simply declare the C operator+(const C& rhs) as if it were a member-defined operator inside the cppclass declaration, in the same way we declared the operator previously. Because Cython does not generate any redeclarations inside a cdef extern block, this bending of the rules will allow us to work around this limitation. By declaring the operator as a class member, Cython sees that C instances support binary addition, even though that addition is implemented as a nonmember function.

C++ Exceptions

Because C++ supports exceptions, Cython has features to detect when they occur and convert them into corresponding Python exceptions automatically. It is not possible, however, to catch C++ exceptions in a Python try/except block, nor is it possible to throw C++ exceptions from Cython.

To enable this functionality, we simply add an except + clause to the function or method declaration that may raise a C++ exception. For instance, to automatically convert a C++ bad_alloc exception into a Python MemoryError, we change the MT_RNG constructor declarations like so:

cdef extern from "mt19937.h" namespace "mtrandom":

cdef cppclass MT_RNG:

MT_RNG(unsigned long s) except +

MT_RNG(unsigned long init_key[], int key_length) except +

This removes the need to check whether the result of a new allocation is NULL; with an except + clause, Cython does the check for us automatically and propagates the exception into Python code.

Cython automatically converts most standard C++ exception types into corresponding Python exception types. The currently supported exceptions and their Python counterparts are in Table 8-2; this list of exceptions may expand or be refined in future releases.

Table 8-2. C++-to-Python exception mapping

C++

Python

bad_alloc

MemoryError

bad_cast

TypeError

domain_error

ValueError

invalid_argument

ValueError

ios_base::failure

IOError

out_of_range

IndexError

overflow_error

OverflowError

range_error

ArithmeticError

underflow_error

ArithmeticError

All others

RuntimeError

The error message is set from the C++ exception’s what method.

To instruct Cython to raise a particular type of Python exception, we can append the Python exception type to the except + clause:

# ...

cdef cppclass MT_RNG:

MT_RNG(unsigned long s) except +MemoryError

MT_RNG(unsigned long init_key[], int key_length) except +MemoryError

# ...

Lastly, a custom exception handler function can be used to do the C++-to-Python exception translation manually. This handler can be defined in C++ or Cython.

To call a cdef function handler whenever a C++ method throws an exception, we would say:

cdef int handler():

# ...

cdef extern from "mt19937.h" namespace "mtrandom":

cdef cppclass MT_RNG:

MT_RNG(unsigned long init_key[], int key_length) except +handler

# ...

If handler does not raise a Python exception, a RuntimeError is raised automatically.

Stack and Heap Allocation of C++ Instances

We’ve already seen how to wrap a simple C++ class in an extension type. This is often the most common use of C++ from Cython, but we can, of course, use the class directly in Cython code without exposing it to Python. For instance, if we need to simply use the MT_RNG class without wrapping it, we can stack-allocate an MT_RNG instance, allowing C++ finalization rules to automatically clean up the stack-allocated instance for us, even in the event of exceptions (i.e., the obscurely named resource-allocation-is-initialization pattern).

To declare and use stack-allocated C++ objects in Cython, we must declare a default constructor for the C++ object in the cdef cppclass block:

cdef extern from "mt19937.h" namespace "mtrandom":

cdef cppclass MT_RNG:

MT_RNG()

void init_genrand(unsigned long s)

# ...

We can now use an MT_RNG object inside a function that makes and returns a list of random values:

def make_random_list(unsigned long seed, unsigned int len):

cdef:

list randlist = [0] * len

MT_RNG rng # calls default constructor

unsigned int i

rng.init_genrand(seed)

for i inrange(len):

randlist[i] = rng.genrand_int32()

return randlist

If there is no nullary constructor, then we cannot use stack-allocated C++ objects in Cython, and we have to use a heap-allocated one. In that case, we need to ensure that we call del on the object (likely in a try/finally block) to ensure it is deleted on the C++ side:

def make_random_list(unsigned long seed, unsigned int len):

cdef:

# ...

MT_RNG *rng

rng = new MT_RNG(seed)

try:

# ...

finally:

del rng

Clearly the stack-allocated version is more convenient, removing the need for the try/finally block to ensure the rng instance is cleaned up.

Besides allocation patterns, subclassing and class hierarchies are important C++ features, and can require some special handling in Cython.

Working with C++ Class Hierarchies

If we want to wrap an MT_RNG subclass named MT_RNGImproved with Cython, there are techniques to handle method overriding.

Suppose our MT_RNG class has a virtual method, serialize, that returns a std::string serialization of the MT_RNG state. Because it is virtual, serialize is meant to be overridden by subclasses, which the MT_RNGImproved subclass does. The virtual keyword is not supported or necessary in Cython, so we leave it out of any method declaration. We can simply declare the serialize method in both the MT_RNG and MT_RNGImproved declarations, and Cython will generate the correct code.

Handling the remaining nonvirtual inherited methods requires more work. Cython’s cppclass declaration does not support subclassing. To work with this limitation, we can handle nonoverridden inherited methods in two ways. We can redeclare the nonvirtual base class methods in the subclass:

cdef extern from "mt19937.h" namespace "mtrandom":

cdef cppclass MT_RNG:

# ...

cdef cppclass MT_RNGImproved:

MT_RNGImproved()

unsigned long genrand_int32()

double genrand_real1()

Or we can explicitly cast a subclass pointer to the base class, thereby accessing the base class’s nonvirtual methods:

cdef MT_RNGImproved *rng = new MT_RNGImproved()

return (<MT_RNG*>rng).genrand_int32()

In either case, Cython will allow us to call a method on an object only if that method is declared explicitly in its type’s interface.

NOTE

When using polymorphism in C++, we must use a pointer to the base class. A pointer to an instance of a subclass can be assigned to the base class’s pointer, which can then be used elsewhere.

Besides interfacing—and wrapping—ordinary C++ classes, Cython also supports templated C++ functions and classes.

C++ Templates

The C++ standard template library (STL) has several templated functions and classes ready for use. We can wrap and use these functions and classes from Cython.

Templated Functions and Cython’s Fused Types

The <algorithm> header declares many fundamental templated functions especially designed to be used on ranges of elements. Two of the simpler templated functions are min and max:

template <class T>

const T& min(const T& a, const T& b);

template <class T>

const T& max(const T& a, const T& b);

How do we declare and use these in Cython?

Declaration is straightforward: we use a cdef extern block as usual. To indicate that these are templated functions, we provide a template parameter name in brackets immediately after the function’s name and before the argument list:

cdef extern from "<algorithm>" namespace "std":

const T max[T](T a, T b) except +

const T min[T](T a, T b) except +

Careful readers will notice that the argument types are declared as non-const value types, and the return types are declared as const values. This code works, since C++ reference variables are passed and returned like values, and reference variables can be assigned to a value-typed variable. Cython currently does not support returning reference types from templated functions, but this support is likely to come in future versions.

Calling min and max from Cython is straightforward. If the templated types can be inferred from the argument type(s), we can call the templated C++ function as if it were nontemplated, which is frequently the case.

If the argument types are ambiguous, we can add brackets after the function name, filling in the specific type to use for the template parameter or parameters.

The cleanest way to wrap these functions is to declare their interface in a definition file, the details of which are covered in Chapter 6. Supposing we put the previous declarations in a definition file _algorithm.pxd, we can access the C++ min and max via the _algorithm Cython namespace.

Fused types (Chapter 3) are ideal for wrapping templated functions such as these:

cimport cython

cimport _algorithm

ctypedef fused long_or_double:

cython.long

cython.double

def min(long_or_double a, long_or_double b):

return _algorithm.min(a, b)

def max(long_or_double a, long_or_double b):

return _algorithm.max(a, b)

By using a long_or_double fused type that includes the Python-compatible numeric types of interest, we make min and max generic templated Cython functions, providing a clean interface. Cython automatically dispatches to the right function specialization when min or max is called from Python.

This covers the basics of declaring, using, and wrapping templated functions; declaring and using templated classes follows a similar pattern.

Templated Classes

Perhaps the most widely used STL container is vector: it is the workhorse container for many C++ algorithms. How do we declare and use it in Cython?

To declare a templated class like vector, we use a cdef extern block in conjunction with a cppclass declaration, as for a nontemplated class. To indicate that the class is templated, we place template parameters in brackets after the class name:

cdef extern from "<vector>" namespace "std":

cdef cppclass vector[T]:

vector() except +

vector(vector&) except +

vector(size_t) except +

vector(size_t, T&) except +

T& operator[](size_t)

void clear()

void push_back(T&)

We use T as the template type, and have declared four of vector’s constructors along with a few of vector’s more common methods. If there is more than one template parameter, we put a comma-separated list of unique parameter names in the brackets.

Suppose we want to declare and use a vector of ints inside a wrapper function. For templated classes, we are required to instantiate them with a specific templated type in brackets after the templated class name:

def wrapper_func(elts):

cdef vector[int] v

for elt inelts:

v.push_back(elt)

# ...

This works for a stack-allocated vector, but creating a heap-allocated vector requires the new operator:

def wrapper_func(elts):

cdef vector[int] *v = new vector[int]()

# ...

When heap-allocating with new, we need to ensure that we call del on the vector pointer when we’re finished using it to prevent memory leaks.

Iterators and Nested Classes

The C++ STL uses the iterator pattern everywhere, and vectors are no exception. To use the vector’s iterator from Cython, we declare the vector’s internal iterator as an internal cppclass:

cdef extern from "<vector>" namespace "std":

cdef cppclass vector[T]:

# ...

cppclass iterator:

T& operator*()

iterator operator++()

iterator operator--()

iterator operator+(size_t)

iterator operator-(size_t)

bint operator==(iterator)

bint operator!=(iterator)

bint operator<(iterator)

bint operator>(iterator)

bint operator<=(iterator)

bint operator>=(iterator)

Suppose we want to rotate a Python list in place by shifting n elements left and putting the shifted elements on the end. The STL has a rotate templated function declared in <algorithm> for just this purpose. We need to pass an std::vector<T>::iterator to indicate the beginning, middle, and end of the vector to rotate. The element pointed to by the middle iterator is rotated to the front of the resulting list.

First we need to declare std::rotate to Cython:

cdef extern from "<algorithm>" namespace "std":

void rotate[iter](iter first, iter middle, iter last)

We place this declaration in our _algorithm.pxd file as before.

Because rotate does not care about the values in the container object being rotated, we can simply create a vector of void pointers that point to the Python list’s contents and use that in our call to _algorithm.rotate.

First, the vector initialization:

def rotate_list(list ll, int rot):

cdef vector[void*] vv

for elt inll:

vv.push_back(<void*>elt)

We iterate through our Python list and initialize our vv vector, casting each element to a void pointer. Note that both the Python list ll and the C++ vector vv share references to the same underlying Python objects.

The rotate_list function’s second argument is the number of elements to rotate by. It can be either positive or negative, and is normalized to a positive value here:

def rotate_list(list ll, int rot):

# ...

if rot < -len(ll) orrot >= len(ll):

raise IndexError()

rot = (rot + len(ll)) % len(ll)

For convenience, let’s declare a ctypedef to make the iterator type more succinct:

ctypedef vector[void*].iterator vvit

Now the call to _algorithm.rotate is straightforward:

def rotate_list(list ll, int rot):

# ...

_algorithm.rotate[vvit](vv.begin(), vv.begin()+rot, vv.end())

Lastly, we create a new list out of the vector’s contents, casting back to Python objects:

def rotate_list(list ll, int rot):

# ...

return [<object>o for o invv]

The entire function is only eight lines of code, three of which are declaration and error checking. After compiling, we can try it out from Python:

In [1]: import wrap_funcs

In [2]: wrap_funcs.rotate_list(range(10), 5)

Out[2]: [5, 6, 7, 8, 9, 0, 1, 2, 3, 4]

It is remarkable that Cython makes possible such a fluid mix of Python and templated C++, all while retaining a Python-like look and feel.

Now that we have some familiarity with interfacing with templated C++ classes and iterators, let’s look at interfacing with the STL. Cython makes this particularly easy.

Included STL Container Class Declarations

Cython includes built-in definition files for several STL classes, primarily containers:

§ string

§ vector

§ map

§ set

§ unordered_map

§ unordered_set

§ pair

§ list

§ queue

§ priority_queue

§ deque

§ stack

To access any of these class declarations, we use the cimport statement with the libcpp package–like Cython namespace, as covered in detail in Chapter 6:

from libcpp.vector cimport vector

cdef vector[int] *vec_int = new vector[int](10)

The libcpp package’s contents are located in the Cython/Includes/libcpp directory included with the Cython source distribution. If we are using any of these templated classes, it is worthwhile to look at the definition file to know exactly the interface Cython exposes.

For example, we can build up a std::map of element names to their atomic numbers in Cython as follows:

from libcpp.string cimport string

from libcpp.map cimport map

from libcpp.pair cimport pair

def periodic_table():

cdef map[string, int] table

cdef pair[string, int] entry

# Insert Hydrogen

entry.first = b"H"; entry.second = 1

table.insert(entry)

# Insert Helium

entry.first = b"He"; entry.second = 2

table.insert(entry)

# ...

Cython automatically converts std::map and other STL containers to and from their Python analogues. We can use this to easily assign a Python dict to a std::map, for example. It also allows us to return a std::map from a def function—Cython automatically copies the std::map’s contents to a new Python dict and returns that. These conversions copy the underlying data, and are triggered when we assign (or cast) from a statically typed Python container to a C++ container type, and vice versa.

Table 8-3 lists all currently supported built-in conversions from Python to C++ containers.

Table 8-3. Python to C++ containers

From Python type

To C++ type(s)

bytes, str, unicode

string

mapping (dict)

map, unordered_map

iterable

set, unordered_set

iterable

vector, list

length two iterable

pair

Table 8-4 lists the allowed conversions from C++ to Python.

Table 8-4. C++ to Python containers

From C++ type

To Python type

string

bytes, str, unicode

map, unordered_map

dict

set, unordered_set

set

vector, list

list

pair

tuple

NOTE

The automatic conversions to and from the Python string types—bytes, str, and unicode—are influenced by the c_string_type and c_string_encoding compiler directives (see Chapters 2 and 3). If neither of these directives is set, then only the bytes type is convertible to and from the std::string type by default.

All conversions are recursive, so a std::map<std::pair<int, int>, std::vector<std::string> > converts to a Python dict with tuple keys of ints and list values of bytes objects.

This powerful feature allows us to return a supported C++ container directly from a def or cpdef function or method, provided the container and its templated type are supported. Cython automatically converts the container’s contents to the right Python container.

Previous examples, such as the periodic_table function that inserts elements into a std::map, can be more simply expressed:

from libcpp.string cimport string

from libcpp.map cimport map

def periodic_table():

cdef map[string, int] table

table = {"H": 1, "He": 2, "Li": 3}

# ...use table in C++...

return table

In this example, assigning a dictionary literal to table automatically converts all key/value pairs to the corresponding C++ std::pair type and stores them in the std::map instance. The complement works as well: returning table converts the std::map<string, int> to a Python dictionary.

Automatic conversions also simplify working with std::vector objects: assigning a Python list to a statically typed vector is much easier than iterating through the list and calling push_back for each element.

Cython also knows how to use standard C++ container objects when an iterable is required—in for loops, list comprehensions, and the like. For this to work, the C++ object must have begin and end methods that return a pointer-like iterator, which is the case for most STL containers. This removes the need to declare and work with C++ iterators explicitly in many situations, and makes working with C++ containers feel like Python.

For example, calling std::sort with the contents of a Python list is simple. First we cimport from libcpp.vector and declare the std::sort templated function:

from libcpp.vector cimport vector

cdef extern from "<algorithm>" namespace "std":

void std_sort "std::sort" [iter](iter first, iter last)

With this in place, the actual sorting function is just three lines:

def sort_list(list ll):

cdef vector[int] vv = ll

std_sort[vector[int].iterator](vv.begin(), vv.end())

return vv

This example serves to demonstrate how straightforward Cython makes conversions between Python and C++ containers, and how easy it is to call into a C++ STL function. It is not intended to demonstrate how to sort a list: the right way to do that, of course, is to use the list.sortmethod or the sorted built-in function.

Memory Management and Smart Pointers

Many C++ libraries use smart pointers, for the many advantages they provide beyond C-style raw pointers. They help clarify and enforce pointer ownership semantics, prevent memory and resource leaks, and simplify memory management when we are dealing with C++ exceptions. Of particular relevance to Python is the shared_ptr smart pointer, which supports basic reference counting. As we know, CPython (and Cython, by extension) also uses reference counting for the majority of its memory management of Python objects. Can we get C++ shared pointers to work nicely with Python reference counting in Cython? To quote a well-known political figure’s campaign slogan, “Yes we can!”

First, let’s declare the smart_ptr template class interface to Cython. We use the declarations from the Boost C++ library, but the C++11 version is very similar:

cdef extern from "boost/smart_ptr/shared_ptr.hpp" namespace "boost":

cdef cppclass shared_ptr[T]:

shared_ptr()

shared_ptr(T *p)

shared_ptr(const shared_ptr&)

long use_count()

T operator*()

Here we have declared that boost::shared_ptr has a single template parameter, used for the type of object pointed to. It has a default constructor, a single-argument constructor, and a copy constructor. Besides these, we declare the use_count method to report the number of reference counts on this shared pointer instance, and operator* to allow us to dereference a shared pointer.

To illustrate working with shared pointers, suppose we have an externally defined function, histogram, that takes a std::vector<int> argument and returns a shared pointer to a vector of integers, which is the number of integers with that value in the input vector. This can arise when a library uses shared pointers to allow objects to share ownership of large containers.

Say also that we want to get the average count in the histogram vector from Python. Using our libcpp.vector and shared_ptr template class declarations, we can define a def function, hist_sum. First, we need to get our shared pointer to a vector of integers:

from libcpp.vector cimport vector

def hist_sum(args):

cdef:

shared_ptr[vector[int]] ptr_hist = histogram(args)

# ...

Now that we have our shared pointer, we can dereference it to access the underlying vector. We need to use cython.operator.dereference to do so, since the shared_ptr does not support indexing with operator[]:

from cython.operator cimport dereference as deref

def hist_sum(args):

cdef:

shared_ptr[vector[int]] ptr_hist = histogram(args)

vector[int] hist = deref(ptr_hist)

# ...

We now can walk through the hist vector to get the average count:

def hist_sum(args):

cdef:

shared_ptr[vector[int]] ptr_hist = histogram(args)

vector[int] hist = deref(ptr_hist)

double weighted_sum = 0.0

int elt, n_total = 0

for idx, elt inenumerate(hist):

weighted_sum += idx * elt

n_total += elt

return weighted_sum / n_total

The nice part about this function is that we are working with a pointer to a vector, but we do not have to worry about memory leaks or who is responsible for cleaning it up. The shared pointer handles that automatically for us, even if exceptions occur.

We can also use smart pointers as the attributes inside extension types. This is useful if we want to share our C++ objects between Python and other C++ code that uses shared pointers.

For example, suppose we want to wrap a C++ vector of integers in an extension type and make it look like a Python list. First, we declare the vector attribute:

cdef class Vector:

cdef shared_ptr[vector[int]] _thisptr

The __cinit__ method just creates an empty vector inside a shared_ptr:

cdef class Vector:

cdef shared_ptr[vector[int]] _thisptr

def __cinit__(self):

self._thisptr = shared_ptr[vector[int]](new vector[int]())

To make our Vector act like a Python list, we can add some def methods. Every time we want to work with the underlying vector, we need to dereference the _thisptr attribute:

from cython.operator cimport dereference as deref

cdef class Vector:

# ...

def __len__(self):

return deref(self._thisptr).size()

def __getitem__(self, int index):

return deref(self._thisptr)[index]

def __setitem__(self, int index, int i):

return deref(self._thisptr)[index] = i

def append(self, int i):

deref(self._thisptr).push_back(i)

def __repr__(self):

return repr([i for i inderef(self._thisptr)])

We place Vector’s definition in vector.pyx and compile it into an extension module. It is list-like enough to allow us to shuffle a Vector from Python:

from vector import Vector

from random import shuffle

v = Vector()

for i inrange(20):

v.append(i)

shuffle(v)

print v

When running our script test_vector.py, we see everything hangs together:

$ python test_vector.py

[19, 1, 15, 13, 12, 18, 8, 2, 16, 4, 3, 14, 17, 11, 10, 9, 0, 6, 5, 7]

To take this example further, we could implement a sort method that uses C++’s std::sort function. Doing so is left as an exercise for the reader.

Because the _thisptr for Vector is a shared pointer, Vector instances can share ownership of the underlying std::vector<int> with C++. This means that Python objects can work with C++ objects in a nice and unobtrusive way, avoiding expensive copies, removing ambiguities regarding pointer ownership, and allowing the two languages’ reference counting systems to work together.

Summary

This chapter covered all of Cython’s current C++ interfacing features. We learned how to

§ declare C++ namespaces, classes, and global constants;

§ make an extension type to wrap a C++ class;

§ use the new and del operators properly to work with C++ memory management;

§ compile C++-based Cython projects;

§ work with overloaded constructors, methods, functions, and operators;

§ easily propagate C++ exceptions to Python;

§ manage stack-allocated C++ objects;

§ work with C++ type hierarchies;

§ declare and use templated functions and classes;

§ use included C++ STL container definition files.

Cython’s C++ support is continually improving and stabilizing. It is expected that some of the more manual tasks in this chapter will be better supported in future releases.


[14] To follow along with the examples in this chapter, please see https://github.com/cythonbook/examples.

[15] We could use a NumPy array rather than the built-in array type. We choose the array type here because it is simple to use and does not introduce an external dependency. Cython’s support for NumPy arrays is covered in Chapter 10.