Cython (2015)

Chapter 2. Compiling and Running Cython Code

I was taught that the way of progress was neither swift nor easy.

— M. Curie

It’s not that I’m so smart, it’s just that I stay with problems longer.

— A. Einstein

One of the more significant differences between Python and C or C++ is that Python is interpreted while C and C++ are compiled. When developing a Python program, we can immediately run code after making a modification, while C and C++ require an intervening compilation step. Compiling a large C or C++ code base can take hours or days. Using Python can allow much more rapid development, leading to a significant productivity boost.

Like C and C++, Cython requires a compilation step before the source can be run. This compilation step can be explicit or implicit. Both modes have their uses. One nice feature of automatically compiling Cython is that it makes working with Cython feel like working with pure Python. Whether compiling explicitly or implicitly, because Cython can be applied selectively to small sections of a Python code base, Cython’s compilation requirement can be minimized.

This chapter will cover the various ways to compile Cython code so that it can be run by Python. There are several options:

§ Cython code can be compiled and run interactively from an IPython interpreter.

§ It can be compiled automatically at import time.

§ It can be separately compiled by build tools like Python’s distutils.

§ It can be integrated into standard build systems such as make, CMake, or SCons.

These options allow us to adapt Cython to particular use cases, from quick interactive exploration on one end to building for the ages on the other.

NOTE

It is not necessary to know all the methods to compile Cython code, so this chapter can be read piecemeal.

In all cases, each method passes Cython code through two compilation stages to generate a compiled module that Python can import and use. Before we cover the particulars of each compilation method, it is helpful to understand what is going on in this pipeline.

The Cython Compilation Pipeline

Because the Cython language is a superset of Python, the Python interpreter cannot import and run it directly. So how do we get from Cython source to valid Python? Via the Cython compilation pipeline.

The pipeline’s job is to transform Cython code into a Python extension module that can be imported and used by the Python interpreter. This pipeline can be run either automatically, without user involvement (making Cython feel much like Python), or explicitly by the end user when more control is required.

NOTE

Cython has a pure-Python mode, which allows the user to bring in Cython-specific declarations in a way that remains valid Python syntax. Code developed in pure-Python mode is more verbose, but can be run directly by the Python interpreter (with no Cython speed improvement) or compiled by Cython. We do not cover pure-Python mode here, leaving its treatment to the online documentation.

The pipeline comprises two stages. The first stage is handled by the cython compiler, which transforms Cython source into optimized and platform-independent C or C++. The second stage compiles the generated C or C++ source into a shared library with a standard C or C++ compiler. The resulting shared library is platform dependent. It is a shared-object file with a .so extension on Linux or Mac OS X, and is a dynamic library with a .pyd extension on Windows. The flags passed to the C or C++ compiler ensure this shared library is a full-fledged Python module. We call this compiled module an extension module, and it can be imported and used as if it were written in pure Python.

Nearly all the complexity of these stages is managed by the tools we will describe in the rest of this chapter. We rarely have to think about what is going on when the compilation pipeline is running, but it is good to keep these stages in mind as we go through the following sections.

NOTE

The cython compiler is a source-to-source compiler, and the generated code is highly optimized. It is not uncommon for Cython-generated C code to be faster than typical hand-written C. When the author teaches Cython, students often write C equivalents to Cython’s code; the Cython version is nearly always faster, and—for equivalent algorithms—is never slower. Cython’s generated C code is also highly portable, supporting all common C compilers and many Python versions simultaneously.

Installing and Testing Our Setup

Now that we know about the two stages in the compilation pipeline, we need to ensure that we have both a C (or C++) compiler and the cython compiler installed and working. Different platforms have different options.

C and C++ compilers

Linux

Refer to the documentation for the distribution’s package manager (for example, yum for RedHat-based distros, apt-get for Debian-based, etc.) for how to install GCC and the Python development package (often called python-dev, or some variant) to acquire the Python headers.

Mac OS X

Install the free OS X developer tools via Xcode; this provides a GCC-like compiler.

Windows

The recommended compiler to use on Windows is Visual Studio, the same version used to compile the Python runtime. If this is not available, then one alternative is the Microsoft-provided minimal Visual C++ compiler for compiling extensions for Python 2.7. Another good alternative is to use the Windows SDK C/C++ compiler. These compilers are the only reliable options for compiling 64-bit extensions. Another option on Windows for 32-bit extensions is to use MinGW. It is not as reliable as the Microsoft-provided compilers, but will likely work for simple examples. The MinGW compiler is distributed via several prepackaged Python software distributions, mentioned in the next section.

Installing Cython

Likely the easiest way to acquire Cython is via a packaged software distribution, such as these popular options:

§ The Sage Mathematics software system

§ Enthought’s Canopy

§ Anaconda, from Continuum Analytics

§ The GPL-licensed and Windows-centric Python(x,y)

Being prepackaged, these options are likely to lag one or two releases behind the most up-to-date version of Cython.

To use the most recent version of Cython, we can install from source. This requires a working C or C++ compiler; see the previous section for details. Likely the easiest way to install from source is via pip, which is commonly available via the listed package distributions and is now distributed with Python (version 3.4) itself:

$ pip install cython

Another option is to download the Cython source code. From the Cython source directory, run:

$ python setup.py install

Once installed—whether via a software distribution or compiled by hand—the cython compiler will be available from the command line:

$ cython -V

Cython version 0.20.2

Once we have a C compiler and the cython compiler in place, we are ready to follow along with the distutils and pyximport sections in this chapter.

Additionally, we will need to have IPython installed to use Cython from within IPython. The packaged distributions include IPython, or we can use pip to install it.

The Standard Way: Using distutils with cythonize

Python’s standard library includes the distutils package for building, packaging, and distributing Python projects. The distutils package has many features; of interest to us is its ability to compile C source into an extension module, the second stage in the pipeline. It manages all platform, architecture, and Python-version details for us, so we can use one distutils script and run it anywhere to generate our extension module.

What about the first pipeline stage? That is the job of the cythonize command, which is included with Cython: it takes a Cython source file (and any other necessary options) and compiles it to a C or C++ source file, and then distutils takes it from there.

By using Python’s distutils module combined with Cython’s cythonize command, we have explicit control over the compilation pipeline. This approach requires that we write a small Python script and run it explicitly. It is the most common way for Python projects to compile and distribute their Cython code to end users.

Our distutils Script

For example, consider the fib.pyx Cython source code from Chapter 1. Our goal is to use distutils to create a compiled extension module—fib.so on Mac OS X or Linux, and fib.pyd on Windows.

We control the behavior of distutils through a Python script, typically named setup.py. A minimal setup.py script for compiling the fib.pyx source file into an extension module is just a few lines long, two of which are imports:^[3^]

from distutils.core import setup

from Cython.Build import cythonize

setup(ext_modules=cythonize('fib.pyx'))

The core of the script is in the setup(cythonize(...)) nested calls. The cythonize function in its simplest usage converts Cython source to C source code by calling the cython compiler. We can pass it a single file, a sequence of files, or a glob pattern that will match Cython files.

NOTE

The cythonize command returns a list of distutils Extension objects that the setup function knows how to turn into Python extension modules. It is designed to make distutils easier to use for Cython projects.

The cythonize command has several other options; see its docstring for details.

Compiling with distutils on Mac OS X and Linux

These two function calls succinctly demonstrate the two stages in the pipeline: cythonize calls the cython compiler on the .pyx source file or files, and setup compiles the generated C or C++ code into a Python extension module.

It is a simple matter to invoke this setup.py script from the command line:

$ python setup.py build_ext --inplace

The build_ext argument is a command instructing distutils to build the Extension object or objects that the cythonize call created. The optional --inplace flag instructs distutils to place each extension module next to its respective Cython .pyx source file.

NOTE

To get the full list of options that the build_ext subcommand supports, we can run:

$ python setup.py build_ext --help

Other options allow us to control the preprocessor, include directories, link directories, and link libraries.

The output from this command will look different on Mac OS X, Linux, and Windows—that’s distutils doing its job for us and handling the platform-specific aspects of the compilation.

For instance, on Mac OS X we will see something like the following, with slight modifications based on our Python version, OS version, architecture, and so on:

$ python setup.py build_ext -i

Compiling fib.pyx because it changed.

Cythonizing fib.pyx

running build_ext

building 'fib' extension

creating build

creating build/temp.macosx-10.4-x86_64-2.7

gcc -fno-strict-aliasing -fno-common -dynamic -g -O2

-DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes

-I/Users/ksmith/Devel/PY64/Python.framework/Versions/2.7/include/python2.7

-c fib.c -o build/temp.macosx-10.4-x86_64-2.7/fib.o

gcc -bundle -undefined dynamic_lookup

build/temp.macosx-10.4-x86_64-2.7/fib.o

-o /Users/ksmith/fib.so

The line Cythonizing fib.pyx is where the call to the cython compiler takes place. If we have a syntax error or otherwise invalid Cython code in fib.pyx, the cython compiler will print out a helpful message and stop things at this step. There are two calls to gcc: the first compiles the generated fib.c code into an object file, and the second links this object file into a Python extension module, fib.so. If the calls are successful, we should see in this directory the generated fib.c source file, the compiled extension module fib.so, and a directory named build with the intermediate build products.

Compiling with distutils on Windows

On Windows we may need to add extra arguments, depending on which compiler we are using and a few other factors:

C:\Users\ksmith> python setup.py build_ext -i --compiler=mingw32 -DMS_WIN64

Compiling fib.pyx because it changed.

Cythonizing fib.pyx

running build_ext

building 'fib' extension

creating build

creating build\temp.win-amd64-2.7

creating build\temp.win-amd64-2.7\Release

C:\Users\ksmith\gcc.exe -mno-cygwin -mdll -O -Wall

-DMS_WIN64=1 -IC:\Users\ksmith\include -c fib.c

-o build\temp.win-amd64-2.7\Release\fib.o

writing build\temp.win-amd64-2.7\Release\fib.def

C:\Users\ksmith\gcc.exe -mno-cygwin -shared -s

build\temp.win-amd64-2.7\Release\fib.o

build\temp.win-amd64-2.7\Release\fib.def

-LC:\Users\ksmith\libs -LC:\Users\ksmith\amd64

-lpython27 -lmsvcr90 -o C:\Users\ksmith\fib.pyd

Here we use the mingw compiler to compile the fib.pyd extension module. Because this Python interpreter is a 64-bit executable, we add an extra -DMS_WIN64 flag to compile in 64-bit mode. Otherwise the steps are the same, with different output that is specific for Windows. The result is the fib.pyd extension module, and usage is identical to the Mac OS X version.

If using a different Windows compiler, like Visual Studio or the SDK C/C++ compiler, we should set the compiler flag to msvc and can remove the -DMS_WIN64 flag:

C:\Users\ksmith> python setup.py build_ext -i --compiler=msvc

...

Consult the linked documentation for details.

Using Our Extension Module

Whether on Mac OS X, Linux, or Windows, once we have compiled our extension module, we can bring up our Python or IPython interpreter and import the fib module:

$ ipython --no-banner

In [1]: import fib

If no ImportError is raised, then the compilation was likely successful.

We can use IPython’s handy introspection features to provide more details about our extension module:

In [2]: fib?

Type: module

String Form:<module 'fib' from 'fib.so'>

File: /Users/ksmith/fib.so

Docstring: <no docstring>

Putting a single ? after an object instructs IPython to tell us what it knows about the object.

We can also inspect the fib.fib function we created:

In [3]: fib.fib?

Type: builtin_function_or_method

String Form:<built-in function fib>

Docstring: Returns the nth Fibonacci number.

Notice that the docstring we defined in Cython shows up in our interactive session. Our fib function is a builtin_function_or_method; that is one way we can tell that this function is implemented in compiled code rather than in straight Python. It is a full-fledged Python function, though.

To really test things out, let’s call fib.fib:

In [4]: fib.fib(90)

Out[4]: 2.880067194370816e+18

When using Cython to wrap C and C++ code, which we will cover in detail in Chapters 7 and 8, we must include other source files in the compilation step.

For example, consider the distutils script setup_wrap.py that compiles the cfib.c wrappers from Chapter 1:

from distutils.core import setup, Extension

from Cython.Build import cythonize

# First create an Extension object with the appropriate name and sources.

ext = Extension(name="wrap_fib", sources=["cfib.c", "wrap_fib.pyx"])

# Use cythonize on the extension object.

setup(ext_modules=cythonize(ext))

This distutils script requires one extra step to wrap an external library: we create an Extension object with all C and Cython sources listed and passed in the sources argument. We then pass this Extension object to cythonize, and cythonize and the setup command ensure that the cfib.c file is compiled into the resulting extension module.

If we are provided a precompiled dynamic library libfib.so rather than source code, we can instruct distutils to link against libfib.so at link time:

from distutils.core import setup, Extension

from Cython.Build import cythonize

ext = Extension(name="wrap_fib",

sources=["wrap_fib.pyx"],

library_dirs=["/path/to/libfib.so"],

libraries=["fib"])

setup(ext_modules=cythonize(ext))

Here we name only wrap_fib.pyx in the sources argument list, and add a library_dirs and a libraries argument to our Extension object with the appropriate values. For more details on all options that the distutils Extension object supports, please see Python’s official documentation.

Interactive Cython with IPython’s %%cython Magic

Using distutils to compile Cython code gives us full control over every step of the process. The downside to using distutils is it requires a separate compilation step and works only with .pyx source files—no interactive use allowed. This is a definite disadvantage, as one of Python’s strengths is its interactive interpreter, which allows us to play around with code and test how something works before committing it to a source file. The IPython project has convenient commands that allow us to interactively use Cython from a live IPython session.

These extra commands are IPython-specific commands called magic commands, and they start with either a single (%) or double (%%) percent sign. They provide functionality beyond what the plain Python interpreter supplies. IPython has several magic commands to allow dynamic compilation of Cython code, which we cover here.

Before we can use these magic Cython commands, we first need to tell IPython to load them. We do that with the %load_ext metamagic command from the IPython interactive interpreter, or in an IPython notebook cell:

In [12]: %load_ext cythonmagic

There will be no output if %load_ext is successful, and IPython will issue an error message if it cannot find the Cython-related magics.^[4^]

Now we can use Cython from IPython via the %%cython magic command:

In [13]: %%cython

...: def fib(int n):

...: cdef int i

...: cdef double a=0.0, b=1.0

...: for i in range(n):

...: a, b = a+b, a

...: return a

...:

In [14]:

The %%cython magic command allows us to write a block of Cython code directly in the IPython interpreter. After exiting the block with two returns, IPython will take the Cython code we defined, paste it into a uniquely named Cython source file, and compile it into an extension module. If compilation is successful, IPython will import everything from that module to make the fib function available in the IPython interactive namespace. The compilation pipeline is still in effect, but it is all done for us automatically.

We can now call the fib function we just defined:

In [14]: fib(90)

Out[14]: 2.880067194370816e+18

NOTE

The %%cython magic command recognizes when it has already compiled an identical code block, in which case it bypasses the compilation step and loads the precompiled block directly.

There may be a pause after we press return when ending a new Cython code block and before the next input prompt appears: that is IPython compiling and loading the code block behind the scenes.

We can always inspect the generated source file if necessary. It is located in the $IPYTHONDIR/cython directory (~/.ipython/cython on an OS X or *nix system). The module names are not easily readable because they are formed from the md5 hash of the Cython source code, but all the contents are there.

We can pass optional arguments to the %%cython magic command. The first set of options control the cython compilation stage:

-n, --name

Specifies the name of the generated .pyx file

--cplus

Instructs cython to generate C++ source

-a, --annotate

Instructs cython to output an annotated source file (see Chapter 9)

-f, --force

Forces cython to regenerate C or C++ source

The second set of options allows us to control the second pipeline stage:

-I, --include

Adds extra directories to search for file inclusions and cimports

-c, --compile-args

Allows inclusion of extra C compiler arguments

--link-args

Allows inclusion of extra link arguments

-L

Adds extra library search directories

-l

Adds extra library names to link against

There are other Cython magic commands that are loaded by %load_ext cythonmagic: the %%cython_inline command and the %%cython_pyximport command. These are not as widely used as the %%cython magic command, which is sufficient for quick interactive use and exploration. The %%cython_inline command—as suggested by its name—simply compiles and runs Cython code embedded in the current Python namespace.

Similarly, %%cython_pyximport builds on the pyximport package that comes with Cython, so we’ll defer its discussion until the next section.

Compiling On-the-Fly with pyximport

Because Cython is Python-centric, it is natural to want to work with Cython source files as if they were regular, dynamic, importable Python modules. Enter pyximport: it retrofits the import statement to recognize .pyx extension modules, sends them through the compilation pipeline automatically, and then imports the extension module for use by Python.

Let’s see an example. The pyximport module comes with Cython, and requires just two statements to get it up and running:

import pyximport

pyximport.install() # .install() called before importing

# Cython extension modules.

We can use pyximport in an interactive IPython session to compile and load our familiar fib.pyx example. First, we bring in pyximport itself:

In [1]: import pyximport

In [2]: pyximport.install()

Out[2]: (None, <pyximport.pyximport.PyxImporter at 0x101548a90>)

With pyximport installed, we can import fib as if it were fib.py, and pyximport compiles it automatically:

In [3]: import fib

Let’s check the __file__ attribute:

In [4]: fib.__file__

Out[4]: '/Users/ksmith/.pyxbld/lib.macosx-10.4-x86_64-2.7/fib.so'

Everything else checks out, and we can run fib.fib as before:

In [5]: type(fib)

Out[5]: module

In [6]: fib.fib(90)

Out[6]: 2.880067194370816e+18

For simple cases like this example, using pyximport removes the need to write a setup.py distutils script, and we can treat fib.pyx as if it were a regular Python module. If a Cython source file is modified, pyximport automatically detects the modification and will recompile the source file the next time it is imported in a new Python interpreter session.

Because Cython modules imported via pyximport depend on both the cython compiler and a properly set up C compiler, it tends not to be used in production environments where these dependencies are not under our control.

Controlling pyximport and Managing Dependencies

The pyximport package also handles more complex use cases. For instance, what if a Cython source file depends on other source files, such as C or C++ source or header files, or other Cython source files? In this case, pyximport needs to recompile the .pyx file if any of its dependencies have been updated, regardless of whether the .pyx file itself has changed. To enable this functionality, we add a file with the same base name as the .pyx source file and with a .pyxdeps extension in the same directory as the Cython source file. It should contain a listing of files that the .pyx file depends on, one file per line. These files can be in other directories relative to the directory of the .pyxdeps file. The entries can also be glob patterns that match multiple files at once. If a .pyxdeps file exists, pyximport will read it at import time and compare the modification time of each listed file with the modification time of the .pyx file being imported. If any file that matches a pattern in the .pyxdeps file is newer than the .pyx file, then pyximport will recompile on import.

The .pyxdeps file is nice to communicate file dependencies to pyximport, but how do we tell pyximport to compile and link several source files into one extension module? That role is filled by a .pyxbld file: its purpose is to customize pyximport for this and other use cases. Like.pyxdeps, a .pyxbld file has the same base name as its Cython source file and replaces the .pyx extension with .pyxbld. It should be located in the same directory as the .pyx file being imported.

What goes inside a .pyxbld file? One or two Python functions, each optional:

make_ext(modname, pyxfilename)

If defined, the make_ext function is called with two string arguments before compilation. The first argument is the name of the module, and the second is the name of the .pyx file being compiled. It returns a distutils.extension.Extension instance, or (equivalently) it can return the result of a call to Cython.Build.cythonize. This allows the user to customize the Extension being used. By adding files to the sources argument when creating an Extension instance, it instructs pyximport to compile external source files and link them with the compiled.pyx file when creating the extension module. See the following example.

make_setup_args

If defined, pyximport calls this function with no arguments to get an extra argument dictionary to pass to distutils.core.setup. This allows the user to control the setup arguments passed in, which provides full control over distutils.

pyximport Example with External Dependencies

For example, suppose we want to wrap an external Fibonacci implementation in C. Two C files are defined, _fib.h and _fib.c. Our fib.pyx file has a cdef extern from "_fib.h" block and a minimal Python wrapper function to call the C implementation of the Fibonacci function. We can set up pyximport to work with this configuration by creating a fib.pyxdeps file that contains one line:

_fib.*

This glob pattern will match both _fib.c and _fib.h, so pyximport will recompile fib.pyx whenever either of these files changes. We can instruct pyximport to compile and link _fib.c together with fib.pyx into an extension module by creating a fib.pyxbld file that defines make_ext:

def make_ext(modname, pyxfilename):

from distutils.extension import Extension

return Extension(modname,

sources=[pyxfilename, '_fib.c'],

include_dirs = ['.'])

The essential line is the sources=[...] argument. It tells distutils to compile _fib.c with fib.pyx and link everything together. The include_dirs argument tells distutils to look in the current directory for the _fib.h header file.

We can import fib.pyx as before, and now it will wrap an external C function. If any of fib.pyx, _fib.h, or _fib.c is changed, pyximport will detect it and recompile everything the next time it is used in a new interpreter session.

Rolling Our Own and Compiling by Hand

For the sake of completeness, suppose we want to create an extension module starting with our fib.pyx source file, without using distutils, IPython’s magic commands, or pyximport. Here we are getting a backstage look at what’s going on, which can be helpful if issues arise.

As mentioned, there are two stages in the Cython compilation pipeline: generating C (or C++) code from Cython source, and compiling the C (or C++) code into an extension module.

The first step is easy—we use the cython command:

$ cython fib.pyx

If there are no compilation errors, then cython will print nothing, and we will see a fib.c file that cython has generated. There are several flags that the cython compiler accepts. To see them and a brief description of what they do, call cython with no arguments:

$ cython

Cython (http://cython.org) is a compiler for code written in the

Cython language. Cython is based on Pyrex by Greg Ewing.

Usage: cython [options] sourcefile.{pyx,py} ...

Options:

-V, --version Display version number of cython

compiler

-I, --include-dir <directory> Search for include files in

named directory (multiple

include directories are

allowed).

-o, --output-file <filename> Specify name of generated C file

-f, --force Compile all source files

(overrides implied -t)

-v, --verbose Be verbose, print file names on

multiple compilation

-w, --working <directory> Sets the working directory for

Cython (the directory modules

are searched from)

-D, --no-docstrings Strip docstrings from the

compiled module.

-a, --annotate Produce a colorized HTML version

of the source.

--line-directives Produce #line directives

pointing to the .pyx source

--cplus Output a C++ rather than C file.

--embed[=<method_name>] Generate a main() function that

embeds the Python interpreter.

-2 Compile based on Python-2 syntax

and code semantics.

-3 Compile based on Python-3 syntax

and code semantics.

--lenient Change some compile time errors

to runtime errors to improve

Python compatibility

--warning-errors, -Werror Make all warnings into errors

--warning-extra, -Wextra Enable extra warnings

-X, --directive <name>=<value>[,<name=value,...]

Overrides a compiler directive

The preceding example includes only the more common options, most of which we will cover in this and future chapters. The arguments most commonly used are --cplus to generate a C++ source file rather than C; -a to generate an annotated HTML version of the source, useful for performance analysis and covered in depth in Chapter 9; and the -2 or -3 arguments to control which major version of the Python language to use and enforce.

To compile our fib.c into a Python extension module, we need to first compile fib.c into an object file with the proper includes and compilation flags, and then compile fib.o into a dynamic library with the right linking flags. Fortunately, Python provides the python-config command-lineutility to help with this process. We can use python-config --cflags to obtain the right compilation flags, and python-config --ldflags gives us the right linking flags:

$ CFLAGS=$(python-config --cflags)

$ LDFLAGS=$(python-config --ldflags)

$ cython fib.pyx # --> outputs fib.c

$ gcc -c fib.c ${CFLAGS} # outputs fib.o

$ gcc fib.o -o fib.so -shared ${LDFLAGS} # --> outputs fib.so

In the last line, the -shared flag instructs gcc to create a shared library. This is necessary on Mac OS X; different platforms and compilers may require a different argument or arguments. It is strongly recommended to use the same compiler that was used to compile the Python interpreter. The python-config command gives back configuration flags that are tailored to this compiler/Python version combination.

This is fine for a simple project with just one extension module, but what about larger projects that have their own build system? The Cython compilation pipeline can work with these as well.

Using Cython with Other Build Systems

Many build tools know how to take a C or C++ source file and compile it into a Python extension module. These tools often provide simple commands that handle the details for us, much like Python’s own distutils package does. The benefit of these build tools is that they have improved dependency management and other advanced features that distutils lacks, which can be a tremendous productivity enhancement for large projects. Cython can be integrated into these build tools if it is not already, and we will cover a few of them here.

CMake and Cython

CMake is a powerful open source build system created by Kitware, Inc. There are third-party build commands that can properly detect the cython compiler and fold Cython code into a standard CMake-compiled project. One version of these commands makes it possible to use the following interface:

# Detects and activates Cython

include(UseCython)

# Specifies that Cython source files should generate C++

set_source_files_properties(

${CYTHON_CMAKE_EXAMPLE_SOURCE_DIR}/src/file.pyx

PROPERTIES CYTHON_IS_CXX TRUE )

# Adds and compiles Cython source into an extension module

cython_add_module( modname file.pyx cpp_source.cxx)

SCons and Cython

SCons is a full build system written in Python. Cython comes with basic SCons support in the Tools directory. There we can find cython.py and pyext.py files to extend SCons with Cython support that can be incorporated into our own SCons-based build system.

Make and Cython

Cython can be incorporated into a make-based build system. To help with portability, it is recommended to query the Python interpreter itself to determine the right compilation and linking flags to use. The python-config utility that comes with CPython can alternatively be used when available. The distutils.sysconfig module can be used to get configuration parameters for these flags. For instance, to access the include directory for the Python header file Python.h where the Python/C API is declared, we can use the following make command:

INCDIR := $(shell python -c \

"from distutils import sysconfig; print(sysconfig.get_python_inc())")

To acquire the Python dynamic libraries to link against, we can use:

LIBS := $(shell python -c \

"from distutils import sysconfig; \

print(sysconfig.get_config_var('LIBS'))")

Other configuration settings are available via the get_config_var function in the distutils.sysconfig module.

NOTE

While these build systems do have dependency-tracking features, be aware that they may not recognize all Cython import and include dependencies (Chapter 6), which can result in a dependent module not being compiled when an imported or included dependency changes. It may be necessary to force recompilation in some instances.

CYTHON STANDALONE EXECUTABLES

Because Cython works closely with the Python/C API and runtime environment, Cython source code is nearly always compiled into a dynamic extension module and imported by Python code. But the cython compiler does have an option to embed the Python interpreter inside a main function. This makes it possible to use Cython to create a standalone executable that can be run directly from the command line.

Consider a simple Python—or Cython—script named irrationals.py:

from math import pi, e

print "e**pi == {:.2f}".format(e**pi)

print "pi**e == {:.2f}".format(pi**e)

Here is its output when run:

$ python irrationals.py

e**pi == 23.14

pi**e == 22.46

To compile this into an executable binary with Cython, we first call cython with the --embed flag:

$ cython --embed irrationals.py

This generates irrationals.c with a main entry point that embeds a Python interpreter. We can compile irrationals.c on Mac OS X or Linux using python-config:

$ gcc $(python-config --cflags) $(python-config --ldflags) ./irrationals.c

This produces an executable a.out that we can run directly:

$ ./a.out

e**pi == 23.14

pi**e == 22.46

This simple example provides a recipe for embedding the Python interpreter in a Cython-generated source file, which may be useful in certain contexts. Remember that the binary still has a runtime dependency on the Python dynamic library.

Compiler Directives

Cython provides compiler directives to control how it compiles Cython source code. Directives can be specified in four separate scopes and can be easily turned on or off for testing and debugging. Not all directives can be set at every scope.

All directives can be set globally for an extension module inside a directive comment. These comments must appear at the top of an extension module, and must come before the first line of source code. A directive comment can come after other comments. All directive comments must start with the comment character followed by cython:, the directive name, and its value.

For instance, to globally set the nonecheck directive (covered in detail in Chapter 5) to True for an extension module source.pyx, we can say:

# cython: nonecheck=True

We can have more than one directive specified on one line. To turn off bounds checking for indexing globally (covered in Chapter 10), we can add a boundscheck=False directive:

# cython: nonecheck=True, boundscheck=False

or we can specify them on separate lines:

# cython: nonecheck=True

# cython: boundscheck=False

Alternatively, we can set directives from the command line using the -X or -—directive option. Doing so overrides the value for the directive set in a directive comment.

For example, to globally set (and overrride) the nonecheck directive in source.pyx to False, we can use:

$ cython --directive nonecheck=False source.pyx

Some directives support function- and context-level scope control, via decorators and context managers, respectively.

For instance, to turn off bounds checking and wraparound checking for an entire function, we can use the decorator forms of the boundscheck and wraparound directives, both described in Chapter 10:

cimport cython

@cython.boundscheck(False)

@cython.wraparound(False)

def fast_indexing():

# ...

If we desire even more local control over these directives, we can use the context-manager form:

cimport cython

def fast_indexing(a):

with cython.boundscheck(False), cython.wraparound(False):

for i inrange(len(a)):

sum += a[i]

These directives are set to False only for the body of the context manager, and revert to their default True value outside.

Neither the decorator form nor the context-manager form of a directive is affected by directive comments or command-line directives.

In the following chapters we will point out what directives are available and what they do. A comprehensive list of directives is also found in the online Cython documentation.

Summary

Now that we have covered the Cython compiler pipeline and various ways to compile Cython source into an importable Python extension module, we have the necessary knowledge to work with the examples throughout the rest of this book.