Python in Practice: Create Better Programs Using Concurrency, Libraries, and Patterns (2014)

Introduction to Python in Practice

This book is aimed at Python programmers who want to broaden and deepen their Python knowledge so that they can improve the quality, reliability, speed, maintainability, and usability of their Python programs. The book presents numerous practical examples and ideas for improved Python programming.

The book has four key themes: design patterns for coding elegance, improved processing speeds using concurrency and compiled Python (Cython), high-level networking, and graphics.

The book Design Patterns: Elements of Reusable Object-Oriented Software (see the Selected Bibliography for details; 285) was published way back in 1995, yet still exerts a powerful influence over object-oriented programming practices. Python in Practice looks at all of the design patterns in the context of Python, providing Python examples of those that are useful, as well as explaining why some are irrelevant to Python programmers. These patterns are covered in Chapter 1, Chapter 2, and Chapter 3.

Python’s GIL (Global Interpreter Lock) prevents Python code from executing on more than one processor core at a time.^* This has led to the myth that Python can’t do threading or take advantage of multi-core hardware. For CPU-bound processing, concurrency can be done using themultiprocessing module, which is not limited by the GIL and can take full advantage of all the available cores. This can easily achieve the speedups we would expect (i.e., roughly proportional to the number of cores). For I/O-bound processing we can also use themultiprocessing module—or we can use the threading module or the concurrent.futures module. If we use threading for I/O-bound concurrency, the GIL’s overhead is usually dominated by network latency and so may not be an issue in practice.

^*This limitation applies to CPython—the reference implementation that most Python programmers use. Some Python implementations don’t have this constraint, most notably, Jython (Python implemented in Java).

Unfortunately, low- and medium-level approaches to concurrency are very error-prone (in any language). We can avoid such problems by avoiding the use of explicit locks, and by making use of Python’s high-level queue and multiprocessing modules’ queues, or theconcurrent.futures module. We will see how to achieve significant performance improvements using high-level concurrency in Chapter 4.

Sometimes programmers use C, C++, or some other compiled language because of another myth—that Python is slow. While Python is in general slower than compiled languages, on modern hardware Python is often more than fast enough for most applications. And in those cases where Python really isn’t fast enough, we can still enjoy the benefits of programming in Python—and at the same time have our code run faster.

To speed up long-running programs we can use the PyPy Python interpreter (pypy.org). PyPy has a just-in-time compiler that can deliver significant speedups. Another way to increase performance is to use code that runs as fast as compiled C; for CPU-bound processing this can comfortably give us 100 × speedups. The easiest way to achieve C-like speed is to use Python modules that are already written in C under the hood: for example, use the standard library’s array module or the third-party numpy module for incredibly fast and memory-efficient array processing (including multi-dimensional arrays with numpy). Another way is to profile using the standard library’s cProfile module to discover where the bottlenecks are, and then write any speed-critical code in Cython—this essentially provides an enhanced Python syntax that compiles into pure C for maximum runtime speed.

Of course, sometimes the functionality we need is already available in a C or C++ library, or a library in another language that uses the C calling convention. In most such cases there will be a third-party Python module that provides access to the library we require—these can be found on the Python Package Index (PyPI; pypi.python.org). But in the uncommon case that such a module isn’t available, the standard library’s ctypes module can be used to access C library functionality—as can the third-party Cython package. Using preexisting C libraries can significantly reduce development times, as well as usually providing very fast processing. Both ctypes and Cython are covered in Chapter 5.

The Python standard library provides a variety of modules for networking, from the low-level socket module, to the mid-level socketserver module, up to the high-level xmlrpclib module. Although low- and mid-level networking makes sense when porting code from another language, if we are starting out in Python we can often avoid the low-level detail and just focus on what we want our networking applications to do by using high-level modules. In Chapter 6 we will see how to do this using the standard library’s xmlrpclib module and the powerful and easy-to-use third-party RPyC module.

Almost every program must provide some kind of user interface so that the program can determine what work it must do. Python programs can be written to support command-line user interfaces, using the argparse module, and full-terminal user interfaces (e.g., on Unix using the third-party urwid package; excess.org/urwid). There are also a great many web frameworks—from the lightweight bottle (bottlepy.org) to heavyweights like Django (www.djangoproject.com) and Pyramid (www.pylonsproject.org)—all of which can be used to provide applications with a web interface. And, of course, Python can be used to create GUI (graphical user interface) applications.

The death of GUI applications in favor of web applications is often reported—and still hasn’t happened. In fact, people seem to prefer GUI applications to web applications. For example, when smartphones became very popular early in the twenty-first century, users invariably preferred to use a purpose-built “app” rather than a web browser and web page for things they did regularly. There are many ways to do GUI programming with Python using third-party packages. However, in Chapter 7 we will see how to create modern-looking GUI applications using Tkinter, which is supplied as part of Python’s standard library.

Most modern computers—including laptops and even smartphones—come equipped with powerful graphics facilities, often in the form of a separate GPU (Graphics Processing Unit) that’s capable of impressive 2D and 3D graphics. Most GPUs support the OpenGL API, and Python programmers can get access to this API through third-party packages. In Chapter 8, we will see how to make use of OpenGL to do 3D graphics.

The purpose of this book is to illustrate how to write better Python applications that have good performance and maintainable code, and are easy to use. This book assumes prior knowledge of Python programming and is intended to be the kind of book people turn to once they’ve learned Python, whether from Python’s documentation or from other books—such as Programming in Python 3, Second Edition (see the Selected Bibliography for details; 287). The book is designed to provide ideas, inspiration, and practical techniques to help readers take their Python programming to the next level.

All the book’s examples have been tested with Python 3.3 (and where possible Python 3.2 and Python 3.1) on Linux, OS X (in most cases), and Windows (in most cases). The examples are available from the book’s web site, www.qtrac.eu/pipbook.html, and should work with all future Python 3.x versions.