Learning Python (2013)

Part IX. Appendixes

Appendix C. Python Changes and This Book

This appendix briefly summarizes changes made in recent releases of Python organized by the book editions where they first appeared, and gives links to their coverage in this book. It is intended as a reference for both readers of prior editions, as well as developers migrating from prior Python releases.

Here’s how changes in Python relate to this book’s recent editions:

§ This fifth edition of 2013 covers Python 3.3 and 2.7.

§ The fourth edition of 2009 covered Python 2.6 and 3.0 (with some 3.1 features).

§ The third edition of 2007 covered Python 2.5.

§ The first and second editions of 1999 and 2003 covered Pythons 2.0 and 2.2.

§ The predecessor of this book, 1996’s Programming Python, covered Python 1.3.

Hence, to see changes made in just this fifth edition, see the Python 2.7, 3.2, and 3.3 changes listed ahead. For changes incorporated into both the fourth and fifth editions (that is, since the third), also see Python 2.6, 3.0, and 3.1 changes here. Third edition language changes are listed very briefly too, though this seems of only historical value today.

Also note that this appendix focuses on major changes and book impacts, and is not intended as a complete guide to Python’s evolution. For the fuller story on changes applied in each new Python release, consult the “What’s New” documents that are part of its standard documentation set, and available at the Documentation page of python.org. Chapter 15 covers Python documentation and its manuals set.

Major 2.X/3.X Differences

Much of this appendix relates Python changes to book coverage. If you’re instead looking for a quick summary of the most prominent 2.X/3.X distinctions, the following may suffice. Note that this section primarily compares the latest 3.X and 2.X releases—3.3 and 2.7. Many 3.X features are not listed here because they were either also added to 2.6 (e.g., the with statement and class decorators), or back-ported later to 2.7 (e.g., set and dictionary comprehensions), but are not available in earlier 2.X releases. See later sections for more fine-grained information about changes in earlier versions, and see Python’s “What’s New” documents for changes that may appear in future releases.

3.X Differences

The following summarizes tools that differ across Python lines.

§ Unicode string model: In 3.X, normal str strings support all Unicode text including ASCII, and the separate bytes type represents raw 8-bit byte sequences. In 2.X, normal str strings support both 8-bit text including ASCII, and a separate unicode type represents richer Unicode text as an option.

§ File model: In 3.X, files created by open are specialized by content—text files implement Unicode encodings and represent content as str strings, and binary files represent content as bytes strings. In 2.X, files use distinct interfaces—files created by open represent content as strstrings for content that is either 8-bit text or bytes-based data, and codecs.open implements Unicode text encodings.

§ Class model: In 3.X, all classes derive from object automatically and acquire the numerous changes and extensions of new-style classes, including their differing inheritance algorithm, built-ins dispatch, and MRO search order for diamond-pattern trees. In 2.X, normal classes follow theclassic model, and explicit inheritance from object or other built-in types enables the new-style model as an option.

§ Built-in iterables: In 3.X, map, zip, range, filter, and dictionary keys, values, and items are all iterable objects that generate values on request. In 2.X, these calls create physical lists.

§ Printing: 3.X provides a built-in function with keyword arguments for configuration, while 2.X provides a statement with special syntax for configuration.

§ Relative imports: Both 2.X and 3.X support from . relative import statements, but 3.X changes the search rule to skip a package’s own directory for normal imports.

§ True division: Both 2.X and 3.X support the // floor division operator, but the / is true division in 3.X and retains fractional remainders, while / is type-specific in 2.X.

§ Integer types: 3.X has a single integer type that supports extended precision. 2.X has both normal int and extended long, and automatic conversion to long.

§ Comprehension scopes: In 3.X, all comprehension forms—list, set, dictionary, generator—localize variables to the expression. In 2.X, list comprehensions do not.

§ PyDoc: An all-browser pydoc –b interface is supported as of 3.2 and required as of 3.3. In 2.X, the original pydoc –g GUI client interface may be used instead.

§ Byte code storage: As of 3.2, 3.X stores byte code files in a __pycache__ subdirectory of the source directory, with version-identifying names. In 2.X, byte code is stored in the source file directory with generic names.

§ Built-in system exceptions: As of 3.3, 3.X has a reworked exception hierarchy for OS and IO classes that includes additional categories and granularity. In 2.X, exception attributes must sometimes be inspected on system errors.

§ Comparisons and sorts: In 3.X, relative magnitude comparisons of both mixed-types and dictionaries are errors, and sorts do not support mixed types or general comparison functions (use key mappers instead). In 2.X all these forms work.

§ String exceptions and module functions: String-based exceptions are fully removed in 3.X, though they are also gone in 2.X as of 2.6 (use classes instead). string module functions redundant with string object methods are also removed in 3.X.

§ Language removals: Per Table C-2, 3.X removes, renames, or relocates many 2.X language items: reload, apply, `x`, <>, 0177, 999L, dict.has_key, raw_input, xrange, file, reduce, and file.xreadlines.

3.X-Only Extensions

The following summarizes tools available in 3.X only.

§ Extended sequence assignment: 3.X allows a * in sequence assignment targets to collect remaining unmatched iterable items in a list. 2.X can achieve similar effects with slicing.

§ Nonlocal: 3.X provides a nonlocal statement, which allows names in enclosing function scopes to be changed from within nested functions. 2.X can achieve similar effects with function attributes, mutable objects, and class state.

§ Function annotations: 3.X allows function arguments and return types to be annotated with objects that are retained in the function but not otherwise used. 2.X may often achieve similar effects with extra objects or decorator arguments.

§ Keyword-only arguments: 3.X allows specification of function arguments that must be passed as keywords, typically used for extra configuration options. 2.X may often achieve similar effects with argument analysis and dictionary pops.

§ Chained exceptions: 3.X allows exceptions to be chained and thus appear in error messages, with a raise from extension; 3.3 allows a None to cancel the chain.

§ Yield from: As of 3.3, the yield statement may delegate to a nested generator with from. 2.X can often achieve similar results with a for loop in simpler use cases.

§ Namespace packages: As of 3.3, the package model is extended to allow packages that span multiple directories with no initialization file, as a fallback option. 2.X might achieve similar effects with import extensions.

§ Windows launcher: As of 3.3, a launcher is shipped with Python for Windows, though this is also available separately for use on other Pythons, including 2.X.

§ Internals: As of 3.2, threading is implemented with time slices instead of virtual machine instruction counts, and 3.3 stores Unicode text in a variable-length scheme instead of fixed-size bytes. 2.X’s string model minimizes Unicode use in general.

General Remarks: 3.X Changes

Although the Python 3.X line covered in the two most recent editions of this book is largely the same language as its 2.X predecessor, it differs in some crucial ways. As discussed in the preface and summarized in the preceding section, 3.X’s nonoptional Unicode model, mandatory new-style classes, and broader emphasis on generators and other functional tools alone can make it a materially different experience.

On the whole, Python 3.X may be a cleaner language, but it is also in many ways a more sophisticated language, relying upon concepts that are substantially more advanced. In fact, some of its changes seem to assume you must already know Python in order to learn Python. The preface mentioned some of the more prominent circular knowledge dependencies in 3.X that imply forward topic dependencies.

As a random example, the rationale for wrapping dictionary views in a list call in 3.X is incredibly subtle and requires substantial foreknowledge—of views, generators, and the iteration protocol, at the least. Keyword arguments are similarly required in simple tools (e.g., printing, string formatting, dictionary creation, and sorting) that crop up long before a newcomer learns enough about functions to understand them fully. One of this book’s goals is to help bridge this knowledge gap in today’s 2.X/3.X dual-version world.

Changes in Libraries and Tools

There are additional changes in Python 3.X not listed in this appendix, simply because they don’t affect this book. For example, some standard libraries and development tools are outside this book’s core language scope, though some are mentioned along the way (e.g., timeit), and others have always been covered here (e.g., PyDoc).

For completeness, the following sections note 3.X developments in these categories. Some of the changes in these categories are also listed later in this appendix, in conjunction with the book edition and Python version in which they were introduced.

Standard library changes

Formally speaking, the Python standard library is not a part of this book’s core language subject, even though it’s always available with Python, and permeates realistic Python programs. In fact, the libraries were not subject to the temporary 3.X language changes moratorium enacted during 3.2’s development.

Because of this, changes in the standard library have a larger impact on applications-focused books like Programming Python than they do here. Although most standard library functionality is still present, Python 3.X takes further liberties with renaming modules, grouping them into packages, and changing API call patterns.

Some library changes are much broader, though. Python 3.X’s Unicode model, for example, creates widespread differences in 3.X’s standard library—it potentially impacts any program that processes file content, filenames, directory walkers, pipes, descriptor files, sockets, text in GUIs, Internet protocols such as FTP and email, CGI scripts, web content of many kinds, and even some persistence tools such as DBM files, shelves, and pickles.

For a more comprehensive list of changes in 3.X’s standard libraries, see the “What’s New” documents for 3.X releases (especially 3.0) in Python’s standard manual set. Because it uses Python 3.X throughout, the aforementioned Programming Python can also serve as a guide to 3.X library changes.

Tools changes

Though most development tools are the same between 2.X and 3.X (e.g., for debugging, profiling, timing, and testing), a few have undergone changes in 3.X along with the language and library. Among these, the PyDoc module documentation system has moved away from its former GUI client model in 3.2 and earlier, replacing it with an all web browser interface.

Other noteworthy changes in this category: the distutils package, used to distribute and install third-party software, is to be subsumed by a new packaging system in 3.X; the new __pycache__ byte code storage scheme described in this book, though an improvement, potentially impacts many Python tools and programs; and the internal implementation of threading changed as of 3.2 to reduce contention by modifying the global interpreter lock (GIL) to use absolute time slices instead of a virtual machine instruction counter.

Migrating to 3.X

If you are migrating from Python 2.X to Python 3.X, be sure to also see the 2to3 automatic code conversion script that is shipped with Python 3.X. It’s currently available in Python’s Tools\Scripts install folder, or via a web search. This script cannot translate everything, and attempts to translate core language code primarily—3.X standard library APIs may differ further. Still, it does a reasonable job of converting much 2.X code to run under 3.X.

Conversely, the 3to2 back-conversion program, currently available in the third-party domain, can also translate much Python 3.X code to run in 2.X environments. Depending on your goals and constraints, either 2to3 or 3to2 may prove useful if you must maintain code for both Python lines; see the Web for details, and additional tools and techniques.

It’s also possible to write code that runs portably on both 2.X and 3.X using techniques presented in this book—importing 3.X features from __future__, avoiding version-specific tools, and so on. Many of the examples in this book are platform-neutral. For examples, see the benchmarking tools in Chapter 21, the module reloaders and comma formatter in Chapter 25, the class tree listers in Chapter 31, most of the larger decorator examples in Chapter 38 and Chapter 39, the joke script at the end of Chapter 41, and more. As long as you understand 2.X/3.X core language differences, coding around them is often straightforward.

If you’re interested in writing code for both 2.X and 3.X, see also six—a library of cross-version mapping and renaming tools, which currently lives at http://packages.python.org/six. Naturally, this package can’t offset every difference in language semantics and library APIs, and in many cases you must use its library tools instead of straight Python to realize its portability gains. In exchange, though, your programs become much more version-neutral when using this library’s tools.

Fifth Edition Python Changes: 2.7, 3.2, 3.3

The following specific changes were made in the Python 2.X and 3.X lines after the fourth edition was published, and have been incorporated into this edition. Specifically, this section documents Python book-related changes in Pythons 2.7, 3.2, and 3.3.

Changes in Python 2.7

On the technical front, Python 2.7 mostly incorporates as back-ports a handful of 3.X features that were covered in the prior edition of this book, but formerly as 3.X-only features. This new fifth edition presents these as 2.7 tools as well. Among these:

§ Set literals:

§ {1, 4, 2, 3, 4}

§ Set and dictionary comprehensions:

§ {c * 4 for c in 'spam'}, {c: c * 4 for c in 'spam'}

§ Dictionary views, incorporated as optional methods:

§ dict.viewkeys(), dict.viewvalues(), dict.viewitems()

§ Comma separators and field autonumbering in str.format (from 3.1):

§ '{:,.2f} {}'.format(1234567.891, 'spam')

§ Nested with statement context managers (from 3.1):

§ with X() as x, Y() as y: ...

§ Float object repr display improvements (back-ported from 3.1: see ahead)

To see where these topics are covered in the book, look for their entries in the 3.0 changes list of Table C-1, or the Python 3.1 changes section, both ahead. They were already present for 3.X, but have been updated to reflect their availability in 2.7 as well.

On the logistical front, per current plans 2.7 will be the last major 2.X series release, but will have a long maintenance period in which it will continue to be used in production work. After 2.7, new development is to shift to the Python 3.X line.

That said, it’s impossible to foresee how this official posture will stand the test of time, given 2.X’s still very wide user base. See the preface for more on this; the optimized PyPy implementation, for example, is still Python 2.X only. Or, to borrow a Monty Python line, “I’m not dead yet...”—stay tuned for developments on the Python 2.X story.

Changes in Python 3.3

Python 3.3 includes a surprisingly large number of changes for a point release. Some of these are not entirely compatible with code written for prior release in the 3.X line. Among these, the new Windows launcher, installed as a mandatory part of 3.3, has broad potential to break existing 3.X scripts run on Windows.

Here’s a brief rundown of noteworthy 3.3 changes, along with their location in this book where applicable. Python 3.3 comes with:

§ A reduced memory footprint that is more in line with 2.X, thanks mainly to its new variable-length string storage scheme, and also to its attribute name-sharing dictionaries system (see Chapter 37 and Chapter 32)

§ A new namespace package model, where new-style packages may span multiple directories and require no __init__.py file (see Chapter 24)

§ New syntax for delegating to subgenerators: yield from ... (see Chapter 20)

§ New syntax for suppressing exception context: raise ... from None (see Chapter 34)

§ New syntax for accepting 2.X’s Unicode literal form to ease migration: 3.3 now treats 2.X’s Unicode literal u'xxxx' the same as its normal string 'xxxx', similar to the way 2.X treats 3.X’s bytes literal b'xxxx' the same as its normal string 'xxxx' (see Chapter 4, Chapter 7, andChapter 37)

§ Reworked OS and IO exception hierarchies, which provide more inclusive general superclasses, as well as new subclasses for common errors that can obviate the need to inspect exception object attributes (see Chapter 35)

§ An all-web-browser-based interface to PyDoc documentation started via pydoc -b, replacing its former standalone GUI client search interface, which was in the Windows 7 and earlier Start button and invoked by pydoc –g (see Chapter 15)

§ Changes to some longstanding standard library modules, including ftplib, time, and email, and potentially distutils; impacts in this book: time has new portable calls in 3.X (see Chapter 21 and Chapter 39)

§ An implementation of the __import__ function in importlib.__import__, in part to unify and more clearly expose its implementation (see Chapter 22 and Chapter 25)

§ A new capability in the Windows 3.3 installer that extends the system PATH setting to include 3.3’s directory as an install-time option to simplify some command lines (see Appendixes A and B)

§ A new Windows launcher, which attempts to interpret Unix-style #! lines for dispatching Python scripts on Windows, and allows both #! lines and new py command lines to select between Python 2.X and 3.X versions explicitly on both a per-file and per-command basis (see the newAppendix B)

Changes in Python 3.2

Python 3.2 continued the 3.X line’s evolution. It was developed during a moratorium on 3.X core language changes, so its relevant changes were minor. Here’s a quick review of major 3.2 changes, and their location in this fifth edition where relevant:

§ Byte-code files storage model change: __pycache__ (see Chapter 2 and Chapter 22)

§ The struct module’s autoencoding for strings is gone (see Chapter 9 and Chapter 37)

§ 3.X str/bytes split supported better by Python itself (not relevant to this book)

§ The cgi.escape call was to be moved in 3.2+ (not relevant to this book)

§ Threading implementation change: time slices (not relevant to this book)

Fourth Edition Python Changes: 2.6, 3.0, 3.1

The fourth edition was updated to cover Python 3.0 and 2.6, and incorporated a small number of major changes made in 3.1. Its 3.0 and 3.1 changes apply to all future releases in the 3.X line including this fifth edition’s Python 3.3, and its 2.6 changes are also part of this edition’s 2.7. As noted earlier, some of the changes described here as 3.X changes also later found their way into Python 2.7 as back-ports (e.g., set literals, and set and dictionary comprehensions).

Changes in Python 3.1

In addition to the 3.0 and 2.6 changes listed in upcoming sections, shortly before going to press the fourth edition was also augmented with notes about prominent extensions in the then upcoming Python 3.1 release, including:

§ Comma separators and automatic field numbering in string format method calls (Chapter 7)

§ Multiple context manager syntax in with statements (Chapter 34)

§ New methods for number objects (Chapter 5)

§ (Not added until this fifth edition) Floating-point display changes (Chapter 4 and Chapter 5)

This fifth edition covers these topics in the chapters just noted. Because Python 3.1 was targeted primarily at optimization and was released relatively soon after 3.0, the fourth edition also applied directly to 3.1. In fact, because Python 3.1 superseded 3.0 entirely, and because the latest Python is usually the best Python to fetch and use anyhow, whenever that edition used the term “Python 3.0” it generally referred to the language variations introduced by Python 3.0 but that are present in the entire 3.X line, including this edition’s Python 3.3.

One notable exception: the fourth edition did not incorporate 3.1’s new repr display scheme for floating-point numbers. The new display algorithm attempts to display floating-point numbers more intelligently when possible, usually with fewer (but occasionally with more) decimal digits—a change that is reflected in this fifth edition.

Changes in Python 3.0 and 2.6

The fourth edition’s language changes stem from Python 3.0 and 2.6. All of its 2.6 and many of its 3.0 changes are shared by Python 2.7 and 3.3 today. Python 2.7 was extended with some 3.0 features not present in 2.6 (see earlier in this appendix), and Python 3.3 inherits all the features introduced by 3.0.

Because there were so many changes in the initial 3.X release, they are noted only briefly in tables here, with links to more details in this book. Table C-1 provides the first set of 3.X changes, listing the most prominent new language features covered in the fourth edition, along with the primary chapters in the current fifth edition in which they appear.

Table C-1. Extensions in Python 2.6 and 3.0

Specific Language Removals in 3.0

In addition to extensions, a number of 2.X language tools have been removed in 3.X in an effort to clean up its design. Table C-2 summarizes the 3.X removals that impact this book, covered in various chapters of this edition as noted. As also shown in this table, many of the 3.X removals have direct replacements, some of which are also available in 2.6 and 2.7 to support future migration to 3.X.

Table C-2. Removals in Python 3.0 that impact this book

Third Edition Python Changes: 2.3, 2.4, 2.5

The third edition of this book was thoroughly updated to reflect Python 2.5 and all changes to the language made after the publication of the second edition in late 2003. (The second edition was based largely on Python 2.2, with some 2.3 features grafted on at the end of the project.) In addition, brief discussions of anticipated changes in the upcoming Python 3.0 release were incorporated where appropriate. Here are some of the major language topics for which new or expanded coverage was provided (chapter numbers here have been updated to reflect this fifth edition):

§ The new B if A else C conditional expression (Chapter 12, Chapter 19)

§ with/as context managers (Chapter 34)

§ try/except/finally unification (Chapter 34)

§ Relative import syntax (Chapter 24)

§ Generator expressions (Chapter 20)

§ New generator function features (Chapter 20)

§ Function decorators (Chapter 32, Chapter 39)

§ The set object type (Chapter 5)

§ New built-in functions: sorted, sum, any, all, enumerate (Chapter 13 and Chapter 14)

§ The decimal fixed-precision object type (Chapter 5)

§ Files, list comprehensions, and iterators (Chapter 14 and Chapter 20)

§ New development tools: Eclipse, distutils, unittest and doctest, IDLE enhancements, Shed Skin, and so on (Chapter 2 and Chapter 36)

Smaller language changes (for instance, the widespread use of True and False; the new sys.exc_info for fetching exception details; and the demise of string-based exceptions, string methods, and the apply and reduce built-ins) were incorporated throughout the book. The third edition also expanded coverage of some of the features that were new in the second edition, including three-limit slices and the arbitrary arguments call syntax that subsumed apply.

Earlier and Later Python Changes

Each edition before the third also incorporated Python changes too—the first two editions from 1999 and 2003 covered Pythons 2.0 and 2.2, and their 1996 Programming Python 1st Edition predecessor, from which my three later books were all derived, began the process with Python 1.3—but I’ve omitted these here because they are now ancient history (well, in computer field terms, at least).

See the first and second editions for more details, if you can manage to scare one up. While it’s impossible to predict the future, given how much has stood the test of time, it’s likely that the core ideas stressed in this book will likely apply to future Pythons as well.