Learning Python (2013)

Part IX. Appendixes

Appendix B. The Python 3.3 Windows Launcher

This appendix describes the new Windows launcher for Python, installed with Python 3.3 automatically, and available separately on the Web for use with older versions. Though the new launcher comes with some pitfalls, it provides some much-needed coherence for program execution when multiple Pythons coexist on the same computer.

I’ve written this page for programmers using Python on Windows. Though it is platform-specific by nature, it’s targeted at both Python beginners (most of whom get started on this platform), as well as Python developers who write code to work portably between Windows and Unix. As we will see, the new launcher changes the rules on Windows radically enough to impact everyone who uses Python on Windows, or may in the future.

The Unix Legacy

To fully understand the launcher’s protocols, we have to begin with a short history lesson. Unix developers long ago devised a protocol for designating a program to run a script’s code. On Unix systems (including Linux and Mac OS X), the first line in a script’s text file is special if it begins with a two-character sequence: #!, sometimes called a shebang (an arguably silly phrase I promise not to repeat from here on).

Chapter 3 gives a brief overview of this topic, but here’s another look. In Unix scripts, such lines designate a program to run the rest of the script’s contents, by coding it after the #!—using either the directory path to the desired program itself, or an invocation of the env Unix utility that looks up the target per your PATH setting, the customizable system environment variable that lists directories to be searched for executables:

#!/usr/local/bin/python

...script's code # Run under this specific program

#!/usr/bin/env python

...script's code # Run under "python" found on PATH

By making such a script executable (e.g., via chmod +x script.py), you can run it by giving just its filename in a command line; the #! line at the top then directs the Unix shell to a program that will run the rest of the file’s code. Depending on the platform’s install structure, the pythonthat these #! lines name might be a real executable, or a symbolic link to a version-specific executable located elsewhere. These lines might also name a more specific executable explicitly, such as python3. Either way, by changing #! lines, symbolic links, or PATH settings, Unix developers can route a script to the appropriate installed Python.

None of this applies to Windows itself, of course, where #! lines have no inherent meaning. Python itself has historically ignored such lines as comments if present on Windows (“#” starts a comment in the language). Still, the idea of selecting Python executables on a per-file basis is a compelling feature in a world where Python 2.X and 3.X often coexist on the same machine. Given that many programmers coded #! lines for portability to Unix anyhow, the idea seemed ripe for emulating.

The Windows Legacy

The install model has been very different on the other side of the fence. In the past (well, in every Python until 3.3), the Windows installer updated the global Windows registry such that the latest Python version installed on your computer was the version that opened Python files when they were clicked or run by direct filename in command lines.

Some Windows users may know this registry as filename associations, configurable in Control Panel’s Default Programs dialog. You do not need to give files executable privileges for this to work, as you do for Unix scripts. In fact, there’s no such concept on Windows—filename associations and commands suffice to launch files as programs.

Under this install model, if you wished to open a file with a different version than the latest install, you had to run a command line giving the full path to the Python you wanted, or update your filename associations manually to use the desired version. You could also point generic pythoncommand lines to a specific Python by setting or changing your PATH setting, but Python didn’t set this for you, and this wouldn’t apply to scripts launched by icon clicks and other contexts.

This reflects the natural order on Windows (when you click on a .doc file, Windows usually opens it in the latest Word installed), and has been the state of things ever since there was a Python on Windows. It’s less ideal if you have Python scripts that require different versions on the same machine, though—a situation that has become increasingly common, and perhaps even normal in the dual Python 2.X/3.X era. Running multiple Pythons on Windows prior to 3.3 can be tedious for developers, and discouraging for newcomers.

Introducing the New Windows Launcher

The new Windows launcher, shipped and installed automatically with Python 3.3 (and presumably later), and available as a standalone package for use with other versions, addresses these deficits in the former install model by providing two new executables:

§ py.exe for console programs

§ pyw.exe for nonconsole (typically GUI) programs

These two programs are registered to open .py and .pyw files, respectively, via Windows filename associations. Like Python’s original python.exe main program (which they do not deprecate but can largely subsume), these new executables are also registered to open byte code files launched directly. Amongst their weapons, these two new executables:

§ Automatically open Python source and byte-code files launched by icon clicks or filename commands, via Windows associations

§ Are normally installed on your system search path and do not require a directory path or PATH settings when used as command lines

§ Allow Python version numbers to be passed in easily as command-line arguments, when starting both scripts and interactive sessions

§ Attempt to parse Unix-style #! comment lines at the top of scripts to determine which Python version should be used to run a file’s code

The net effect is that under the new launcher, when multiple Pythons are installed on Windows, you are no longer limited to either the latest version installed or explicit/full command lines. Instead, you can now select versions explicitly on both a per-file and per-command basis, and specify versions in either partial or full form in both contexts. Here’s how this works:

1. To select versions per file, use Unix-style top-of-script comments like these:

2. To select versions per command, use command lines of the following forms:

For example, the first of these techniques can serve as a sort of directive to declare which Python version the script depends upon, and will be applied by the launcher whenever the script is run by command line or icon click (these are variants of a file named script.py):

#!python3

...

...a 3.X script # Runs under latest 3.X installed

...

#!python2

...

...a 2.X script # Runs under latest 2.X installed

...

#!python2.6

...

...a 2.6 script # Runs under 2.6 (only)

...

On Windows, command lines are typed in a Command Prompt window, designated by its C:\code> prompt in this appendix. The first of the following is the same as both the second and an icon click, because of filename associations:

C:\code> script.py # Run per file's #! line if present, else per default

C:\code> py script.py # Ditto, but py.exe is run explicitly

Alternatively, the second technique just listed can select versions with argument switches in command lines instead:

C:\code> py −3 script.py # Runs under latest 3.X

C:\code> py −2 script.py # Runs under latest 2.X

C:\code> py −2.6 script.py # Runs under 2.6 (only)

This works both when launching scripts and starting the interactive interpreter (when no script is named):

C:\code> py −3 # Starts latest 3.X, interactive

C:\code> py −2 # Starts latest 2.X, interactive

C:\code> py −3.1 # Starts 3.1 (only), interactive

C:\code> py # Starts default Python (initially 2.X: see ahead)

If there are both #! lines in the file and a version number switch in the command line used to start it, the command line’s version overrides that in the file’s directive:

#! python3.2

...

...a 3.X script

...

C\code> py script.py # Runs under 3.2, per file directive

C\code> py −3.1 script.py # Runs under 3.1, even if 3.2 present

The launcher also applies heuristics to select a specific Python version when it is missing or only partly described. For instance, the latest 2.X is run when only a 2 is specified, and a 2.X is preferred for files that do not name a version in a #! line when launched by icon click or generic command lines (e.g., py m.py, m.py), unless you configure the default to use 3.X instead by setting PY_PYTHON or a configuration file entry (more on this ahead).

Especially in the current dual 2.X/3.X Python world, explicit version selection seems a useful addition for Windows, where many (and probably most) newcomers get their first exposure to the language. Although it is not without potential pitfalls—including failures on unrecognized Unix #!lines and a puzzling 2.X default—it does allow for a more graceful coexistence of 2.X and 3.X files on the same machine, and provides a rational approach to version control in command lines.

For the complete story on the Windows launcher, including more advanced features and use cases I’ll either condense or largely omit here, see Python’s release notes and try a web search to find the PEP (the proposal document). Among other things, the launcher also allows selecting between 32- and 64-bit installs, specifying defaults in configuration files, and defining custom #! command string expansion.

A Windows Launcher Tutorial

Some readers familiar with Unix scripting may find the prior section enough to get started. For others, this section provides additional context in the form of a tutorial, which gives concrete examples of the launcher in action for you to trace through. This section also discloses additional launcher details along the way, though, so even well-seasoned Unix veterans may benefit from a quick scan here before FTPing all their Python scripts to the local Windows box.

To get started, we’ll be using the following simple script, what.py, which can be run under both 2.X and 3.X to echo the version number of the Python that runs its code. It uses sys.version—a string whose first component after splitting on whitespace is Python’s version number:

#!python3

import sys

print(sys.version.split()[0]) # First part of string

If you want to work along, type this script’s code in your favorite text file editor, open a Command Prompt window for typing the command lines we’ll be running, and cd to the directory where you’ve save the script (C:\code is where I’m working, but feel free to save this wherever you wish, and see Chapter 3 for more Windows usage pointers).

This script’s first-line comment serves to designate the required Python version; it must begin with #! per Unix convention, and allows for a space before the python3 or not. On my machine I currently have Pythons 2.7, 3,1, 3.2, and 3.3 all installed; let’s watch which version is invoked as the script’s first line is modified in the following sections, exploring file directives, command lines, and defaults along the way.

Step 1: Using Version Directives in Files

As this script is coded, when run by icon click or command line, the first line directs the registered py.exe launcher to run using the latest 3.X installed:

#! python3

import sys

print(sys.version.split()[0])

C:\code> what.py # Run per file directive

3.3.0

C:\code> py what.py # Ditto: latest 3.X

3.3.0

Again, the space after #! is optional; I added a space to demonstrate the point here. Note that the first what.py command here is equivalent to both an icon click and a full py what.py, because the py.exe program is registered to open .py files automatically in the Windows filename associations registry when the launcher is installed.

Also note that when launcher documentation (including this appendix) talks about the latest version, it means the highest-numbered version. That is, it refers to the latest released, not the latest installed on your computer (e.g., if you install 3.1 after 3.3, #!python3 selects the latter). The launcher cycles through the Pythons on your computer to find the highest-numbered version that matches your specification or defaults; this differs from the former last-installed-wins model.

Now, changing the first line name to python2 triggers the latest (really, highest-numbered) 2.X installed instead. Here’s this change at work; I’ll omit the last two lines of our script from this point on because they won’t be altered:

#! python2

...rest of script unchanged

C:\code> what.py # Run with latest 2.X per #!

2.7.3

And you can request a more specific version if needed—for example, if you don’t want the latest in a Python line:

#! python3.1

...

C:\code> what.py # Run with 3.1 per #!

3.1.4

This is true even if the requested version is not installed—which is treated as an error case by the launcher:

#! python2.6

...

C:\code> what.py

Requested Python version (2.6) is not installed

Unrecognized Unix #! lines are also treated as errors, unless you give a version number as a command-line switch to compensate, as the next section describes in more detail (and as the section on launcher issues will revisit as a pitfall):

#!/bin/python

...

C:\code> what.py

Unable to create process using '/bin/python "C:\code\what.py" '

C:\code> py what.py

Unable to create process using '/bin/python what.py'

C:\code> py −3 what.py

3.3.0

Technically, the launcher recognizes Unix-style #! lines at the top of script files that follow one of the following four patterns:

#!/usr/bin/env python*

#!/usr/bin/python*

#!/usr/local/bin/python*

#!python*

Any #! line that does not take one of these recognized and parseable forms is assumed to be a fully specified command line to start a process to run the file, which is passed to Windows as is, and generates the error message we saw previously if it is not a valid Windows command. (The launcher also supports “customized” command expansions via its configuration files, which are attempted before passing unrecognized commands on to Windows, but we’ll gloss over these here.)

In recognizable #! lines, directory paths are coded per Unix convention, for portability to that platform. The * part at the end of the four preceding recognized patterns denotes an optional Python version number, in one of three forms:

Partial (e.g., python3)

To run the version installed with the highest minor release number among those with the major release number given

Full (e.g., python3.1)

To run that specific version only, optionally suffixed by −32 to prefer a 32-bit version (e.g., python3.1-32)

Omitted (e.g., python)

To run the launcher’s default version, which is 2 unless changed (e.g., by setting the PY_PYTHON environment variable to 3), another pitfall described ahead

Files with no #! line at all behave the same as those that name just a generic python—the aforementioned omitted case—and are influenced by PY_PYTHON default settings. The first case, partials, may also be affected by version-specific environment settings (e.g., set PY_PYTHON3 to 3.1to select 3.1 for python3, and set PY_PYTHON2 to 2.6 to pick 2.6 for python2). We’ll revisit defaults later in this tutorial.

First, though, note that anything after the * part in a #! line’s format is assumed to be command-line arguments to Python itself (i.e., program python.exe), unless you also give arguments in a py command line that are deemed to supersede #! line arguments by the launcher:

#!python3 [any python.exe arguments go here]

...

These include all the Python command-line arguments we met in Appendix A. But this leads us to launcher command lines in general, and will suffice as a natural segue to the next section.

Step 2: Using Command-Line Version Switches

As mentioned, version switches on command lines can be used to select a Python version if one isn’t present in the file. You run a py or pyw command line to pass them a switch this way, instead of relying on filename associations in the registry, and instead of (or in addition to) giving versions in #! lines in files. In the following, we modify our script so that it has no #! directive:

# not a launcher directive

...

C:\code> py −3 what.py # Run per command-line switch

3.3.0

C:\code> py −2 what.py # Ditto: latest 2.X installed

2.7.3

C:\code> py −3.2 what.py # Ditto: 3.2 specifically (and only)

3.2.3

C:\code> py what.py # Run per launcher's default (ahead)

2.7.3

But command-line switches also take precedence over a version designation in a file’s directive:

#! python3.1

...

C:\code> what.py # Run per file directive

3.1.4

C:\code> py what.py # Ditto

3.1.4

C:\code> py −3.2 what.py # Switches override directives

3.2.3

C:\code> py −2 what.py # Ditto

2.7.3

Formally, the launcher accepts the following command-line argument types (which exactly mirror the * part at the end of a file’s #! line described in the prior section):

−2 Launch the latest Python 2.X version

-3 Launch the latest Python 3.X version

-X.Y Launch the specified Python version (X is 2 or 3)

-X.Y−32 Launch the specified 32-bit Python version

And the launcher’s command lines take the following general form:

py [py.exe arg] [python.exe args] script.py [script.py args]

Anything following the launcher’s own argument (if present) is treated as though it were passed to the python.exe program—typically, this includes any arguments for Python itself, followed by the script filename, followed by any arguments meant for the script.

The usual -m mod, -c cmd, and - program specification forms work in a py command line too, as do all the other Python command-line arguments covered in Appendix A. As mentioned earlier, arguments to python.exe can also appear at the end of the #! directive line in a file, if used, though arguments in py command lines override them.

To see how this works, let’s write a new script that extends the prior to display command-line arguments; sys.argv is the script’s own arguments, and I’m using the Python (python.exe) -i switch, which directs it to the interactive prompt (>>>) after a script runs:

# args.py, show my arguments too

import sys

print(sys.version.split()[0])

print(sys.argv)

C:\code> py −3 -i args.py -a 1 -b -c # −3: py, -i: python, rest: script

3.3.0

['args.py', '-a', '1', '-b', '-c']

>>> ^Z

C:\code> py -i args.py -a 1 -b -c # Args to python, script

2.7.3

['args.py', '-a', '1', '-b', '-c']

>>> ^Z

C:\code> py −3 -c print(99) # −3 to py, rest to python: "-c cmd"

99

C:\code> py −2 -c "print 99"

99

Notice how the first two launches run the default Python unless a version is given in the command line, because no #! line appears in the script itself. Somewhat coincidentally, that leads us to the last topic of this tutorial.

Step 3: Using and Changing Defaults

As also mentioned, the launcher defaults to 2.X for a generic python in a #! directive with no specific version number. This is true whether this generic form appears in a full Unix path (e.g., #!/usr/bin/python) or not (#!python). Here’s the latter case in action, coded in our originalwhat.py script:

#!python

... # Same as #!/usr/bin/python

C:\code> what.py # Run per launcher default

2.7.3

The default is also applied when no directive is present at all—perhaps the most common case for code written to be used on Windows primarily or exclusively:

# not a launcher directive

...

C:\code> what.py # Also run per default

2.7.3

C:\code> py what.py # Ditto

2.7.3

But you can set the launcher’s default to 3.X with initialization file or environment variable settings, which will apply to both files run from command lines and by icon clicks via their name’s association with py.exe or pyw.exe in the Windows registry:

# not a launcher directive

...

C:\code> what.py # Run per default

2.7.3

C:\code> set PY_PYTHON=3 # Or via Control Panel/System

C:\code> what.py # Run per changed default

3.3.0

As suggested earlier, for more fine-grained control you can also set version-specific environment variables to direct partial selections to a specific release, instead of falling back on the installed release with the highest minor number:

#!python3

...

C:\code> py what.py # Runs "latest" 3.X

3.3.0

C:\code> set PY_PYTHON3=3.1 # Use PY_PYTHON2 for 2.X

C:\code> py what.py # Override highest-minor choice

3.1.4

The set used in these interactions applies to its Command Prompt window only; making such settings in the Control Panel’s System window will make them apply globally across your machine (see Appendix A for help with these settings). You may or may not want to set defaults this way depending on the majority of the Python code you’ll be running. Many Python 2.X users can probably rely on defaults unchanged, and override them in #! lines or py command lines as needed.

However, the setting used for directive-less files, PY_PYTHON, seems fairly crucial. Most programmers who have used Python on Windows in the past will probably expect 3.X to be the default after installing 3.3, especially given that the launcher is installed by 3.3 in the first place—a seeming paradox, which leads us to the next section.

Pitfalls of the New Windows Launcher

Though the new Windows launcher in 3.3 is a nice addition, like much in 3.X it may have been nicer had it appeared years ago. Unfortunately, it comes with some backward incompatibilities, which may be an inevitable byproduct of today’s multiversion Python world, but which may also break some existing programs. This includes examples in books I’ve written, and probably many others. While porting code to 3.3, I’ve come across three launcher issues worth noting:

§ Unrecognized Unix !# lines now make scripts fail on Windows.

§ The launcher defaults to using 2.X unless told otherwise.

§ The new PATH extension is off by default and seems contradictory.

The rest of this section gives a rundown of each of these three issues in turn. In the following, I use the programs in my book Programming Python, 4th Edition, as an example to illustrate the impacts of launcher incompatibilities, because porting these 3.1/3.2 examples to 3.3 was my first exposure to the new launcher. In my specific case, installing 3.3 broke numerous book examples that worked formerly under 3.2 and 3.1. The causes for these failures outlined here may break your code too.

Pitfall 1: Unrecognized Unix !# Lines Fail

The new Windows launcher recognizes Unix #! lines that begin with #!/usr/bin/env python but not the other common Unix form #!/bin/env python (which is actually mandated on some Unixes). Scripts that use the latter of these, including some of my book examples, worked on Windows in the past because their #! lines coded for Unix compatibility have been ignored as comments by all Windows Pythons to date. These scripts now fail to run in 3.3 because the new launcher doesn’t recognize their directive’s format and posts an error message.

More generally, scripts with any #! Unix line not recognized will now fail to run on Windows. This includes scripts having any first line that begins with a #! that is not followed by one of the four recognized patterns described earlier: /usr/bin/env python*, /usr/bin/python*,/usr/local/bin/python*, or python*. Anything else won’t work, and requires code changes. For instance, a somewhat common #!/bin/python line also causes a script to now fail on Windows, unless a version number is given in command-line switches.

Unix-style #! lines probably aren’t present in Windows-only programs, but can be common in programs meant to be run on Unix too. Treating unrecognized Unix directives as errors on Windows seems a bit extreme, especially given that this is new behavior in 3.3, and will likely be unexpected. Why not just ignore unrecognized #! lines and run the file with the default Python—like every Windows Python to date has? It’s possible that this might be improved in a future 3.X release (there may be some pushback on this), but today you must change any files using a#!/bin/env or other unrecognized pattern, if you want them to run under the launcher installed with Python 3.3 on Windows.

Book examples impact and fix

With respect to the book examples I ported to 3.3, this broke roughly a dozen scripts that started with #!/bin/env python. Regrettably, this includes some of the book’s user-friendly and top-level demo launcher scripts (PyGadgets and PyDemos). To fix, I changed these to use the accepted #!/usr/bin/env python form instead. Altering your Windows file associations to omit the launcher altogether may be another option (e.g., associating .py files with python.exe instead of py.exe), but this negates the launcher’s benefits, and seems a bit much to ask of users, especially newcomers.

One open issue here: strangely, passing any command-line switch to the launcher, even a python.exe argument, seems to negate this effect and fall back on the default Python—m.py and py m.py both issue errors on unrecognized #! lines, but py -i m.py runs such a file with the default Python. This seems a possible launcher bug, but also relies on the default, the subject of the next issue.

Pitfall 2: The Launcher Defaults to 2.X

Oddly, the Windows 3.3 launcher defaults to using an installed Python 2.X when running scripts that don’t select 3.X explicitly. That is, scripts that either have no #! directive or use one that names python generically will be run by a 2.X Python by default when launched by icon clicks, direct filename command lines (m.py), or launcher command lines that give no version switch (py m.py). This is true even if 3.3 is installed after a 2.X on your machine, and has the potential to make many 3.X scripts fail initially.

The implications of this are potentially broad. As one example, clicking the icon of a directive-less 3.X file just after installing 3.3 may now fail, because the associated launcher assumes you mean to use 2.X by default. This probably won’t be a pleasant first encounter for some Python newcomers! This assumes the 3.X file has no #! directive that provides an explicit python3 version number, but most scripts meant to run on Windows won’t have a #! line at all, and many files coded before the launcher came online won’t accommodate its version number expectations. Most 3.X users will be basically compelled to set PY_PYTHON after installing 3.3—hardly a usability win.

Program launches that don’t give an explicit version number might be arguably ambiguous on Unix too, and often rely on symbolic links from python to a specific version (which is most likely 2.X today—a state the new Windows launcher seems to emulate). But as for the prior issue, this probably shouldn’t trigger a new error on Windows in 3.3 for scripts that worked there formerly. Most programmers wouldn’t expect Unix comment lines to matter on Windows, and wouldn’t expect 2.X to be used by default just after installing 3.X.

Book examples impact and fix

In terms of my book examples port, this 2.X default caused multiple 3.X script failures after installing 3.3, for both scripts with no #! line, as well as scripts with a Unix-compatible #!/usr/bin/python line. To fix just the latter, change all scripts in this category to name python3explicitly instead of just python. To fix both the former and the latter in a single step, set the Windows launcher’s default to be 3.X globally with either a py.ini configuration file (see the launcher’s documentation for details) or a PY_PYTHON environment variable setting as shown in the earlier examples (e.g., set PY_PYTHON=3). As mentioned in the prior point, manually changing your file associations is another solution, but none of these options seem simpler than those imposed by prior install schemes.

Pitfall 3: The New PATH Extension Option

Besides installing the new launcher, the Windows Python 3.3 installer can automatically add the directory containing 3.3’s python.exe executable to your system PATH setting. The reasoning behind this is that it might make life easier for some Windows beginners—they can type justpython instead of the full directory path to it. This isn’t a feature of the launcher per se, and shouldn’t cause scripts to fail in general. It had no impact on the book examples. But it seems to clash with the launcher’s operation and goals, and may be best avoided. This is a bit subtle, but I’ll explain why.

As described, the new launcher’s py and pyw executables are by default installed on your system search path, and running them requires neither directory paths nor PATH settings. If you start scripts with py instead of python command lines, the new PATH feature is irrelevant. In fact, pycompletely subsumes python in most contexts. Given that file associations will launch py or pyw instead of python anyhow, you probably should too—using python instead of py may prove redundant and inconsistent, and might even launch a version different than that used in launcher contexts should the two schemes’ settings grow out of sync. In short, adding python to PATH seems contradictory to the new launcher’s worldview, and potentially error-prone.

Also note that updating your PATH assumes you want a python command to run 3.3 normally, and this feature is disabled by default; be sure to select this in the install screen if you want this to work (but not if you don’t!). Due to the second pitfall mentioned earlier, many users may still need to set PY_PYTHON to 3 for programs run by icon clicks that invoke the new launcher, which seems no simpler than setting PATH, a step that the launcher was meant to remove. You may be better served by using just the launcher’s executables, and changing just PY_PYTHON as needed.

Conclusions: A Net Win for Windows

To be fair, some of the prior section’s pitfalls may be an inevitable consequence of trying to simultaneously support a Unix feature on Windows and multiple installed versions. In exchange, it provides a coherent way to manage mixed-version scripts and installations. You’ll probably find the Windows launcher shipped with 3.3 and later to be a major asset once you start using it, and get past any initial incompatibilities you may encounter.

In fact, you may also want to start getting into the habit of coding compatible Unix-style #! lines in your Windows scripts, with explicit version numbers (e.g., #!/usr/bin/python3). Not only does this declare your code’s requirements and arrange for its proper execution on Windows, it will also subvert the launcher’s defaults, and may also make your script usable as a Unix executable in the future.

But you should be aware that the launcher may break some formerly valid scripts having #! lines, may choose a default version that you don’t expect and your scripts can’t use, and may require configuration and code changes on the order of those it was intended to obviate. The new boss is better than the old boss, but seems to have gone to the same school.

For more on Windows usage, see Appendix A for installation and configuration, Chapter 3 for general concepts, and platform-specific documents in Python’s manuals set.