Learning Python (2013)

---

^[48^] As we saw briefly in “Other Ways to Access Globals” in Chapter 17, because a function can access its enclosing module by going through the sys.modules table like this, it can also be used to emulate the effect of the global statement. For instance, the effect of global X; X=0 can be simulated (albeit with much more typing!) by saying this inside a function: import sys; glob=sys.modules[__name__]; glob.X=0. Remember, each module gets a __name__ attribute for free; it’s visible as a global name inside the functions within the module. This trick provides another way to change both local and global variables of the same name inside a function.

^[49^] You can preload tools such as mydir.listing and the reloader we’ll meet in a moment into the interactive namespace by importing them in the file referenced by the PYTHONSTARTUP environment variable. Because code in the startup file runs in the interactive namespace (module __main__), importing common tools in the startup file can save you some typing. See Appendix A for more details.

Importing Modules by Name String

The module name in an import or from statement is a hardcoded variable name. Sometimes, though, your program will get the name of a module to be imported as a string at runtime—from a user selection in a GUI, or a parse of an XML document, for instance. Unfortunately, you can’t use import statements directly to load a module given its name as a string—Python expects a variable name that’s taken literally and not evaluated, not a string or expression. For instance:

>>> import 'string'

File "<stdin>", line 1

import "string"

^

SyntaxError: invalid syntax

It also won’t work to simply assign the string to a variable name:

x = 'string'

import x

Here, Python will try to import a file x.py, not the string module—the name in an import statement both becomes a variable assigned to the loaded module and identifies the external file literally.

Running Code Strings

To get around this, you need to use special tools to load a module dynamically from a string that is generated at runtime. The most general approach is to construct an import statement as a string of Python code and pass it to the exec built-in function to run (exec is a statement in Python 2.X, but it can be used exactly as shown here—the parentheses are simply ignored):

>>> modname = 'string'

>>> exec('import ' + modname) # Run a string of code

>>> string # Imported in this namespace

<module 'string' from 'C:\\Python33\\lib\\string.py'>

We met the exec function (and its cousin for expressions, eval) earlier, in Chapter 3 and Chapter 10. It compiles a string of code and passes it to the Python interpreter to be executed. In Python, the byte code compiler is available at runtime, so you can write programs that construct and run other programs like this. By default, exec runs the code in the current scope, but you can get more specific by passing in optional namespace dictionaries if needed. It also has security issues noted earlier in the book, which may be minor in a code string you are building yourself.

Direct Calls: Two Options

The only real drawback to exec here is that it must compile the import statement each time it runs, and compiling can be slow. Precompiling to byte code with the compile built-in may help for code strings run many times, but in most cases it’s probably simpler and may run quicker to use the built-in __import__ function to load from a name string instead, as noted in Chapter 22. The effect is similar, but __import__ returns the module object, so assign it to a name here to keep it:

>>> modname = 'string'

>>> string = __import__(modname)

>>> string

<module 'string' from 'C:\\Python33\\lib\\string.py'>

As also noted in Chapter 22, the newer call importlib.import_module does the same work, and is generally preferred in more recent Pythons for direct calls to import by name string—at least per the current “official” policy stated in Python’s manuals:

>>> import importlib

>>> modname = 'string'

>>> string = importlib.import_module(modname)

>>> string

<module 'string' from 'C:\\Python33\\lib\\string.py'>

The import_module call takes a module name string, and an optional second argument that gives the package used as the anchor point for resolving relative imports, which defaults to None. This call works the same as __import__ in its basic roles, but see Python’s manuals for more details.

Though both calls still work, in Pythons where both are available, the original __import__ is generally intended for customizing import operations by reassignment in the built-in scope (and any future changes in “official” policy are beyond the scope of this book!).

Example: Transitive Module Reloads

This section develops a module tool that ties together and applies some earlier topics, and serves as a larger case study to close out this chapter and part. We studied module reloads in Chapter 23, as a way to pick up changes in code without stopping and restarting a program. When you reload a module, though, Python reloads only that particular module’s file; it doesn’t automatically reload modules that the file being reloaded happens to import.

For example, if you reload some module A, and A imports modules B and C, the reload applies only to A, not to B and C. The statements inside A that import B and C are rerun during the reload, but they just fetch the already loaded B and C module objects (assuming they’ve been imported before). In actual yet abstract code, here’s the file A.py:

# A.py

import B # Not reloaded when A is!

import C # Just an import of an already loaded module: no-ops

% python

>>> . . .

>>> from imp import reload

>>> reload(A)

By default, this means that you cannot depend on reloads to pick up changes in all the modules in your program transitively—instead, you must use multiple reload calls to update the subcomponents independently. This can require substantial work for large systems you’re testing interactively. You can design your systems to reload their subcomponents automatically by adding reload calls in parent modules like A, but this complicates the modules’ code.

A Recursive Reloader

A better approach is to write a general tool to do transitive reloads automatically by scanning modules’ __dict__ namespace attributes and checking each item’s type to find nested modules to reload. Such a utility function could call itself recursively to navigate arbitrarily shaped and deep import dependency chains. Module __dict__ attributes were introduced in Chapter 23 and employed earlier in this chapter, and the type call was presented in Chapter 9; we just need to combine the two tools.

The module reloadall.py listed next defines a reload_all function that automatically reloads a module, every module that the module imports, and so on, all the way to the bottom of each import chain. It uses a dictionary to keep track of already reloaded modules, recursion to walk the import chains, and the standard library’s types module, which simply predefines type results for built-in types. The visited dictionary technique works to avoid cycles here when imports are recursive or redundant, because module objects are immutable and so can be dictionary keys; as we learned in Chapter 5 and Chapter 8, a set would offer similar functionality if we use visited.add(module) to insert:

#!python

"""

reloadall.py: transitively reload nested modules (2.X + 3.X).

Call reload_all with one or more imported module module objects.

"""

import types

from imp import reload # from required in 3.X

def status(module):

print('reloading ' + module.__name__)

def tryreload(module):

try:

reload(module) # 3.3 (only?) fails on some

except:

print('FAILED: %s' % module)

def transitive_reload(module, visited):

if not module in visited: # Trap cycles, duplicates

status(module) # Reload this module

tryreload(module) # And visit children

visited[module] = True

for attrobj in module.__dict__.values(): # For all attrs

if type(attrobj) == types.ModuleType: # Recur if module

transitive_reload(attrobj, visited)

def reload_all(*args):

visited = {} # Main entry point

for arg in args: # For all passed in

if type(arg) == types.ModuleType:

transitive_reload(arg, visited)

def tester(reloader, modname): # Self-test code

import importlib, sys # Import on tests only

if len(sys.argv) > 1: modname = sys.argv[1] # command line (or passed)

module = importlib.import_module(modname) # Import by name string

reloader(module) # Test passed-in reloader

if __name__ == '__main__':

tester(reload_all, 'reloadall') # Test: reload myself?

Besides namespace dictionaries, this script makes use of other tools we’ve studied here: it includes a __name__ test to launch self-test code when run as a top-level script only, and its tester function uses sys.argv to inspect command-line arguments and importlib to import a module by name string passed in as a function or command-line argument. One curious bit: notice how this code must wrap the basic reload call in a try statement to catch exceptions—in Python 3.3, reloads sometimes fail due to a rewrite of the import machinery. The try was previewed inChapter 10, and is covered in full in Part VII.

Testing recursive reloads

Now, to leverage this utility for normal use, import its reload_all function and pass it an already loaded module object—just as you would for the built-in reload function. When the file runs standalone, its self-test code calls reload_all automatically, reloading its own module by default if no command-line arguments are used. In this mode, the module must import itself because its own name is not defined in the file without an import. This code works in both 3.X and 2.X because we’ve used + and % instead of a comma in the prints, though the set of modules used and thus reloaded may vary across lines:

C:\code> c:\Python33\python reloadall.py

reloading reloadall

reloading types

c:\code> C:\Python27\python reloadall.py

reloading reloadall

reloading types

With a command-line argument, the tester instead reloads the given module by its name string—here, the benchmark module we coded in Chapter 21. Note that we give a module name in this mode, not a filename (as for import statements, don’t include the .py extension); the script ultimately imports the module using the module search path as usual:

c:\code> reloadall.py pybench

reloading pybench

reloading timeit

reloading itertools

reloading sys

reloading time

reloading gc

reloading os

reloading errno

reloading ntpath

reloading stat

reloading genericpath

reloading copyreg

Perhaps most commonly, we can also deploy this module at the interactive prompt—here, in 3.3 for some standard library modules. Notice how os is imported by tkinter, but tkinter reaches sys before os can (if you want to test this on Python 2.X, substitute Tkinter for tkinter):

>>> from reloadall import reload_all

>>> import os, tkinter

>>> reload_all(os) # Normal usage mode

reloading os

reloading ntpath

reloading stat

reloading sys

reloading genericpath

reloading errno

reloading copyreg

>>> reload_all(tkinter)

reloading tkinter

reloading _tkinter

reloading warnings

reloading sys

reloading linecache

reloading tokenize

reloading builtins

FAILED: <module 'builtins'>

reloading re

...etc...

reloading os

reloading ntpath

reloading stat

reloading genericpath

reloading errno

...etc...

And finally here is a session that shows the effect of normal versus transitive reloads—changes made to the two nested files are not picked up by reloads, unless the transitive utility is used:

import b # File a.py

X = 1

import c # File b.py

Y = 2

Z = 3 # File c.py

C:\code> py −3

>>> import a

>>> a.X, a.b.Y, a.b.c.Z

(1, 2, 3)

# Without stopping Python, change all three files' assignment values and save

>>> from imp import reload

>>> reload(a) # Built-in reload is top level only

<module 'a' from '.\\a.py'>

>>> a.X, a.b.Y, a.b.c.Z

(111, 2, 3)

>>> from reloadall import reload_all

>>> reload_all(a) # Normal usage mode

reloading a

reloading b

reloading c

>>> a.X, a.b.Y, a.b.c.Z # Reloads all nested modules too

(111, 222, 333)

Study the reloader’s code and results for more on its operation. The next section exercises its tools further.

Alternative Codings

For all the recursion fans in the audience, the following lists an alternative recursive coding for the function in the prior section—it uses a set instead of a dictionary to detect cycles, is marginally more direct because it eliminates a top-level loop, and serves to illustrate recursive function techniques in general (compare with the original to see how this differs). This version also gets some of its work for free from the original, though the order in which it reloads modules might vary if namespace dictionary order does too:

"""

reloadall2.py: transitively reload nested modules (alternative coding)

"""

import types

from imp import reload # from required in 3.X

from reloadall import status, tryreload, tester

def transitive_reload(objects, visited):

for obj in objects:

if type(obj) == types.ModuleType and obj not in visited:

status(obj)

tryreload(obj) # Reload this, recur to attrs

visited.add(obj)

transitive_reload(obj.__dict__.values(), visited)

def reload_all(*args):

transitive_reload(args, set())

if __name__ == '__main__':

tester(reload_all, 'reloadall2') # Test code: reload myself?

As we saw in Chapter 19, there is usually an explicit stack or queue equivalent to most recursive functions, which may be preferable in some contexts. The following is one such transitive reloader; it uses a generator expression to filter out nonmodules and modules already visited in the current module’s namespace. Because it both pops and adds items at the end of its list, it is stack based, though the order of both pushes and dictionary values influences the order in which it reaches and reloads modules—it visits submodules in namespace dictionaries from right to left, unlike the left-to-right order of the recursive versions (trace through the code to see how). We could change this, but dictionary order is arbitrary anyhow.

"""

reloadall3.py: transitively reload nested modules (explicit stack)

"""

import types

from imp import reload # from required in 3.X

from reloadall import status, tryreload, tester

def transitive_reload(modules, visited):

while modules:

next = modules.pop() # Delete next item at end

status(next) # Reload this, push attrs

tryreload(next)

visited.add(next)

modules.extend(x for x in next.__dict__.values()

if type(x) == types.ModuleType and x not in visited)

def reload_all(*modules):

transitive_reload(list(modules), set())

if __name__ == '__main__':

tester(reload_all, 'reloadall3') # Test code: reload myself?

If the recursion and nonrecursion used in this example is confusing, see the discussion of recursive functions in Chapter 19 for background on the subject.

Testing reload variants

To prove that these work the same, let’s test all three of our reloader variants. Thanks to their common testing function, we can run all three from a command line both with no arguments to test the module reloading itself, and with the name of a module to be reloaded listed on the command line (in sys.argv):

c:\code> reloadall.py

reloading reloadall

reloading types

c:\code> reloadall2.py

reloading reloadall2

reloading types

c:\code> reloadall3.py

reloading reloadall3

reloading types

Though it’s hard to see here, we really are testing the individual reloader alternatives—each of these tests shares a common tester function, but passes it the reload_all from its own file. Here are the variants reloading the 3.X tkinter GUI module and all the modules its imports reach:

c:\code> reloadall.py tkinter

reloading tkinter

reloading _tkinter

reloading tkinter._fix

...etc...

c:\code> reloadall2.py tkinter

reloading tkinter

reloading tkinter.constants

reloading tkinter._fix

...etc...

c:\code> reloadall3.py tkinter

reloading tkinter

reloading sys

reloading tkinter.constants

...etc...

All three work on both Python 3.X and 2.X too—they’re careful to unify prints with formatting, and avoid using version-specific tools (though you must use 2.X module names like Tkinter, and I’m using the 3.3 Windows launcher here to run per Appendix B):

c:\code> py −2 reloadall.py

reloading reloadall

reloading types

c:\code> py −2 reloadall2.py Tkinter

reloading Tkinter

reloading _tkinter

reloading FixTk

...etc...

As usual we can test interactively, too, by importing and calling either a module’s main reload entry point with a module object, or the testing function with a reloader function and module name string:

C:\code> py −3

>>> import reloadall, reloadall2, reloadall3

>>> import tkinter

>>> reloadall.reload_all(tkinter) # Normal use case

reloading tkinter

reloading tkinter._fix

reloading os

...etc...

>>> reloadall.tester(reloadall2.reload_all, 'tkinter') # Testing utility

reloading tkinter

reloading tkinter._fix

reloading os

...etc...

>>> reloadall.tester(reloadall3.reload_all, 'reloadall3') # Mimic self-test code

reloading reloadall3

reloading types

Finally, if you look at the output of tkinter reloads earlier, you may notice that each of the three variants may produce results in a different order; they all depend on namespace dictionary ordering, and the last also relies on the order in which items are added to its stack. In fact, under Python 3.3, the reload order for a given reloader can vary from run to run. To ensure that all three are reloading the same modules irrespective of the order in which they do so, we can use sets (or sorts) to test for order-neutral equality of their printed messages—obtained here by running shell commands with the os.popen utility we met in Chapter 13 and used in Chapter 21:

>>> import os

>>> res1 = os.popen('reloadall.py tkinter').read()

>>> res2 = os.popen('reloadall2.py tkinter').read()

>>> res3 = os.popen('reloadall3.py tkinter').read()

>>> res1[:75]

'reloading tkinter\nreloading tkinter.constants\nreloading tkinter._fix\nreload'

>>> res1 == res2, res2 == res3

(False, False)

>>> set(res1) == set(res2), set(res2) == set(res3)

(True, True)

Run these scripts, study their code, and experiment on your own for more insight; these are the sort of importable tools you might want to add to your own source code library. Watch for a similar testing technique in the coverage of class tree listers in Chapter 31, where we’ll apply it to passed class objects and extend it further.

Also keep in mind that all three variants reload only modules that were loaded with import statements—since names copied with from statements do not cause a module to be nested and referenced in the importer’s namespace, their containing module is not reloaded. More fundamentally, the transitive reloaders rely on the fact that module reloads update module objects in place, such that all references to those modules in any scope will see the updated version automatically. Because they copy names out, from importers are not updated by reloads—transitive or not—and supporting this may require either source code analysis, or customization of the import operation (see Chapter 22 for pointers).

Tool impacts like this are perhaps another reason to prefer import to from—which brings us to the end of this chapter and part, and the standard set of warnings for this part’s topic.

Module Gotchas

In this section, we’ll take a look at the usual collection of boundary cases that can make life interesting for Python beginners. Some are review here, and a few are so obscure that coming up with representative examples can be a challenge, but most illustrate something important about the language.

Module Name Clashes: Package and Package-Relative Imports

If you have two modules of the same name, you may only be able to import one of them—by default, the one whose directory is leftmost in the sys.path module search path will always be chosen. This isn’t an issue if the module you prefer is in your top-level script’s directory; since that is always first in the module path, its contents will be located first automatically. For cross-directory imports, however, the linear nature of the module search path means that same-named files can clash.

To fix, either avoid same-named files or use the package imports feature of Chapter 24. If you need to get to both same-named files, structure your source files in subdirectories, such that package import directory names make the module references unique. As long as the enclosing package directory names are unique, you’ll be able to access either or both of the same-named modules.

Note that this issue can also crop up if you accidentally use a name for a module of your own that happens to be the same as a standard library module you need—your local module in the program’s home directory (or another directory early in the module path) can hide and replace the library module.

To fix, either avoid using the same name as another module you need or store your modules in a package directory and use Python 3.X’s package-relative import model, available in 2.X as an option. In this model, normal imports skip the package directory (so you’ll get the library’s version), but special dotted import statements can still select the local version of the module if needed.

Statement Order Matters in Top-Level Code

As we’ve seen, when a module is first imported (or reloaded), Python executes its statements one by one, from the top of the file to the bottom. This has a few subtle implications regarding forward references that are worth underscoring here:

§ Code at the top level of a module file (not nested in a function) runs as soon as Python reaches it during an import; because of that, it cannot reference names assigned lower in the file.

§ Code inside a function body doesn’t run until the function is called; because names in a function aren’t resolved until the function actually runs, they can usually reference names anywhere in the file.

Generally, forward references are only a concern in top-level module code that executes immediately; functions can reference names arbitrarily. Here’s a file that illustrates forward reference dos and don’ts:

func1() # Error: "func1" not yet assigned

def func1():

print(func2()) # OK: "func2" looked up later

func1() # Error: "func2" not yet assigned

def func2():

return "Hello"

func1() # OK: "func1" and "func2" assigned

When this file is imported (or run as a standalone program), Python executes its statements from top to bottom. The first call to func1 fails because the func1 def hasn’t run yet. The call to func2 inside func1 works as long as func2’s def has been reached by the time func1 is called—and it hasn’t when the second top-level func1 call is run. The last call to func1 at the bottom of the file works because func1 and func2 have both been assigned.

Mixing defs with top-level code is not only difficult to read, it’s also dependent on statement ordering. As a rule of thumb, if you need to mix immediate code with defs, put your defs at the top of the file and your top-level code at the bottom. That way, your functions are guaranteed to be defined and assigned by the time Python runs the code that uses them.

from Copies Names but Doesn’t Link

Although it’s commonly used, the from statement is the source of a variety of potential gotchas in Python. As we’ve learned, the from statement is really an assignment to names in the importer’s scope—a name-copy operation, not a name aliasing. The implications of this are the same as for all assignments in Python, but they’re subtle, especially given that the code that shares the objects lives in different files. For instance, suppose we define the following module, nested1.py:

# nested1.py

X = 99

def printer(): print(X)

If we import its two names using from in another module, nested2.py, we get copies of those names, not links to them. Changing a name in the importer resets only the binding of the local version of that name, not the name in nested1.py:

# nested2.py

from nested1 import X, printer # Copy names out

X = 88 # Changes my "X" only!

printer() # nested1's X is still 99

% python nested2.py

99

If we use import to get the whole module and then assign to a qualified name, however, we change the name in nested1.py. Attribute qualification directs Python to a name in the module object, rather than a name in the importer, nested3.py:

# nested3.py

import nested1 # Get module as a whole

nested1.X = 88 # OK: change nested1's X

nested1.printer()

% python nested3.py

88

from * Can Obscure the Meaning of Variables

I mentioned this earlier but saved the details for here. Because you don’t list the variables you want when using the from module import * statement form, it can accidentally overwrite names you’re already using in your scope. Worse, it can make it difficult to determine where a variable comes from. This is especially true if the from * form is used on more than one imported file.

For example, if you use from * on three modules in the following, you’ll have no way of knowing what a raw function call really means, short of searching all three external module files—all of which may be in other directories:

>>> from module1 import * # Bad: may overwrite my names silently

>>> from module2 import * # Worse: no way to tell what we get!

>>> from module3 import *

>>> . . .

>>> func() # Huh???

The solution again is not to do this: try to explicitly list the attributes you want in your from statements, and restrict the from * form to at most one imported module per file. That way, any undefined names must by deduction be in the module named in the single from *. You can avoid the issue altogether if you always use import instead of from, but that advice is too harsh; like much else in programming, from is a convenient tool if used wisely. Even this example isn’t an absolute evil—it’s OK for a program to use this technique to collect names in a single space for convenience, as long as it’s well known.

reload May Not Impact from Imports

Here’s another from-related gotcha: as discussed previously, because from copies (assigns) names when run, there’s no link back to the modules where the names came from. Names imported with from simply become references to objects, which happen to have been referenced by the same names in the importee when the from ran.

Because of this behavior, reloading the importee has no effect on clients that import its names using from. That is, the client’s names will still reference the original objects fetched with from, even if the names in the original module are later reset:

from module import X # X may not reflect any module reloads!

. . .

from imp import reload

reload(module) # Changes module, but not my names

X # Still references old object

To make reloads more effective, use import and name qualification instead of from. Because qualifications always go back to the module, they will find the new bindings of module names after reloading has updated the module’s content in place:

import module # Get module, not names

. . .

from imp import reload

reload(module) # Changes module in place

module.X # Get current X: reflects module reloads

As a related consequence, our transitive reloader earlier in this chapter doesn’t apply to names fetched with from, only import; again, if you’re going to use reloads, you’re probably better off with import.

reload, from, and Interactive Testing

In fact, the prior gotcha is even more subtle than it appears. Chapter 3 warned that it’s usually better not to launch programs with imports and reloads because of the complexities involved. Things get even worse when from is brought into the mix. Python beginners most often stumble onto its issues in scenarios like this—imagine that after opening a module file in a text edit window, you launch an interactive session to load and test your module with from:

from module import function

function(1, 2, 3)

Finding a bug, you jump back to the edit window, make a change, and try to reload the module this way:

from imp import reload

reload(module)

This doesn’t work, because the from statement assigned only the name function, not module. To refer to the module in a reload, you have to first bind its name with an import statement at least once:

from imp import reload

import module

reload(module)

function(1, 2, 3)

However, this doesn’t quite work either—reload updates the module object in place, but as discussed in the preceding section, names like function that were copied out of the module in the past still refer to the old objects; in this instance, function is still the original version of the function. To really get the new function, you must refer to it as module.function after the reload, or rerun the from:

from imp import reload

import module

reload(module)

from module import function # Or give up and use module.function()

function(1, 2, 3)

Now, the new version of the function will finally run, but it seems an awful lot of work to get there.

As you can see, there are problems inherent in using reload with from: not only do you have to remember to reload after imports, but you also have to remember to rerun your from statements after reloads. This is complex enough to trip up even an expert once in a while. In fact, the situation has gotten even worse in Python 3.X, because you must also remember to import reload itself!

The short story is that you should not expect reload and from to play together nicely. Again, the best policy is not to combine them at all—use reload with import, or launch your programs other ways, as suggested in Chapter 3: using the Run→Run Module menu option in IDLE, file icon clicks, system command lines, or the exec built-in function.

Recursive from Imports May Not Work

I saved the most bizarre (and, thankfully, obscure) gotcha for last. Because imports execute a file’s statements from top to bottom, you need to be careful when using modules that import each other. This is often called recursive imports, but the recursion doesn’t really occur (in fact, circularmay be a better term here)—such imports won’t get stuck in infinite importing loops. Still, because the statements in a module may not all have been run when it imports another module, some of its names may not yet exist.

If you use import to fetch the module as a whole, this probably doesn’t matter; the module’s names won’t be accessed until you later use qualification to fetch their values, and by that time the module is likely complete. But if you use from to fetch specific names, you must bear in mind that you will only have access to names in that module that have already been assigned when a recursive import is kicked off.

For instance, consider the following modules, recur1 and recur2. recur1 assigns a name X, and then imports recur2 before assigning the name Y. At this point, recur2 can fetch recur1 as a whole with an import—it already exists in Python’s internal modules table, which makes it importable, and also prevents the imports from looping. But if recur2 uses from, it will be able to see only the name X; the name Y, which is assigned below the import in recur1, doesn’t yet exist, so you get an error:

# recur1.py

X = 1

import recur2 # Run recur2 now if it doesn't exist

Y = 2

# recur2.py

from recur1 import X # OK: "X" already assigned

from recur1 import Y # Error: "Y" not yet assigned

C:\code> py −3

>>> import recur1

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

File ".\recur1.py", line 2, in <module>

import recur2

File ".\recur2.py", line 2, in <module>

from recur1 import Y

ImportError: cannot import name Y

Python avoids rerunning recur1’s statements when they are imported recursively from recur2 (otherwise the imports would send the script into an infinite loop that might require a Ctrl-C solution or worse), but recur1’s namespace is incomplete when it’s imported by recur2.

The solution? Don’t use from in recursive imports (no, really!). Python won’t get stuck in a cycle if you do, but your programs will once again be dependent on the order of the statements in the modules. In fact, there are two ways out of this gotcha:

§ You can usually eliminate import cycles like this by careful design—maximizing cohesion and minimizing coupling are good first steps.

§ If you can’t break the cycles completely, postpone module name accesses by using import and attribute qualification (instead of from and direct names), or by running your froms either inside functions (instead of at the top level of the module) or near the bottom of your file to defer their execution.

There is additional perspective on this issue in the exercises at the end of this chapter—which we’ve officially reached.

Chapter Summary

This chapter surveyed some more advanced module-related concepts. We studied data hiding techniques, enabling new language features with the __future__ module, the __name__ usage mode variable, transitive reloads, importing by name strings, and more. We also explored and summarized module design issues, wrote some more substantial programs, and looked at common mistakes related to modules to help you avoid them in your code.

The next chapter begins our look at Python’s class—its object-oriented programming tool. Much of what we’ve covered in the last few chapters will apply there, too: classes live in modules and are namespaces as well, but they add an extra component to attribute lookup called inheritance search. As this is the last chapter in this part of the book, however, before we dive into that topic, be sure to work through this part’s set of lab exercises. And before that, here is this chapter’s quiz to review the topics covered here.

Test Your Knowledge: Quiz

1. What is significant about variables at the top level of a module whose names begin with a single underscore?

2. What does it mean when a module’s __name__ variable is the string "__main__"?

3. If the user interactively types the name of a module to test, how can your code import it?

4. How is changing sys.path different from setting PYTHONPATH to modify the module search path?

5. If the module __future__ allows us to import from the future, can we also import from the past?

Test Your Knowledge: Answers

1. Variables at the top level of a module whose names begin with a single underscore are not copied out to the importing scope when the from * statement form is used. They can still be accessed by an import or the normal from statement form, though. The __all__ list is similar, but the logical converse; its contents are the only names that are copied out on a from *.

2. If a module’s __name__ variable is the string "__main__", it means that the file is being executed as a top-level script instead of being imported from another file in the program. That is, the file is being used as a program, not a library. This usage mode variable supports dual-mode code and tests.

3. User input usually comes into a script as a string; to import the referenced module given its string name, you can build and run an import statement with exec, or pass the string name in a call to the __import__ or importlib.import_module.

4. Changing sys.path only affects one running program (process), and is temporary—the change goes away when the program ends. PYTHONPATH settings live in the operating system—they are picked up globally by all your programs on a machine, and changes to these settings endure after programs exit.

5. No, we can’t import from the past in Python. We can install (or stubbornly use) an older version of the language, but the latest Python is generally the best Python (at least within lines—see 2.X longevity!).

Test Your Knowledge: Part V Exercises

See Part V in Appendix D for the solutions.

1. Import basics. Write a program that counts the lines and characters in a file (similar in spirit to part of what wc does on Unix). With your text editor, code a Python module called mymod.py that exports three top-level names:

o A countLines(name) function that reads an input file and counts the number of lines in it (hint: file.readlines does most of the work for you, and len does the rest, though you could count with for and file iterators to support massive files too).

o A countChars(name) function that reads an input file and counts the number of characters in it (hint: file.read returns a single string, which may be used in similar ways).

o A test(name) function that calls both counting functions with a given input filename. Such a filename generally might be passed in, hardcoded, input with the input built-in function, or pulled from a command line via the sys.argv list shown in this chapter’s formats.py andreloadall.py examples; for now, you can assume it’s a passed-in function argument.

All three mymod functions should expect a filename string to be passed in. If you type more than two or three lines per function, you’re working much too hard—use the hints I just gave!

Next, test your module interactively, using import and attribute references to fetch your exports. Does your PYTHONPATH need to include the directory where you created mymod.py? Try running your module on itself: for example, test("mymod.py"). Note that test opens the file twice; if you’re feeling ambitious, you may be able to improve this by passing an open file object into the two count functions (hint: file.seek(0) is a file rewind).

2. from/from *. Test your mymod module from exercise 1 interactively by using from to load the exports directly, first by name, then using the from * variant to fetch everything.

3. __main__. Add a line in your mymod module that calls the test function automatically only when the module is run as a script, not when it is imported. The line you add will probably test the value of __name__ for the string "__main__", as shown in this chapter. Try running your module from the system command line; then, import the module and test its functions interactively. Does it still work in both modes?

4. Nested imports. Write a second module, myclient.py, that imports mymod and tests its functions; then run myclient from the system command line. If myclient uses from to fetch from mymod, will mymod’s functions be accessible from the top level of myclient? What if it imports with import instead? Try coding both variations in myclient and test interactively by importing myclient and inspecting its __dict__ attribute.

5. Package imports. Import your file from a package. Create a subdirectory called mypkg nested in a directory on your module import search path, copy or move the mymod.py module file you created in exercise 1 or 3 into the new directory, and try to import it with a package import of the form import mypkg.mymod and call its functions. Try to fetch your counter functions with a from too.

You’ll need to add an __init__.py file in the directory your module was moved to make this go, but it should work on all major Python platforms (that’s part of the reason Python uses “.” as a path separator). The package directory you create can be simply a subdirectory of the one you’re working in; if it is, it will be found via the home directory component of the search path, and you won’t have to configure your path. Add some code to your __init__.py, and see if it runs on each import.

6. Reloads. Experiment with module reloads: perform the tests in Chapter 23’s changer.py example, changing the called function’s message and/or behavior repeatedly, without stopping the Python interpreter. Depending on your system, you might be able to edit changer in another window, or suspend the Python interpreter and edit in the same window (on Unix, a Ctrl-Z key combination usually suspends the current process, and an fg command later resumes it, though a text edit window probably works just as well).

7. Circular imports. In the section on recursive (a.k.a. circular) import gotchas, importing recur1 raised an error. But if you restart Python and import recur2 interactively, the error doesn’t occur—test this and see for yourself. Why do you think it works to import recur2, but notrecur1? (Hint: Python stores new modules in the built-in sys.modules table—a dictionary—before running their code; later imports fetch the module from this table first, whether the module is “complete” yet or not.) Now, try running recur1 as a top-level script file: python recur1.py. Do you get the same error that occurs when recur1 is imported interactively? Why? (Hint: when modules are run as programs, they aren’t imported, so this case has the same effect as importing recur2 interactively; recur2 is the first module imported.) What happens when you run recur2 as a script? Circular imports are uncommon and rarely this bizarre in practice. On the other hand, if you can understand why they are a potential problem, you know a lot about Python’s import semantics.