Learning Python (2013)

---

^[37^] In the section Function Gotchas, we’ll also see that there is a similar issue with using mutable objects like lists and dictionaries for default arguments (e.g., def f(a=[]))—because defaults are implemented as single objects attached to functions, mutable defaults retain state from call to call, rather then being initialized anew on each call. Depending on whom you ask, this is either considered a feature that supports another way to implement state retention, or a strange corner of the language; more on this at the end of Chapter 21.

The nonlocal Statement in 3.X

In the prior section we explored the way that nested functions can reference variables in an enclosing function’s scope, even if that function has already returned. It turns out that, in Python 3.X (though not in 2.X), we can also change such enclosing scope variables, as long as we declare them in nonlocal statements. With this statement, nested defs can have both read and write access to names in enclosing functions. This makes nested scope closures more useful, by providing changeable state information.

The nonlocal statement is similar in both form and role to global, covered earlier. Like global, nonlocal declares that a name will be changed in an enclosing scope. Unlike global, though, nonlocal applies to a name in an enclosing function’s scope, not the global module scope outside all defs. Also unlike global, nonlocal names must already exist in the enclosing function’s scope when declared—they can exist only in enclosing functions and cannot be created by a first assignment in a nested def.

In other words, nonlocal both allows assignment to names in enclosing function scopes and limits scope lookups for such names to enclosing defs. The net effect is a more direct and reliable implementation of changeable state information, for contexts that do not desire or need classes with attributes, inheritance, and multiple behaviors.

nonlocal Basics

Python 3.X introduces a new nonlocal statement, which has meaning only inside a function:

def func():

nonlocal name1, name2, ... # OK here

>>> nonlocal X

SyntaxError: nonlocal declaration not allowed at module level

This statement allows a nested function to change one or more names defined in a syntactically enclosing function’s scope. In Python 2.X, when one function def is nested in another, the nested function can reference any of the names defined by assignment in the enclosing def’s scope, but it cannot change them. In 3.X, declaring the enclosing scopes’ names in a nonlocal statement enables nested functions to assign and thus change such names as well.

This provides a way for enclosing functions to provide writeable state information, remembered when the nested function is later called. Allowing the state to change makes it more useful to the nested function (imagine a counter in the enclosing scope, for instance). In 2.X, programmers usually achieve similar goals by using classes or other schemes. Because nested functions have become a more common coding pattern for state retention, though, nonlocal makes it more generally applicable.

Besides allowing names in enclosing defs to be changed, the nonlocal statement also forces the issue for references—much like the global statement, nonlocal causes searches for the names listed in the statement to begin in the enclosing defs’ scopes, not in the local scope of the declaring function. That is, nonlocal also means “skip my local scope entirely.”

In fact, the names listed in a nonlocal must have been previously defined in an enclosing def when the nonlocal is reached, or an error is raised. The net effect is much like global: global means the names reside in the enclosing module, and nonlocal means they reside in an enclosing def. nonlocal is even more strict, though—scope search is restricted to only enclosing defs. That is, nonlocal names can appear only in enclosing defs, not in the module’s global scope or built-in scopes outside the defs.

The addition of nonlocal does not alter name reference scope rules in general; they still work as before, per the “LEGB” rule described earlier. The nonlocal statement mostly serves to allow names in enclosing scopes to be changed rather than just referenced. However, both global andnonlocal statements do tighten up and even restrict the lookup rules somewhat, when coded in a function:

§ global makes scope lookup begin in the enclosing module’s scope and allows names there to be assigned. Scope lookup continues on to the built-in scope if the name does not exist in the module, but assignments to global names always create or change them in the module’s scope.

§ nonlocal restricts scope lookup to just enclosing defs, requires that the names already exist there, and allows them to be assigned. Scope lookup does not continue on to the global or built-in scopes.

In Python 2.X, references to enclosing def scope names are allowed, but not assignment. However, you can still use classes with explicit attributes to achieve the same changeable state information effect as nonlocals (and you may be better off doing so in some contexts); globals and function attributes can sometimes accomplish similar goals as well. More on this in a moment; first, let’s turn to some working code to make this more concrete.

nonlocal in Action

On to some examples, all run in 3.X. References to enclosing def scopes work in 3X as they do in 2.X—in the following, tester builds and returns the function nested, to be called later, and the state reference in nested maps the local scope of tester using the normal scope lookup rules:

C:\code> c:\python33\python

>>> def tester(start):

state = start # Referencing nonlocals works normally

def nested(label):

print(label, state) # Remembers state in enclosing scope

return nested

>>> F = tester(0)

>>> F('spam')

spam 0

>>> F('ham')

ham 0

Changing a name in an enclosing def’s scope is not allowed by default, though; this is the normal case in 2.X as well:

>>> def tester(start):

state = start

def nested(label):

print(label, state)

state += 1 # Cannot change by default (never in 2.X)

return nested

>>> F = tester(0)

>>> F('spam')

UnboundLocalError: local variable 'state' referenced before assignment

Using nonlocal for changes

Now, under 3.X, if we declare state in the tester scope as nonlocal within nested, we get to change it inside the nested function, too. This works even though tester has returned and exited by the time we call the returned nested function through the name F:

>>> def tester(start):

state = start # Each call gets its own state

def nested(label):

nonlocal state # Remembers state in enclosing scope

print(label, state)

state += 1 # Allowed to change it if nonlocal

return nested

>>> F = tester(0)

>>> F('spam') # Increments state on each call

spam 0

>>> F('ham')

ham 1

>>> F('eggs')

eggs 2

As usual with enclosing scope references, we can call the tester factory (closure) function multiple times to get multiple copies of its state in memory. The state object in the enclosing scope is essentially attached to the nested function object returned; each call makes a new, distinctstate object, such that updating one function’s state won’t impact the other. The following continues the prior listing’s interaction:

>>> G = tester(42) # Make a new tester that starts at 42

>>> G('spam')

spam 42

>>> G('eggs') # My state information updated to 43

eggs 43

>>> F('bacon') # But F's is where it left off: at 3

bacon 3 # Each call has different state information

In this sense, Python’s nonlocals are more functional than function locals typical in some other languages: in a closure function, nonlocals are per-call, multiple copy data.

Boundary cases

Though useful, nonlocals come with some subtleties to be aware of. First, unlike the global statement, nonlocal names really must have previously been assigned in an enclosing def’s scope when a nonlocal is evaluated, or else you’ll get an error—you cannot create them dynamically by assigning them anew in the enclosing scope. In fact, they are checked at function definition time before either an enclosing or nested function is called:

>>> def tester(start):

def nested(label):

nonlocal state # Nonlocals must already exist in enclosing def!

state = 0

print(label, state)

return nested

SyntaxError: no binding for nonlocal 'state' found

>>> def tester(start):

def nested(label):

global state # Globals don't have to exist yet when declared

state = 0 # This creates the name in the module now

print(label, state)

return nested

>>> F = tester(0)

>>> F('abc')

abc 0

>>> state

0

Second, nonlocal restricts the scope lookup to just enclosing defs; nonlocals are not looked up in the enclosing module’s global scope or the built-in scope outside all defs, even if they are already there:

>>> spam = 99

>>> def tester():

def nested():

nonlocal spam # Must be in a def, not the module!

print('Current=', spam)

spam += 1

return nested

SyntaxError: no binding for nonlocal 'spam' found

These restrictions make sense once you realize that Python would not otherwise generally know which enclosing scope to create a brand-new name in. In the prior listing, should spam be assigned in tester, or the module outside? Because this is ambiguous, Python must resolve nonlocals at function creation time, not function call time.

Why nonlocal? State Retention Options

Given the extra complexity of nested functions, you might wonder what the fuss is about. Although it’s difficult to see in our small examples, state information becomes crucial in many programs. While functions can return results, their local variables won’t normally retain other values that must live on between calls. Moreover, many applications require such values to differ per context of use.

As mentioned earlier, there are a variety of ways to “remember” information across function and method calls in Python. While there are tradeoffs for all, nonlocal does improve this story for enclosing scope references—the nonlocal statement allows multiple copies of changeable state to be retained in memory. It addresses simple state-retention needs where classes may not be warranted and global variables do not apply, though function attributes can often serve similar roles more portably. Let’s review the options to see how they stack up.

State with nonlocal: 3.X only

As we saw in the prior section, the following code allows state to be retained and modified in an enclosing scope. Each call to tester creates a self-contained package of changeable information, whose names do not clash with any other part of the program:

>>> def tester(start):

state = start # Each call gets its own state

def nested(label):

nonlocal state # Remembers state in enclosing scope

print(label, state)

state += 1 # Allowed to change it if nonlocal

return nested

>>> F = tester(0)

>>> F('spam') # State visible within closure only

spam 0

>>> F.state

AttributeError: 'function' object has no attribute 'state'

We need to declare variables nonlocal only if they must be changed (other enclosing scope name references are automatically retained as usual), and nonlocal names are still not visible outside the enclosing function.

Unfortunately, this code works in Python 3.X only. If you are using Python 2.X, other options are available, depending on your goals. The next three sections present some alternatives. Some of the code in these sections uses tools we haven’t covered yet and is intended partially as preview, but we’ll keep the examples simple here so that you can compare and contrast along the way.

State with Globals: A Single Copy Only

One common prescription for achieving the nonlocal effect in 2.X and earlier is to simply move the state out to the global scope (the enclosing module):

>>> def tester(start):

global state # Move it out to the module to change it

state = start # global allows changes in module scope

def nested(label):

global state

print(label, state)

state += 1

return nested

>>> F = tester(0)

>>> F('spam') # Each call increments shared global state

spam 0

>>> F('eggs')

eggs 1

This works in this case, but it requires global declarations in both functions and is prone to name collisions in the global scope (what if “state” is already being used?). A worse, and more subtle, problem is that it only allows for a single shared copy of the state information in the module scope—if we call tester again, we’ll wind up resetting the module’s state variable, such that prior calls will see their state overwritten:

>>> G = tester(42) # Resets state's single copy in global scope

>>> G('toast')

toast 42

>>> G('bacon')

bacon 43

>>> F('ham') # But my counter has been overwritten!

ham 44

As shown earlier, when you are using nonlocal and nested function closures instead of global, each call to tester remembers its own unique copy of the state object.

State with Classes: Explicit Attributes (Preview)

The other prescription for changeable state information in 2.X and earlier is to use classes with attributes to make state information access more explicit than the implicit magic of scope lookup rules. As an added benefit, each instance of a class gets a fresh copy of the state information, as a natural byproduct of Python’s object model. Classes also support inheritance, multiple behaviors, and other tools.

We haven’t explored classes in detail yet, but as a brief preview for comparison, the following is a reformulation of the earlier tester/nested functions as a class, which records state in objects explicitly as they are created. To make sense of this code, you need to know that a def within aclass like this works exactly like a normal def, except that the function’s self argument automatically receives the implied subject of the call (an instance object created by calling the class itself). The function named __init__ is run automatically when the class is called:

>>> class tester: # Class-based alternative (see Part VI)

def __init__(self, start): # On object construction,

self.state = start # save state explicitly in new object

def nested(self, label):

print(label, self.state) # Reference state explicitly

self.state += 1 # Changes are always allowed

>>> F = tester(0) # Create instance, invoke __init__

>>> F.nested('spam') # F is passed to self

spam 0

>>> F.nested('ham')

ham 1

In classes, we save every attribute explicitly, whether it’s changed or just referenced, and they are available outside the class. As for nested functions and nonlocal, the class alternative supports multiple copies of the retained data:

>>> G = tester(42) # Each instance gets new copy of state

>>> G.nested('toast') # Changing one does not impact others

toast 42

>>> G.nested('bacon')

bacon 43

>>> F.nested('eggs') # F's state is where it left off

eggs 2

>>> F.state # State may be accessed outside class

3

With just slightly more magic—which we’ll delve into later in this book—we could also make our class objects look like callable functions using operator overloading. __call__ intercepts direct calls on an instance, so we don’t need to call a named method:

>>> class tester:

def __init__(self, start):

self.state = start

def __call__(self, label): # Intercept direct instance calls

print(label, self.state) # So .nested() not required

self.state += 1

>>> H = tester(99)

>>> H('juice') # Invokes __call__

juice 99

>>> H('pancakes')

pancakes 100

Don’t sweat the details in this code too much at this point in the book; it’s mostly a preview, intended for general comparison to closures only. We’ll explore classes in depth in Part VI, and will look at specific operator overloading tools like __call__ in Chapter 30. The point to notice here is that classes can make state information more obvious, by leveraging explicit attribute assignment instead of implicit scope lookups. In addition, class attributes are always changeable and don’t require a nonlocal statement, and classes are designed to scale up to implementing richer objects with many attributes and behaviors.

While using classes for state information is generally a good rule of thumb to follow, they might also be overkill in cases like this, where state is a single counter. Such trivial state cases are more common than you might think; in such contexts, nested defs are sometimes more lightweight than coding classes, especially if you’re not familiar with OOP yet. Moreover, there are some scenarios in which nested defs may actually work better than classes—stay tuned for the description of method decorators in Chapter 39 for an example that is far beyond this chapter’s already well-stretched scope!

State with Function Attributes: 3.X and 2.X

As a portable and often simpler state-retention option, we can also sometimes achieve the same effect as nonlocals with function attributes—user-defined names attached to functions directly. When you attach user-defined attributes to nested functions generated by enclosing factory functions, they can also serve as per-call, multiple copy, and writeable state, just like nonlocal scope closures and class attributes. Such user-defined attribute names won’t clash with names Python creates itself, and as for nonlocal, need be used only for state variables that must be changed; other scope references are retained and work normally.

Crucially, this scheme is portable—like classes, but unlike nonlocal, function attributes work in both Python 3.X and 2.X. In fact, they’ve been available since 2.1, much longer than 3.X’s nonlocal. Because factory functions make a new function on each call anyhow, this does not require extra objects—the new function’s attributes become per-call state in much the same way as nonlocals, and are similarly associated with the generated function in memory.

Moreover, function attributes allow state variables to be accessed outside the nested function, like class attributes; with nonlocal, state variables can be seen directly only within the nested def. If you need to access a call counter externally, it’s a simple function attribute fetch in this model.

Here’s a final version of our example based on this technique—it replaces a nonlocal with an attribute attached to the nested function. This scheme may not seem as intuitive to some at first glance; you access state though the function’s name instead of as simple variables, and must initialize after the nested def. Still, it’s far more portable, allows state to be accessed externally, and saves a line by not requiring a nonlocal declaration:

>>> def tester(start):

def nested(label):

print(label, nested.state) # nested is in enclosing scope

nested.state += 1 # Change attr, not nested itself

nested.state = start # Initial state after func defined

return nested

>>> F = tester(0)

>>> F('spam') # F is a 'nested' with state attached

spam 0

>>> F('ham')

ham 1

>>> F.state # Can access state outside functions too

2

Because each call to the outer function produces a new nested function object, this scheme supports multiple copy per-call changeable data just like nonlocal closures and classes—a usage mode that global variables cannot provide:

>>> G = tester(42) # G has own state, doesn't overwrite F's

>>> G('eggs')

eggs 42

>>> F('ham')

ham 2

>>> F.state # State is accessible and per-call

3

>>> G.state

43

>>> F is G # Different function objects

False

This code relies on the fact that the function name nested is a local variable in the tester scope enclosing nested; as such, it can be referenced freely inside nested. This code also relies on the fact that changing an object in place is not an assignment to a name; when it incrementsnested.state, it is changing part of the object nested references, not the name nested itself. Because we’re not really assigning a name in the enclosing scope, no nonlocal declaration is required.

Function attributes are supported in both Python 3.X and 2.X; we’ll explore them further in Chapter 19. Importantly, we’ll see there that Python uses naming conventions in both 2.X and 3.X that ensure that the arbitrary names you assign as function attributes won’t clash with names related to internal implementation, making the namespace equivalent to a scope. Subjective factors aside, function attributes’ utility does overlap with the newer nonlocal in 3.X, making the latter technically redundant and far less portable.

State with mutables: Obscure ghost of Pythons past?

On a related note, it’s also possible to change a mutable object in the enclosing scope in 2.X and 3.X without declaring its name nonlocal. The following, for example, works the same as the previous version, is just as portable, and provides changeable per-call state:

def tester(start):

def nested(label):

print(label, state[0]) # Leverage in-place mutable change

state[0] += 1 # Extra syntax, deep magic?

state = [start]

return nested

This leverages the mutability of lists, and like function attributes, relies on the fact that in-place object changes do not classify a name as local. This is perhaps more obscure than either function attributes or 3.X’s nonlocal, though—a technique that predates even function attributes, and seems to lie today somewhere on the spectrum from clever hack to dark magic! You’re probably better off using named function attributes than lists and numeric offsets this way, though this may show up in code you must use.

To summarize: globals, nonlocals, classes, and function attributes all offer changeable state-retention options. Globals support only single-copy shared data; nonlocals can be changed in 3.X only; classes require a basic knowledge of OOP; and both classes and function attributes provide portable solutions that allow state to be accessed directly from outside the stateful callable object itself. As usual, the best tool for your program depends upon your program’s goals.

We’ll revisit all the state options introduced here in Chapter 39 in a more realistic context—decorators, a tool that by nature involves multilevel state retention. State options have additional selection factors (e.g., performance), which we’ll have to leave unexplored here for space (we’ll learn how to time code speed in Chapter 21). For now, it’s time to move on to explore argument passing modes.

WHY YOU WILL CARE: CUSTOMIZING OPEN

For another example of closures at work, consider changing the built-in open call to a custom version, as suggested in this chapter’s earlier sidebar Breaking the Universe in Python 2.X If the custom version needs to call the original, it must save it before changing it, and retain it for later use—a classic state retention scenario. Moreover, if we wish to support multiple customizations to the same function, globals won’t do: we need per-customizer state.

The following, coded for Python 3.X in file makeopen.py, is one way to achieve this (in 2.X, change the built-in scope name and prints). It uses a nested scope closure to remember a value for later use, without relying on global variables—which can clash and allow just one value, and without using a class—that may require more code than is warranted here:

import builtins

def makeopen(id):

original = builtins.open

def custom(*kargs, **pargs):

print('Custom open call %r:' % id , kargs, pargs)

return original(*kargs, **pargs)

builtins.open = custom

To change open for every module in a process, this code reassigns it in the built-in scope to a custom version coded with a nested def, after it saving the original in the enclosing scope so the customization can call it later. This code is also partially preview, as it relies on starred-argument forms to collect and later unpack arbitrary positional and keyword arguments meant for open—a topic coming up in the next chapter. Much of the magic here, though, is nested scope closures: the custom open found by the scope lookup rules retains the original for later use:

>>> F = open('script2.py') # Call built-in open in builtins

>>> F.read()

'import sys\nprint(sys.path)\nx = 2\nprint(x ** 32)\n'

>>> from makeopen import makeopen # Import open resetter function

>>> makeopen('spam') # Custom open calls built-in open

>>> F = open('script2.py') # Call custom open in builtins

Custom open call 'spam': ('script2.py',) {}

>>> F.read()

'import sys\nprint(sys.path)\nx = 2\nprint(x ** 32)\n'

Because each customization remembers the former built-in scope version in its own enclosing scope, they can even be nested naturally in ways that global variables cannot support—each call to the makeopen closure function remembers its own versions of id and original, so multiple customizations may be run:

>>> makeopen('eggs') # Nested customizers work too!

>>> F = open('script2.py') # Because each retains own state

Custom open call 'eggs': ('script2.py',) {}

Custom open call 'spam': ('script2.py',) {}

>>> F.read()

'import sys\nprint(sys.path)\nx = 2\nprint(x ** 32)\n'

As is, our function simply adds possibly nested call tracing to a built-in function, but the general technique may have other applications. A class-based equivalent to this may require more code because it would need to save the id and original values explicitly in object attributes—but requires more background knowledge than we yet have, so consider this a Part VI preview only:

import builtins

class makeopen: # See Part VI: call catches self()

def __init__(self, id):

self.id = id

self.original = builtins.open

builtins.open = self

def __call__(self, *kargs, **pargs):

print('Custom open call %r:' % self.id, kargs, pargs)

return self.original(*kargs, **pargs)

The point to notice here is that classes may be more explicit but also may take extra code when state retention is the only goal. We’ll see additional closure use cases later, especially when exploring decorators in Chapter 39, where we’ll find the closures are actually preferred to classes in certain roles.

Chapter Summary

In this chapter, we studied one of two key concepts related to functions: scopes, which determine how variables are looked up when used. As we learned, variables are considered local to the function definitions in which they are assigned, unless they are specifically declared to be global or nonlocal. We also explored some more advanced scope concepts here, including nested function scopes and function attributes. Finally, we looked at some general design ideas, such as the need to avoid globals and cross-file changes.

In the next chapter, we’re going to continue our function tour with the second key function-related concept: argument passing. As we’ll find, arguments are passed into a function by assignment, but Python also provides tools that allow functions to be flexible in how items are passed. Before we move on, let’s take this chapter’s quiz to review the scope concepts we’ve covered here.

Test Your Knowledge: Quiz

1. What is the output of the following code, and why?

2. >>> X = 'Spam'

3. >>> def func():

4. print(X)

5.

6. >>> func()

7. What is the output of this code, and why?

8. >>> X = 'Spam'

9. >>> def func():

10. X = 'NI!'

11.

12.>>> func()

13.>>> print(X)

14.What does this code print, and why?

15.>>> X = 'Spam'

16.>>> def func():

17. X = 'NI'

18. print(X)

19.

20.>>> func()

21.>>> print(X)

22.What output does this code produce? Why?

23.>>> X = 'Spam'

24.>>> def func():

25. global X

26. X = 'NI'

27.

28.>>> func()

29.>>> print(X)

30.What about this code—what’s the output, and why?

31.>>> X = 'Spam'

32.>>> def func():

33. X = 'NI'

34. def nested():

35. print(X)

36. nested()

37.

38.>>> func()

39.>>> X

40.How about this example: what is its output in Python 3.X, and why?

41.>>> def func():

42. X = 'NI'

43. def nested():

44. nonlocal X

45. X = 'Spam'

46. nested()

47. print(X)

48.

49.>>> func()

50.Name three or more ways to retain state information in a Python function.

Test Your Knowledge: Answers

1. The output here is 'Spam', because the function references a global variable in the enclosing module (because it is not assigned in the function, it is considered global).

2. The output here is 'Spam' again because assigning the variable inside the function makes it a local and effectively hides the global of the same name. The print statement finds the variable unchanged in the global (module) scope.

3. It prints 'NI' on one line and 'Spam' on another, because the reference to the variable within the function finds the assigned local and the reference in the print statement finds the global.

4. This time it just prints 'NI' because the global declaration forces the variable assigned inside the function to refer to the variable in the enclosing global scope.

5. The output in this case is again 'NI' on one line and 'Spam' on another, because the print statement in the nested function finds the name in the enclosing function’s local scope, and the print at the end finds the variable in the global scope.

6. This example prints 'Spam', because the nonlocal statement (available in Python 3.X but not 2.X) means that the assignment to X inside the nested function changes X in the enclosing function’s local scope. Without this statement, this assignment would classify X as local to the nested function, making it a different variable; the code would then print 'NI' instead.

7. Although the values of local variables go away when a function returns, you can make a Python function retain state information by using shared global variables, enclosing function scope references within nested functions, or using default argument values. Function attributes can sometimes allow state to be attached to the function itself, instead of looked up in scopes. Another alternative, using classes and OOP, sometimes supports state retention better than any of the scope-based techniques because it makes it explicit with attribute assignments; we’ll explore this option in Part VI.