Learning Python (2013)

Part IV. Functions and Generators

Chapter 18. Arguments

Chapter 17 explored the details behind Python’s scopes—the places where variables are defined and looked up. As we learned, the place where a name is defined in our code determines much of its meaning. This chapter continues the function story by studying the concepts in Pythonargument passing—the way that objects are sent to functions as inputs. As we’ll see, arguments (a.k.a. parameters) are assigned to names in a function, but they have more to do with object references than with variable scopes. We’ll also find that Python provides extra tools, such as keywords, defaults, and arbitrary argument collectors and extractors that allow for wide flexibility in the way arguments are sent to a function, and we’ll put them to work in examples.

Argument-Passing Basics

Earlier in this part of the book, I noted that arguments are passed by assignment. This has a few ramifications that aren’t always obvious to newcomers, which I’ll expand on in this section. Here is a rundown of the key points in passing arguments to functions:

§ Arguments are passed by automatically assigning objects to local variable names. Function arguments—references to (possibly) shared objects sent by the caller—are just another instance of Python assignment at work. Because references are implemented as pointers, all arguments are, in effect, passed by pointer. Objects passed as arguments are never automatically copied.

§ Assigning to argument names inside a function does not affect the caller. Argument names in the function header become new, local names when the function runs, in the scope of the function. There is no aliasing between function argument names and variable names in the scope of the caller.

§ Changing a mutable object argument in a function may impact the caller. On the other hand, as arguments are simply assigned to passed-in objects, functions can change passed-in mutable objects in place, and the results may affect the caller. Mutable arguments can be input and output for functions.

For more details on references, see Chapter 6; everything we learned there also applies to function arguments, though the assignment to argument names is automatic and implicit.

Python’s pass-by-assignment scheme isn’t quite the same as C++’s reference parameters option, but it turns out to be very similar to the argument-passing model of the C language (and others) in practice:

§ Immutable arguments are effectively passed “by value.” Objects such as integers and strings are passed by object reference instead of by copying, but because you can’t change immutable objects in place anyhow, the effect is much like making a copy.

§ Mutable arguments are effectively passed “by pointer.” Objects such as lists and dictionaries are also passed by object reference, which is similar to the way C passes arrays as pointers—mutable objects can be changed in place in the function, much like C arrays.

Of course, if you’ve never used C, Python’s argument-passing mode will seem simpler still—it involves just the assignment of objects to names, and it works the same whether the objects are mutable or not.

Arguments and Shared References

To illustrate argument-passing properties at work, consider the following code:

>>> def f(a): # a is assigned to (references) the passed object

a = 99 # Changes local variable a only

>>> b = 88

>>> f(b) # a and b both reference same 88 initially

>>> print(b) # b is not changed

88

In this example the variable a is assigned the object 88 at the moment the function is called with f(b), but a lives only within the called function. Changing a inside the function has no effect on the place where the function is called; it simply resets the local variable a to a completely different object.

That’s what is meant by a lack of name aliasing—assignment to an argument name inside a function (e.g., a=99) does not magically change a variable like b in the scope of the function call. Argument names may share passed objects initially (they are essentially pointers to those objects), but only temporarily, when the function is first called. As soon as an argument name is reassigned, this relationship ends.

At least, that’s the case for assignment to argument names themselves. When arguments are passed mutable objects like lists and dictionaries, we also need to be aware that in-place changes to such objects may live on after a function exits, and hence impact callers. Here’s an example that demonstrates this 'margin-top:0cm;margin-right:0cm;margin-bottom:0cm; margin-left:20.0pt;margin-bottom:.0001pt;line-height:normal;vertical-align: baseline'>>>> def changer(a, b): # Arguments assigned references to objects

a = 2 # Changes local name's value only

b[0] = 'spam' # Changes shared object in place

>>> X = 1

>>> L = [1, 2] # Caller:

>>> changer(X, L) # Pass immutable and mutable objects

>>> X, L # X is unchanged, L is different!

(1, ['spam', 2])

In this code, the changer function assigns values to argument a itself, and to a component of the object referenced by argument b. These two assignments within the function are only slightly different in syntax but have radically different results:

§ Because a is a local variable name in the function’s scope, the first assignment has no effect on the caller—it simply changes the local variable a to reference a completely different object, and does not change the binding of the name X in the caller’s scope. This is the same as in the prior example.

§ Argument b is a local variable name, too, but it is passed a mutable object (the list that L references in the caller’s scope). As the second assignment is an in-place object change, the result of the assignment to b[0] in the function impacts the value of L after the function returns.

Really, the second assignment statement in changer doesn’t change b—it changes part of the object that b currently references. This in-place change impacts the caller only because the changed object outlives the function call. The name L hasn’t changed either—it still references the same, changed object—but it seems as though L differs after the call because the value it references has been modified within the function. In effect, the list name L serves as both input to and output from the function.

Figure 18-1 illustrates the name/object bindings that exist immediately after the function has been called, and before its code has run.

If this example is still confusing, it may help to notice that the effect of the automatic assignments of the passed-in arguments is the same as running a series of simple assignment statements. In terms of the first argument, the assignment has no effect on the caller:

>>> X = 1

>>> a = X # They share the same object

>>> a = 2 # Resets 'a' only, 'X' is still 1

>>> print(X)

1

The assignment through the second argument does affect a variable at the call, though, because it is an in-place object change:

>>> L = [1, 2]

>>> b = L # They share the same object

>>> b[0] = 'spam' # In-place change: 'L' sees the change too

>>> print(L)

['spam', 2]

Figure 18-1. References: arguments. Because arguments are passed by assignment, argument names in the function may share objects with variables in the scope of the call. Hence, in-place changes to mutable arguments in a function can impact the caller. Here, a and b in the function initially reference the objects referenced by variables X and L when the function is first called. Changing the list through variable b makes L appear different after the call returns.

If you recall our discussions about shared mutable objects in Chapter 6 and Chapter 9, you’ll recognize the phenomenon at work: changing a mutable object in place can impact other references to that object. Here, the effect is to make one of the arguments work like both an input and an output of the function.

Avoiding Mutable Argument Changes

This behavior of in-place changes to mutable arguments isn’t a bug—it’s simply the way argument passing works in Python, and turns out to be widely useful in practice. Arguments are normally passed to functions by reference because that is what we normally want. It means we can pass large objects around our programs without making multiple copies along the way, and we can easily update these objects as we go. In fact, as we’ll see in Part VI, Python’s class model depends upon changing a passed-in “self” argument in place, to update object state.

If we don’t want in-place changes within functions to impact objects we pass to them, though, we can simply make explicit copies of mutable objects, as we learned in Chapter 6. For function arguments, we can always copy the list at the point of call, with tools like list, list.copy as of 3.3, or an empty slice:

L = [1, 2]

changer(X, L[:]) # Pass a copy, so our 'L' does not change

We can also copy within the function itself, if we never want to change passed-in objects, regardless of how the function is called:

def changer(a, b):

b = b[:] # Copy input list so we don't impact caller

a = 2

b[0] = 'spam' # Changes our list copy only

Both of these copying schemes don’t stop the function from changing the object—they just prevent those changes from impacting the caller. To really prevent changes, we can always convert to immutable objects to force the issue. Tuples, for example, raise an exception when changes are attempted:

L = [1, 2]

changer(X, tuple(L)) # Pass a tuple, so changes are errors

This scheme uses the built-in tuple function, which builds a new tuple out of all the items in a sequence (really, any iterable). It’s also something of an extreme—because it forces the function to be written to never change passed-in arguments, this solution might impose more limitations on the function than it should, and so should generally be avoided (you never know when changing arguments might come in handy for other calls in the future). Using this technique will also make the function lose the ability to call any list-specific methods on the argument, including methods that do not change the object in place.

The main point to remember here is that functions might update mutable objects like lists and dictionaries passed into them. This isn’t necessarily a problem if it’s expected, and often serves useful purposes. Moreover, functions that change passed-in mutable objects in place are probably designed and intended to do so—the change is likely part of a well-defined API that you shouldn’t violate by making copies.

However, you do have to be aware of this property—if objects change out from under you unexpectedly, check whether a called function might be responsible, and make copies when objects are passed if needed.

Simulating Output Parameters and Multiple Results

We’ve already discussed the return statement and used it in a few examples. Here’s another way to use this statement: because return can send back any sort of object, it can return multiple values by packaging them in a tuple or other collection type. In fact, although Python doesn’t support what some languages label “call by reference” argument passing, we can usually simulate it by returning tuples and assigning the results back to the original argument names in the caller:

>>> def multiple(x, y):

x = 2 # Changes local names only

y = [3, 4]

return x, y # Return multiple new values in a tuple

>>> X = 1

>>> L = [1, 2]

>>> X, L = multiple(X, L) # Assign results to caller's names

>>> X, L

(2, [3, 4])

It looks like the code is returning two values here, but it’s really just one—a two-item tuple with the optional surrounding parentheses omitted. After the call returns, we can use tuple assignment to unpack the parts of the returned tuple. (If you’ve forgotten why this works, flip back to “Tuples” in Chapter 4 and Chapter 9, and “Assignment Statements” in Chapter 11.) The net effect of this coding pattern is to both send back multiple results and simulate the output parameters of other languages by explicit assignments. Here, X and L change after the call, but only because the code said so.

NOTE

Unpacking arguments in Python 2.X: The preceding example unpacks a tuple returned by the function with tuple assignment. In Python 2.X, it’s also possible to automatically unpack tuples in arguments passed to a function. In 2.X (only), a function defined by this header:

def f((a, (b, c))):

can be called with tuples that match the expected structure: f((1, (2, 3))) assigns a, b, and c to 1, 2, and 3, respectively. Naturally, the passed tuple can also be an object created before the call (f(T)). This def syntax is no longer supported in Python 3.X. Instead, code this function as:

def f(T): (a, (b, c)) = T

to unpack in an explicit assignment statement. This explicit form works in both 3.X and 2.X. Argument unpacking is reportedly an obscure and rarely used feature in Python 2.X (except in code that uses it!). Moreover, a function header in 2.X supports only the tuple form of sequence assignment; more general sequence assignments (e.g., def f((a, [b, c])):) fail on syntax errors in 2.X as well and require the explicit assignment form mandated in 3.X. Conversely, arbitrary sequences in the call successfully match tuples in the header (e.g., f((1, [2, 3])), f((1, "ab"))).

Tuple unpacking argument syntax is also disallowed by 3.X in lambda function argument lists: see the Chapter 20 sidebar Why You Will Care: List Comprehensions and map for a lambda unpacking example. Somewhat asymmetrically, tuple unpacking assignment is still automatic in 3.X for loops targets; see Chapter 13 for examples.

Special Argument-Matching Modes

As we’ve just seen, arguments are always passed by assignment in Python; names in the def header are assigned to passed-in objects. On top of this model, though, Python provides additional tools that alter the way the argument objects in a call are matched with argument names in the header prior to assignment. These tools are all optional, but they allow us to write functions that support more flexible calling patterns, and you may encounter some libraries that require them.

By default, arguments are matched by position, from left to right, and you must pass exactly as many arguments as there are argument names in the function header. However, you can also specify matching by name, provide default values, and use collectors for extra arguments.

Argument Matching Basics

Before we go into the syntactic details, I want to stress that these special modes are optional and deal only with matching objects to names; the underlying passing mechanism after the matching takes place is still assignment. In fact, some of these tools are intended more for people writing libraries than for application developers. But because you may stumble across these modes even if you don’t code them yourself, here’s a synopsis of the available tools:

Positionals: matched from left to right

The normal case, which we’ve mostly been using so far, is to match passed argument values to argument names in a function header by position, from left to right.

Keywords: matched by argument name

Alternatively, callers can specify which argument in the function is to receive a value by using the argument’s name in the call, with the name=value syntax.

Defaults: specify values for optional arguments that aren’t passed

Functions themselves can specify default values for arguments to receive if the call passes too few values, again using the name=value syntax.

Varargs collecting: collect arbitrarily many positional or keyword arguments

Functions can use special arguments preceded with one or two * characters to collect an arbitrary number of possibly extra arguments. This feature is often referred to as varargs, after a variable-length argument list tool in the C language; in Python, the arguments are collected in a normal object.

Varargs unpacking: pass arbitrarily many positional or keyword arguments

Callers can also use the * syntax to unpack argument collections into separate arguments. This is the inverse of a * in a function header—in the header it means collect arbitrarily many arguments, while in the call it means unpack arbitrarily many arguments, and pass them individually as discrete values.

Keyword-only arguments: arguments that must be passed by name

In Python 3.X (but not 2.X), functions can also specify arguments that must be passed by name with keyword arguments, not by position. Such arguments are typically used to define configuration options in addition to actual arguments.

Argument Matching Syntax

Table 18-1 summarizes the syntax that invokes the special argument-matching modes.

Table 18-1. Function argument-matching forms

These special matching modes break down into function calls and definitions as follows:

§ In a function call (the first four rows of the table), simple values are matched by position, but using the name=value form tells Python to match by name to arguments instead; these are called keyword arguments. Using a *iterable or **dict in a call allows us to package up arbitrarily many positional or keyword objects in sequences (and other iterables) and dictionaries, respectively, and unpack them as separate, individual arguments when they are passed to the function.

§ In a function header (the rest of the table), a simple name is matched by position or name depending on how the caller passes it, but the name=value form specifies a default value. The *name form collects any extra unmatched positional arguments in a tuple, and the **name form collects extra keyword arguments in a dictionary. In Python 3.X, any normal or defaulted argument names following a *name or a bare * are keyword-only arguments and must be passed by keyword in calls.

Of these, keyword arguments and defaults are probably the most commonly used in Python code. We’ve informally used both of these earlier in this book:

§ We’ve already used keywords to specify options to the 3.X print function, but they are more general—keywords allow us to label any argument with its name, to make calls more informational.

§ We met defaults earlier, too, as a way to pass in values from the enclosing function’s scope, but they are also more general—they allow us to make any argument optional, providing its default value in a function definition.

As we’ll see, the combination of defaults in a function header and keywords in a call further allows us to pick and choose which defaults to override.

In short, special argument-matching modes let you be fairly liberal about how many arguments must be passed to a function. If a function specifies defaults, they are used if you pass too few arguments. If a function uses the * variable argument list forms, you can seemingly pass too manyarguments; the * names collect the extra arguments in data structures for processing in the function.

The Gritty Details

If you choose to use and combine the special argument-matching modes, Python will ask you to follow these ordering rules among the modes’ optional components:

§ In a function call, arguments must appear in this order: any positional arguments (value); followed by a combination of any keyword arguments (name=value) and the *iterable form; followed by the **dict form.

§ In a function header, arguments must appear in this order: any normal arguments (name); followed by any default arguments (name=value); followed by the *name (or * in 3.X) form; followed by any name or name=value keyword-only arguments (in 3.X); followed by the **nameform.

In both the call and header, the **args form must appear last if present. If you mix arguments in any other order, you will get a syntax error because the combinations can be ambiguous. The steps that Python internally carries out to match arguments before assignment can roughly be described as follows:

1. Assign nonkeyword arguments by position.

2. Assign keyword arguments by matching names.

3. Assign extra nonkeyword arguments to *name tuple.

4. Assign extra keyword arguments to **name dictionary.

5. Assign default values to unassigned arguments in header.

After this, Python checks to make sure each argument is passed just one value; if not, an error is raised. When all matching is complete, Python assigns argument names to the objects passed to them.

The actual matching algorithm Python uses is a bit more complex (it must also account for keyword-only arguments in 3.X, for instance), so we’ll defer to Python’s standard language manual for a more exact description. It’s not required reading, but tracing Python’s matching algorithm may help you to understand some convoluted cases, especially when modes are mixed.

NOTE

In Python 3.X only, argument names in a function header can also have annotation values, specified as name:value (or name:value=default when defaults are present). This is simply additional syntax for arguments and does not augment or change the argument-ordering rules described here. The function itself can also have an annotation value, given as def f()->value. Python attaches annotation values to the function object. See the discussion of function annotation in Chapter 19 for more details.

Keyword and Default Examples

This is all simpler in code than the preceding descriptions may imply. If you don’t use any special matching syntax, Python matches names by position from left to right, like most other languages. For instance, if you define a function that requires three arguments, you must call it with three arguments:

>>> def f(a, b, c): print(a, b, c)

>>> f(1, 2, 3)

1 2 3

Here, we pass by position—a is matched to 1, b is matched to 2, and so on (this works the same in Python 3.X and 2.X, but extra tuple parentheses are displayed in 2.X because we’re using 3.X print calls again).

Keywords

In Python, though, you can be more specific about what goes where when you call a function. Keyword arguments allow us to match by name, instead of by position. Using the same function:

>>> f(c=3, b=2, a=1)

1 2 3

The c=3 in this call, for example, means send 3 to the argument named c. More formally, Python matches the name c in the call to the argument named c in the function definition’s header, and then passes the value 3 to that argument. The net effect of this call is the same as that of the prior call, but notice that the left-to-right order of the arguments no longer matters when keywords are used because arguments are matched by name, not by position. It’s even possible to combine positional and keyword arguments in a single call. In this case, all positionals are matched first from left to right in the header, before keywords are matched by name:

>>> f(1, c=3, b=2) # a gets 1 by position, b and c passed by name

1 2 3

When most people see this the first time, they wonder why one would use such a tool. Keywords typically have two roles in Python. First, they make your calls a bit more self-documenting (assuming that you use better argument names than a, b, and c!). For example, a call of this form:

func(name='Bob', age=40, job='dev')

is much more meaningful than a call with three naked values separated by commas, especially in larger programs—the keywords serve as labels for the data in the call. The second major use of keywords occurs in conjunction with defaults, which we turn to next.

Defaults

We talked about defaults in brief earlier, when discussing nested function scopes. In short, defaults allow us to make selected function arguments optional; if not passed a value, the argument is assigned its default before the function runs. For example, here is a function that requires one argument and defaults two:

>>> def f(a, b=2, c=3): print(a, b, c) # a required, b and c optional

When we call this function, we must provide a value for a, either by position or by keyword; however, providing values for b and c is optional. If we don’t pass values to b and c, they default to 2 and 3, respectively:

>>> f(1) # Use defaults

1 2 3

>>> f(a=1)

1 2 3

If we pass two values, only c gets its default, and with three values, no defaults are used:

>>> f(1, 4) # Override defaults

1 4 3

>>> f(1, 4, 5)

1 4 5

Finally, here is how the keyword and default features interact. Because they subvert the normal left-to-right positional mapping, keywords allow us to essentially skip over arguments with defaults:

>>> f(1, c=6) # Choose defaults

1 2 6

Here, a gets 1 by position, c gets 6 by keyword, and b, in between, defaults to 2.

Be careful not to confuse the special name=value syntax in a function header and a function call; in the call it means a match-by-name keyword argument, while in the header it specifies a default for an optional argument. In both cases, this is not an assignment statement (despite its appearance); it is special syntax for these two contexts, which modifies the default argument-matching mechanics.

Combining keywords and defaults

Here is a slightly larger example that demonstrates keywords and defaults in action. In the following, the caller must always pass at least two arguments (to match spam and eggs), but the other two are optional. If they are omitted, Python assigns toast and ham to the defaults specified in the header:

def func(spam, eggs, toast=0, ham=0): # First 2 required

print((spam, eggs, toast, ham))

func(1, 2) # Output: (1, 2, 0, 0)

func(1, ham=1, eggs=0) # Output: (1, 0, 0, 1)

func(spam=1, eggs=0) # Output: (1, 0, 0, 0)

func(toast=1, eggs=2, spam=3) # Output: (3, 2, 1, 0)

func(1, 2, 3, 4) # Output: (1, 2, 3, 4)

Notice again that when keyword arguments are used in the call, the order in which the arguments are listed doesn’t matter; Python matches by name, not by position. The caller must supply values for spam and eggs, but they can be matched by position or by name. Again, keep in mind that the form name=value means different things in the call and the def: a keyword in the call and a default in the header.

NOTE

Beware mutable defaults: As footnoted in the prior chapter, if you code a default to be a mutable object (e.g., def f(a=[])), the same, single mutable object is reused every time the function is later called—even if it is changed in place within the function. The net effect is that the argument’s default retains its value from the prior call, and is not reset to its original value coded in the defheader. To reset anew on each call, move the assignment into the function body instead. Mutable defaults allow state retention, but this is often a surprise. Since this is such a common trap, we’ll postpone further exploration until this part’s “gotchas” list at the end of Chapter 21.

Arbitrary Arguments Examples

The last two matching extensions, * and **, are designed to support functions that take any number of arguments. Both can appear in either the function definition or a function call, and they have related purposes in the two locations.

Headers: Collecting arguments

The first use, in the function definition, collects unmatched positional arguments into a tuple:

>>> def f(*args): print(args)

When this function is called, Python collects all the positional arguments into a new tuple and assigns the variable args to that tuple. Because it is a normal tuple object, it can be indexed, stepped through with a for loop, and so on:

>>> f()

()

>>> f(1)

(1,)

>>> f(1, 2, 3, 4)

(1, 2, 3, 4)

The ** feature is similar, but it only works for keyword arguments—it collects them into a new dictionary, which can then be processed with normal dictionary tools. In a sense, the ** form allows you to convert from keywords to dictionaries, which you can then step through with keyscalls, dictionary iterators, and the like (this is roughly what the dict call does when passed keywords, but it returns the new dictionary):

>>> def f(**args): print(args)

>>> f()

{}

>>> f(a=1, b=2)

{'a': 1, 'b': 2}

Finally, function headers can combine normal arguments, the *, and the ** to implement wildly flexible call signatures. For instance, in the following, 1 is passed to a by position, 2 and 3 are collected into the pargs positional tuple, and x and y wind up in the kargs keyword dictionary:

>>> def f(a, *pargs, **kargs): print(a, pargs, kargs)

>>> f(1, 2, 3, x=1, y=2)

1 (2, 3) {'y': 2, 'x': 1}

Such code is rare, but shows up in functions that need to support multiple call patterns (for backward compatibility, for instance). In fact, these features can be combined in even more complex ways that may seem ambiguous at first glance—an idea we will revisit later in this chapter. First, though, let’s see what happens when * and ** are coded in function calls instead of definitions.

Calls: Unpacking arguments

In all recent Python releases, we can use the * syntax when we call a function, too. In this context, its meaning is the inverse of its meaning in the function definition—it unpacks a collection of arguments, rather than building a collection of arguments. For example, we can pass four arguments to a function in a tuple and let Python unpack them into individual arguments:

>>> def func(a, b, c, d): print(a, b, c, d)

>>> args = (1, 2)

>>> args += (3, 4)

>>> func(*args) # Same as func(1, 2, 3, 4)

1 2 3 4

Similarly, the ** syntax in a function call unpacks a dictionary of key/value pairs into separate keyword arguments:

>>> args = {'a': 1, 'b': 2, 'c': 3}

>>> args['d'] = 4

>>> func(**args) # Same as func(a=1, b=2, c=3, d=4)

1 2 3 4

Again, we can combine normal, positional, and keyword arguments in the call in very flexible ways:

>>> func(*(1, 2), **{'d': 4, 'c': 3}) # Same as func(1, 2, d=4, c=3)

1 2 3 4

>>> func(1, *(2, 3), **{'d': 4}) # Same as func(1, 2, 3, d=4)

1 2 3 4

>>> func(1, c=3, *(2,), **{'d': 4}) # Same as func(1, 2, c=3, d=4)

1 2 3 4

>>> func(1, *(2, 3), d=4) # Same as func(1, 2, 3, d=4)

1 2 3 4

>>> func(1, *(2,), c=3, **{'d':4}) # Same as func(1, 2, c=3, d=4)

1 2 3 4

This sort of code is convenient when you cannot predict the number of arguments that will be passed to a function when you write your script; you can build up a collection of arguments at runtime instead and call the function generically this way. Again, don’t confuse the */** starred-argument syntax in the function header and the function call—in the header it collects any number of arguments, while in the call it unpacks any number of arguments. In both, one star means positionals, and two applies to keywords.

NOTE

As we saw in Chapter 14, the *pargs form in a call is an iteration context, so technically it accepts any iterable object, not just tuples or other sequences as shown in the examples here. For instance, a file object works after the *, and unpacks its lines into individual arguments (e.g., func(*open('fname')). Watch for additional examples of this utility in Chapter 20, after we study generators.

This generality is supported in both Python 3.X and 2.X, but it holds true only for calls—a *pargs in a call allows any iterable, but the same form in a def header always bundles extra arguments into a tuple. This header behavior is similar in spirit and syntax to the * in Python 3.X extended sequence unpacking assignment forms we met in Chapter 11 (e.g., x, *y = z), though that star usage always creates lists, not tuples.

Applying functions generically

The prior section’s examples may seem academic (if not downright esoteric), but they are used more often than you might expect. Some programs need to call arbitrary functions in a generic fashion, without knowing their names or arguments ahead of time. In fact, the real power of the special “varargs” call syntax is that you don’t need to know how many arguments a function call requires before you write a script. For example, you can use if logic to select from a set of functions and argument lists, and call any of them generically (functions in some of the following examples are hypothetical):

if sometest:

action, args = func1, (1,) # Call func1 with one arg in this case

else:

action, args = func2, (1, 2, 3) # Call func2 with three args here

...etc...

action(*args) # Dispatch generically

This leverages both the * form, and the fact that functions are objects that may be both referenced by, and called through, any variable. More generally, this varargs call syntax is useful anytime you cannot predict the arguments list. If your user selects an arbitrary function via a user interface, for instance, you may be unable to hardcode a function call when writing your script. To work around this, simply build up the arguments list with sequence operations, and call it with starred-argument syntax to unpack the arguments:

>>> ...define or import func3...

>>> args = (2,3)

>>> args += (4,)

>>> args

(2, 3, 4)

>>> func3(*args)

Because the arguments list is passed in as a tuple here, the program can build it at runtime. This technique also comes in handy for functions that test or time other functions. For instance, in the following code we support any function with any arguments by passing along whatever arguments were sent in (this is file tracer0.py in the book examples package):

def tracer(func, *pargs, **kargs): # Accept arbitrary arguments

print('calling:', func.__name__)

return func(*pargs, **kargs) # Pass along arbitrary arguments

def func(a, b, c, d):

return a + b + c + d

print(tracer(func, 1, 2, c=3, d=4))

This code uses the built-in __name__ attribute attached to every function (as you might expect, it’s the function’s name string), and uses stars to collect and then unpack the arguments intended for the traced function. In other words, when this code is run, arguments are intercepted by the tracer and then propagated with varargs call syntax:

calling: func

10

For another example of this technique, see the preview near the end of the preceding chapter, where it was used to reset the built-in open function. We’ll code additional examples of such roles later in this book; see especially the sequence timing examples in Chapter 21 and the various decorator utilities we will code in Chapter 39. It’s a common technique in general tools.

The defunct apply built-in (Python 2.X)

Prior to Python 3.X, the effect of the *args and **args varargs call syntax could be achieved with a built-in function named apply. This original technique has been removed in 3.X because it is now redundant (3.X cleans up many such dusty tools that have been subsumed over the years). It’s still available in all Python 2.X releases, though, and you may come across it in older 2.X code.

In short, the following are equivalent prior to Python 3.X:

func(*pargs, **kargs) # Newer call syntax: func(*sequence, **dict)

apply(func, pargs, kargs) # Defunct built-in: apply(func, sequence, dict)

For example, consider the following function, which accepts any number of positional or keyword arguments:

>>> def echo(*args, **kwargs): print(args, kwargs)

>>> echo(1, 2, a=3, b=4)

(1, 2) {'a': 3, 'b': 4}

In Python 2.X, we can call it generically with apply, or with the call syntax that is now required in 3.X:

>>> pargs = (1, 2)

>>> kargs = {'a':3, 'b':4}

>>> apply(echo, pargs, kargs)

(1, 2) {'a': 3, 'b': 4}

>>> echo(*pargs, **kargs)

(1, 2) {'a': 3, 'b': 4}

Both forms work for built-in functions in 2.X too (notice 2.X’s trailing L for its long integers):

>>> apply(pow, (2, 100))

1267650600228229401496703205376L

>>> pow(*(2, 100))

1267650600228229401496703205376L

The unpacking call syntax form is newer than the apply function, is preferred in general, and is required in 3.X. (Technically, it was added in 2.0, was documented as deprecated in 2.3, is still usable without warning in 2.7, and is gone in 3.0 and later.) Apart from its symmetry with the *collector forms in def headers, and the fact that it requires fewer keystrokes, the newer call syntax also allows us to pass along additional arguments without having to manually extend argument sequences or dictionaries:

>>> echo(0, c=5, *pargs, **kargs) # Normal, keyword, *sequence, **dictionary

(0, 1, 2) {'a': 3, 'c': 5, 'b': 4}

That is, the call syntax form is more general. Since it’s required in 3.X, you should now disavow all knowledge of apply (unless, of course, it appears in 2.X code you must use or maintain...).

Python 3.X Keyword-Only Arguments

Python 3.X generalizes the ordering rules in function headers to allow us to specify keyword-only arguments—arguments that must be passed by keyword only and will never be filled in by a positional argument. This is useful if we want a function to both process any number of arguments and accept possibly optional configuration options.

Syntactically, keyword-only arguments are coded as named arguments that may appear after *args in the arguments list. All such arguments must be passed using keyword syntax in the call. For example, in the following, a may be passed by name or position, b collects any extra positional arguments, and c must be passed by keyword only. In 3.X:

>>> def kwonly(a, *b, c):

print(a, b, c)

>>> kwonly(1, 2, c=3)

1 (2,) 3

>>> kwonly(a=1, c=3)

1 () 3

>>> kwonly(1, 2, 3)

TypeError: kwonly() missing 1 required keyword-only argument: 'c'

We can also use a * character by itself in the arguments list to indicate that a function does not accept a variable-length argument list but still expects all arguments following the * to be passed as keywords. In the next function, a may be passed by position or name again, but b and c must be keywords, and no extra positionals are allowed:

>>> def kwonly(a, *, b, c):

print(a, b, c)

>>> kwonly(1, c=3, b=2)

1 2 3

>>> kwonly(c=3, b=2, a=1)

1 2 3

>>> kwonly(1, 2, 3)

TypeError: kwonly() takes 1 positional argument but 3 were given

>>> kwonly(1)

TypeError: kwonly() missing 2 required keyword-only arguments: 'b' and 'c'

You can still use defaults for keyword-only arguments, even though they appear after the * in the function header. In the following code, a may be passed by name or position, and b and c are optional but must be passed by keyword if used:

>>> def kwonly(a, *, b='spam', c='ham'):

print(a, b, c)

>>> kwonly(1)

1 spam ham

>>> kwonly(1, c=3)

1 spam 3

>>> kwonly(a=1)

1 spam ham

>>> kwonly(c=3, b=2, a=1)

1 2 3

>>> kwonly(1, 2)

TypeError: kwonly() takes 1 positional argument but 2 were given

In fact, keyword-only arguments with defaults are optional, but those without defaults effectively become required keywords for the function:

>>> def kwonly(a, *, b, c='spam'):

print(a, b, c)

>>> kwonly(1, b='eggs')

1 eggs spam

>>> kwonly(1, c='eggs')

TypeError: kwonly() missing 1 required keyword-only argument: 'b'

>>> kwonly(1, 2)

TypeError: kwonly() takes 1 positional argument but 2 were given

>>> def kwonly(a, *, b=1, c, d=2):

print(a, b, c, d)

>>> kwonly(3, c=4)

3 1 4 2

>>> kwonly(3, c=4, b=5)

3 5 4 2

>>> kwonly(3)

TypeError: kwonly() missing 1 required keyword-only argument: 'c'

>>> kwonly(1, 2, 3)

TypeError: kwonly() takes 1 positional argument but 3 were given

Ordering rules

Finally, note that keyword-only arguments must be specified after a single star, not two—named arguments cannot appear after the **args arbitrary keywords form, and a ** can’t appear by itself in the arguments list. Both attempts generate a syntax error:

>>> def kwonly(a, **pargs, b, c):

SyntaxError: invalid syntax

>>> def kwonly(a, **, b, c):

SyntaxError: invalid syntax

This means that in a function header, keyword-only arguments must be coded before the **args arbitrary keywords form and after the *args arbitrary positional form, when both are present. Whenever an argument name appears before *args, it is a possibly default positional argument, not keyword-only:

>>> def f(a, *b, **d, c=6): print(a, b, c, d) # Keyword-only before **!

SyntaxError: invalid syntax

>>> def f(a, *b, c=6, **d): print(a, b, c, d) # Collect args in header

>>> f(1, 2, 3, x=4, y=5) # Default used

1 (2, 3) 6 {'y': 5, 'x': 4}

>>> f(1, 2, 3, x=4, y=5, c=7) # Override default

1 (2, 3) 7 {'y': 5, 'x': 4}

>>> f(1, 2, 3, c=7, x=4, y=5) # Anywhere in keywords

1 (2, 3) 7 {'y': 5, 'x': 4}

>>> def f(a, c=6, *b, **d): print(a, b, c, d) # c is not keyword-only here!

>>> f(1, 2, 3, x=4)

1 (3,) 2 {'x': 4}

In fact, similar ordering rules hold true in function calls: when keyword-only arguments are passed, they must appear before a **args form. The keyword-only argument can be coded either before or after the *args, though, and may be included in **args:

>>> def f(a, *b, c=6, **d): print(a, b, c, d) # KW-only between * and **

>>> f(1, *(2, 3), **dict(x=4, y=5)) # Unpack args at call

1 (2, 3) 6 {'y': 5, 'x': 4}

>>> f(1, *(2, 3), **dict(x=4, y=5), c=7) # Keywords before **args!

SyntaxError: invalid syntax

>>> f(1, *(2, 3), c=7, **dict(x=4, y=5)) # Override default

1 (2, 3) 7 {'y': 5, 'x': 4}

>>> f(1, c=7, *(2, 3), **dict(x=4, y=5)) # After or before *

1 (2, 3) 7 {'y': 5, 'x': 4}

>>> f(1, *(2, 3), **dict(x=4, y=5, c=7)) # Keyword-only in **

1 (2, 3) 7 {'y': 5, 'x': 4}

Trace through these cases on your own, in conjunction with the general argument-ordering rules described formally earlier. They may appear to be worst cases in the artificial examples here, but they can come up in real practice, especially for people who write libraries and tools for other Python programmers to use.

Why keyword-only arguments?

So why care about keyword-only arguments? In short, they make it easier to allow a function to accept both any number of positional arguments to be processed, and configuration options passed as keywords. While their use is optional, without keyword-only arguments extra work may be required to provide defaults for such options and to verify that no superfluous keywords were passed.

Imagine a function that processes a set of passed-in objects and allows a tracing flag to be passed:

process(X, Y, Z) # Use flag's default

process(X, Y, notify=True) # Override flag default

Without keyword-only arguments we have to use both *args and **args and manually inspect the keywords, but with keyword-only arguments less code is required. The following guarantees that no positional argument will be incorrectly matched against notify and requires that it be a keyword if passed:

def process(*args, notify=False): ...

Since we’re going to see a more realistic example of this later in this chapter, in “Emulating the Python 3.X print Function,” I’ll postpone the rest of this story until then. For an additional example of keyword-only arguments in action, see the iteration options timing case study in Chapter 21. And for additional function definition enhancements in Python 3.X, stay tuned for the discussion of function annotation syntax in Chapter 19.

The min Wakeup Call!

OK—it’s time for something more realistic. To make this chapter’s concepts more concrete, let’s work through an exercise that demonstrates a practical application of argument-matching tools.

Suppose you want to code a function that is able to compute the minimum value from an arbitrary set of arguments and an arbitrary set of object data types. That is, the function should accept zero or more arguments, as many as you wish to pass. Moreover, the function should work for all kinds of Python object types: numbers, strings, lists, lists of dictionaries, files, and even None.

The first requirement provides a natural example of how the * feature can be put to good use—we can collect arguments into a tuple and step over each of them in turn with a simple for loop. The second part of the problem definition is easy: because every object type supports comparisons, we don’t have to specialize the function per type (an application of polymorphism); we can simply compare objects blindly and let Python worry about what sort of comparison to perform according to the objects being compared.

Full Credit

The following file shows three ways to code this operation, at least one of which was suggested by a student in one of my courses (this example is often a group exercise to circumvent dozing after lunch):

§ The first function fetches the first argument (args is a tuple) and traverses the rest by slicing off the first (there’s no point in comparing an object to itself, especially if it might be a large structure).

§ The second version lets Python pick off the first and rest of the arguments automatically, and so avoids an index and slice.

§ The third converts from a tuple to a list with the built-in list call and employs the list sort method.

The sort method is coded in C, so it can be quicker than the other approaches at times, but the linear scans of the first two techniques may make them faster much of the time.^[38^] The file mins.py contains the code for all three solutions:

def min1(*args):

res = args[0]

for arg in args[1:]:

if arg < res:

res = arg

return res

def min2(first, *rest):

for arg in rest:

if arg < first:

first = arg

return first

def min3(*args):

tmp = list(args) # Or, in Python 2.4+: return sorted(args)[0]

tmp.sort()

return tmp[0]

print(min1(3, 4, 1, 2))

print(min2("bb", "aa"))

print(min3([2,2], [1,1], [3,3]))

All three solutions produce the same result when the file is run. Try typing a few calls interactively to experiment with these on your own:

% python mins.py

1

aa

[1, 1]

Notice that none of these three variants tests for the case where no arguments are passed in. They could, but there’s no point in doing so here—in all three solutions, Python will automatically raise an exception if no arguments are passed in. The first variant raises an exception when we try to fetch item 0, the second when Python detects an argument list mismatch, and the third when we try to return item 0 at the end.

This is exactly what we want to happen—because these functions support any data type, there is no valid sentinel value that we could pass back to designate an error, so we may as well let the exception be raised. There are exceptions to this rule (e.g., you might test for errors yourself if you’d rather avoid actions run before reaching the code that triggers an error automatically), but in general it’s better to assume that arguments will work in your functions’ code and let Python raise errors for you when they do not.

Bonus Points

You can get bonus points here for changing these functions to compute the maximum, rather than minimum, values. This one’s easy: the first two versions only require changing < to >, and the third simply requires that we return tmp[−1] instead of tmp[0]. For an extra point, be sure to set the function name to “max” as well (though this part is strictly optional).

It’s also possible to generalize a single function to compute either a minimum or a maximum value, by evaluating comparison expression strings with a tool like the eval built-in function (see the library manual, and various appearances here, especially in Chapter 10) or passing in an arbitrary comparison function. The file minmax.py shows how to implement the latter scheme:

def minmax(test, *args):

res = args[0]

for arg in args[1:]:

if test(arg, res):

res = arg

return res

def lessthan(x, y): return x < y # See also: lambda, eval

def grtrthan(x, y): return x > y

print(minmax(lessthan, 4, 2, 1, 5, 6, 3)) # Self-test code

print(minmax(grtrthan, 4, 2, 1, 5, 6, 3))

% python minmax.py

1

6

Functions are another kind of object that can be passed into a function like this one. To make this a max (or other) function, for example, we simply pass in the right sort of test function. This may seem like extra work, but the main point of generalizing functions this way—instead of cutting and pasting to change just a single character—is that we’ll only have one version to change in the future, not two.

The Punch Line...

Of course, all this was just a coding exercise. There’s really no reason to code min or max functions, because both are built-ins in Python! We met them briefly in Chapter 5 in conjunction with numeric tools, and again in Chapter 14 when exploring iteration contexts. The built-in versions work almost exactly like ours, but they’re coded in C for optimal speed and accept either a single iterable or multiple arguments. Still, though it’s superfluous in this context, the general coding pattern we used here might be useful in other scenarios.