Python Unlocked (2015)
Chapter 3. Functions and Utilities
After learning about how objects are linked to one another, let's take a look at the functions that are the means to execute code in language. We will discuss how to define and call functions with various combinations. Then, we will cover some very useful utilities that are available to us to use in day-to-day programming. We will cover the following topics:
· Defining functions
· Decorating callables
· Utilities
Defining functions
Key 1: How to define functions.
Functions are used to group a set of instructions and logic that performs a specific task. So, we should make functions perform one specific task and choose a name that gives us a hint about that. If a function is important and performs complex stuff, we should always add docstrings to this function so that it is easy for us to later visit and modify this function.
While defining a function, we can define the following:
1. Positional arguments (simply pass the object according to position), which are as follows:
2. >>> def foo(a,b):
3. ... print(a,b)
4. ...
5. >>> foo(1,2)
1 2
6. Default arguments (if value is not passed, the default is used), which are as follows:
7. >>> def foo(a,b=3):
8. ... print(a,b)
9. ...
10.>>> foo(3)
11.3 3
12.>>> foo(3,4)
3 4
13. Keyword only arguments (must be passed as a positional or as a keyword argument), which are as follows:
14.>>> def foo(a,*,b):
15.... print(a,b)
16....
17.>>> foo(2,3)
18.Traceback (most recent call last):
19. File "<stdin>", line 1, in <module>
20.TypeError: foo() takes 1 positional argument but 2 were given
21.>>> foo(1,b=4)
1 4
22. An argument list, which is as follows:
23.>>> def foo(a,*pa):
24.... print(a,pa)
25....
26.>>> foo(1)
27.1 ()
28.>>> foo(1,2)
29.1 (2,)
30.>>> foo(1,2,3)
1 (2, 3)
31. A keyword argument dictionary, which is as follows:
32.>>> def foo(a,**kw):
33.... print(a,kw)
34....
35.>>> foo(2)
36.2 {}
37.>>> foo(2,b=4)
38.2 {'b': 4}
39.>>> foo(2,b=4,v=5)
2 {'b': 4, 'v': 5}
When a function is called, this is how arguments are passed on:
40. All positional arguments that are passed are consumed.
41. If the function takes an argument list and there are more passed positional arguments after the first step, then the rest of the arguments are collected in an argument list:
42.>>> def foo1(a,*args):
43.... print(a,args)
44....
45.>>> def foo2(a,):
46.... print(a)
47....
48.>>> foo1(1,2,3,4)
49.1 (2, 3, 4)
50.>>> foo2(1,2,3,4)
51.Traceback (most recent call last):
52. File "<stdin>", line 1, in <module>
TypeError: foo2() takes 1 positional argument but 4 were given
53. If passed position arguments are less than the defined positional arguments, then the passed keyword arguments are used for values for positional arguments. If no keyword argument is found for the positional argument, we get an error:
54.>>> def foo(a,b,c):
55.... print(a,b,c)
56....
57.>>> foo(1,c=3,b=2)
58.1 2 3
59.>>> foo(1,b=2)
60.Traceback (most recent call last):
61. File "<stdin>", line 1, in <module>
TypeError: foo() missing 1 required positional argument: 'c'
62. Passed keyword variables are used only for keyword arguments:
63.>>> def foo(a,b,*,c):
64.... print(a,b,c)
65....
66.>>> foo(1,2,3)
67.Traceback (most recent call last):
68. File "<stdin>", line 1, in <module>
69.TypeError: foo() takes 2 positional arguments but 3 were given
70.>>> foo(1,2,c=3)
71.1 2 3
72.>>> foo(c=3,b=2,a=1)
1 2 3
73. If more keywords remain and the called function takes a keyword argument list, then the rest of the keyword arguments are passed as a keyword argument list. If the keyword argument list is not taken by the function, we get an error:
74.>>> def foo(a,b,*args,c,**kwargs):
75.... print(a,b,args,c,kwargs)
76....
77.>>> foo(1,2,3,4,5,c=6,d=7,e=8)
1 2 (3, 4, 5) 6 {'d': 7, 'e': 8}
Here is an example function that uses all of the preceding combinations:
>>> def foo(a,b,c=2,*pa,d,e=5,**ka):
... print(a,b,c,d,e,pa,ka)
...
>>> foo(1,2,d=4)
1 2 2 4 5 () {}
>>> foo(1,2,3,4,5,d=6,e=7,g=10,h=11)
1 2 3 6 7 (4, 5) {'h': 11, 'g': 10}
Decorating callables
Key 2: Changing the behavior of callables.
Decorators are callable objects, which replace the original callable objects with some other objects. In this case, as we are replacing a callable with another object, what we mostly want mostly is the replaced object to be callable.
Language provides syntax to do so easily, but first, let's take a look at how we can manually do this:
>>> def wrap(func):
... def newfunc(*args):
... print("newfunc",args)
... return newfunc
...
>>> def realfunc(*args):
... print("real func",args)
...
>>>
>>> realfunc = wrap(realfunc)
>>>
>>> realfunc(1,2,4)
('newfunc', (1, 2, 4))
With the decorator syntax, it becomes easy. Taking the definition of wrap and newfunc from the preceding code snippet, we get this:
>>> @wrap
... def realfunc(args):
... print("real func",args)
...
>>> realfunc(1,2,4)
('newfunc', (1, 2, 4))
To store some kind of state in the decorator function, say to make decorator more useful and applicable to wider application code base, we can use closures or class instances as decorators. In the second chapter, we saw that closures can be used to store state; let's look at how we can utilize them to store information in decorators. In this snippet, the deco function is the new function that will replace the add function. A prefix variable is available in the closure of this function. This variable can be injected at decorator creation time:
>>> def closure_deco(prefix):
... def deco(func):
... return lambda x:x+prefix
... return deco
...
>>> @closure_deco(2)
... def add(a):
... return a+1
...
>>> add(2)
4
>>> add(3)
5
>>> @closure_deco(3)
... def add(a):
... return a+1
...
>>> add(2)
5
>>> add(3)
6
We could have used a class to do the same thing as well. Here, we save state on an instance of class:
>>> class Deco:
... def __init__(self,addval):
... self.addval = addval
... def __call__(self, func):
... return lambda x:x+self.addval
...
>>> @Deco(2)
... def add(a):
... return a+1
...
>>> add(1)
3
>>> add(2)
4
>>> @Deco(3)
... def add(a):
... return a+1
...
>>> add(1)
4
>>> add(2)
5
As decorator works on any callable, it works similarly on methods and class definitions as well, but when doing so, we should take into consideration the different arguments that are implicitly passed for the method that is being decorated. Let's first take a simple method being decorated like this:
>>> class K:
... def do(*args):
... print("imethod",args)
...
>>> k = K()
>>> k.do(1,2,3)
('imethod', (<__main__.K instance at 0x7f12ea070bd8>, 1, 2, 3))
>>>
>>> # using a decorator on methods give similar results
...
>>> class K:
... @wrap
... def do(*args):
... print("imethod",args)
...
>>> k = K()
>>> k.do(1,2,3)
('newfunc', (<__main__.K instance at 0x7f12ea070b48>, 1, 2, 3))
As the function that is replaced becomes the method of the class itself, this works perfectly. This is not true for static and class methods. They employ descriptors to call methods, hence, their behavior breaks with decorators and the returned function behaves like a simple method. We can make this work by first checking whether the overridden function is a descriptor and if yes, then calling its __get__ method instead:
>>> class K:
... @wrap
... @staticmethod
... def do(*args):
... print("imethod",args)
... @wrap
... @classmethod
... def do2(*args):
... print("imethod",args)
...
>>> k = K()
>>> k.do(1,2,3)
('newfunc', (<__main__.K instance at 0x7f12ea070cb0>, 1, 2, 3))
>>> k.do2(1,2,3)
('newfunc', (<__main__.K instance at 0x7f12ea070cb0>, 1, 2, 3))
We can also make this work easily using static and class methods decorators on top of any other decorator. This makes the actual method that is found by the attribute look up as a descriptor and normal execution happens for staticmethod and classmethod.
This works fine, as follows:
>>> class K:
... @staticmethod
... @wrap
... def do(*args):
... print("imethod",args)
... @classmethod
... @wrap
... def do2(*args):
... print("imethod",args)
...
>>> k = K()
>>> k.do(1,2,3)
('newfunc', (1, 2, 3))
>>> k.do2(1,2,3)
('newfunc', (<class __main__.K at 0x7f12ea05e1f0>, 1, 2, 3))
We can use decorators for classes as class is just a type of callable. Hence, we can use decorators to alter the instance creation process so that when we call class, we get an instance. A class object will be passed to decorator and then decorator can replace it with another callable or class. Here, the cdeco decorator is passing a new class that replaced cls:
>>> def cdeco(cls):
... print("cdecorator working")
... class NCls:
... def do(*args):
... print("Ncls do",args)
... return NCls
...
>>> @cdeco
... class Cls:
... def do(*args):
... print("Cls do",args)
...
cdecorator working
>>> b = Cls()
>>> c = Cls()
>>> c.do(1,2,3)
('Ncls do', (<__main__.NCls instance at 0x7f12ea070cf8>, 1, 2, 3))
Normally, we use this to change the attributes, and add new attributes to the class definition. We can also use this to register the class to some registry, and so on. In the following code snippet, we check whether class has a do method. If we find one, we replace it with newfunc:
>>> def cdeco(cls):
... if hasattr(cls,'do'):
... cls.do = wrap(cls.do)
... return cls
...
>>> @cdeco
... class Cls:
... def do(*args):
... print("Cls do",args)
...
>>> c = Cls()
>>> c.do(1,2,3)
('newfunc', (<__main__.Cls instance at 0x7f12ea070cb0>, 1, 2, 3))
Utilities
Key 3: Easy iterations by comprehensions.
We have various syntax and utilities to iterate efficiently over iterators. Comprehensions work on iterator and provide results as another iterator. They are implemented in native C, and hence, they are faster than for loops.
We have list, dictionary, and set comprehensions, which produce list, dictionary, and set as result, respectively. Also, iterators avoid declaring extra variables that we need in a loop:
>>> ll = [ i+1 for i in range(10)]
>>> print(type(ll),ll)
<class 'list'> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> ld = { i:'val'+str(i) for i in range(10) }
>>> print(type(ld),ld)
<class 'dict'> {0: 'val0', 1: 'val1', 2: 'val2', 3: 'val3', 4: 'val4', 5: 'val5', 6: 'val6', 7: 'val7', 8: 'val8', 9: 'val9'}
>>> ls = {i for i in range(10)}
>>> print(type(ls),ls)
<class 'set'> {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
Generator expression creates generators, which can be used to produce generators for an iteration like this. To materialize a generator, we use it to create set, dict, or list:
>>> list(( i for i in range(10)))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> dict(( (i,'val'+str(i)) for i in range(10)))
{0: 'val0', 1: 'val1', 2: 'val2', 3: 'val3', 4: 'val4', 5: 'val5', 6: 'val6', 7: 'val7', 8: 'val8', 9: 'val9'}
>>> set(( i for i in range(10)))
{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
Generator objects do not compute all the values of the iterable at once but one by one when requested by a loop. This conserves memory, and we may not be interested in using the whole iterable. Generators are not silver bullets to be used everywhere. They do not always result in a performance increase. It depends on the consumer, and the cost of generating one sequence:
>>> def func(val):
... for i in (j for j in range(1000)):
... k = i + 5
...
>>> def func_iter(val):
... for i in [ j for j in range(1000)]:
... k = i + 5
...
>>> timeit.timeit(stmt="func(1000)", globals={'func':func_iter},number=10000)
0.6765081569974427
>>> timeit.timeit(stmt="func(1000)", globals={'func':func},number=10000)
0.838760247999744
Key 4: Some helpful utilities.
The itertools utility is a good module with many helpful functions for iterations. Some of my favorites are the following:
· itertools.chain(* iterable): This returns a single iterable from a list of iterables. First, all the elements of the first iterable are exhausted, and then of the second, and so on until all iterables are exhausted:
· >>> list(itertools.chain(range(3),range(2),range(4)))
· [0, 1, 2, 0, 1, 0, 1, 2, 3]
>>>
· itertools.cycle: This creates a copy of the iterator and continues to replay the results infinitely:
· >>> cc = cycle(range(4))
· >>> cc.__next__()
· 0
· >>> cc.__next__()
· 1
· >>> cc.__next__()
· 2
· >>> cc.__next__()
· 3
· >>> cc.__next__()
· 0
· >>> cc.__next__()
· 1
>>> cc.__next__()
· itertools.tee(iterable,number): This returns n independent iterables from a single iterable:
· >>> i,j = tee(range(10),2)
· >>> i
· <itertools._tee object at 0x7ff38e2b2ec8>
· >>> i.__next__()
· 0
· >>> i.__next__()
· 1
· >>> i.__next__()
· 2
· >>> j.__next__()
0
· functools.lru_cache: This decorator uses memorizing. It saves the results that are mapped to arguments. Hence, it is very useful to speed up functions that take a similar argument, and whose results are not dependent on time or state:
· In [7]: @lru_cache(maxsize=None)
· def fib(n):
· if n<2:
· return n
· return fib(n-1) + fib(n-2)
· ...:
·
· In [8]: %timeit fib(30)
· 10000000 loops, best of 3: 105 ns per loop
·
· In [9]:
· def fib(n):
· if n<2:
· return n
· return fib(n-1) + fib(n-2)
· ...:
·
· In [10]: %timeit fib(30)
1 loops, best of 3: 360 ms per loop
· functools.wraps: We have just seen how to create decorators, and how to wrap functions. The returned function from decorator retains its name and attributes, such as docstrings, which is not helpful for the users or fellow developers. We can use this decorator to match the returned function to the decorated function. The following snippet shows how it is used:
· >>> def deco(func):
· ... @wraps(func) # this will update wrapper to match func
· ... def wrapper(*args, **kwargs):
· ... """i am imposter"""
· ... print("wrapper")
· ... return func(*args, **kwargs)
· ... return wrapper
· ...
· >>> @deco
· ... def realfunc(*args,**kwargs):
· ... """i am real function """
· ... print("realfunc",args,kwargs)
· ...
· >>> realfunc(1,2)
· wrapper
· realfunc (1, 2) {}
· >>> print(realfunc.__name__, realfunc.__doc__)
realfunc i am real function
· Lambda functions: These functions are simple anonymous functions.Lambda functions cannot have statements or annotations. They are very useful in creating closures and callbacks in GUI programming:
· >>> def log(prefix):
· ... return lambda x:'%s : %s'%(prefix,x)
· ...
· >>> err = log("error")
· >>> warn = log("warn")
· >>>
· >>> print(err("an error occurred"))
· error : an error occurred
· >>> print(warn("some thing is not right"))
warn : some thing is not right
Sometimes, lambda functions make code easy to understand.
The following is a small program to create the diamond pattern using the iterations technique and the lambda function:
>>> import itertools
>>> af = lambda x:[i for i in itertools.chain(range(1,x+1),range(x-1,0,-1))]
>>> output = '\n'.join(['%s%s'%(' '*(5-i),' '.join([str(j) for j in af(i)])) for i in af(5)])
>>> print(output)
1
1 2 1
1 2 3 2 1
1 2 3 4 3 2 1
1 2 3 4 5 4 3 2 1
1 2 3 4 3 2 1
1 2 3 2 1
1 2 1
1
Summary
In this chapter, we covered how to define functions and pass arguments to them. Then, we discussed decorators in detail; decorators are very popular in frameworks. Toward the end, we collected various utilities that are available in Python, which makes coding a little easier for us.
In the next chapter, we will discuss algorithms and data structures.