Learning Python (2013)

Part VIII. Advanced Topics

Chapter 41. All Good Things

Welcome to the end of the book! Now that you’ve made it this far, I want to say a few words in closing about Python’s evolution before turning you loose on the software field. This topic is subjective by nature, of course, but vital to all Python users nonetheless.

You’ve now had a chance to see the entire language yourself—including some advanced features that may seem at odds with its scripting paradigm. Though many will understandably accept this as status quo, in an open source project it’s crucial that some ask the “why” questions too. Ultimately, the trajectory of the Python story—and its true conclusion—is at least in part up to you.

The Python Paradox

If you’ve read this book, or reasonable subsets of it, you should now be able to weigh Python’s tradeoffs fairly. As you’ve seen, Python is a powerful, expressive, and even fun programming language, which will serve as an enabling technology for wherever you choose to go next. At the same time, you’ve also seen that today’s Python is something of a paradox: it has expanded to incorporate tools that many consider both needlessly redundant and curiously advanced—and at a rate that appears to be only accelerating.

For my part, as one of Python’s earliest advocates, I’ve watched it morph over the years from simple to sophisticated tool, with a steadily shifting scope. By most measures, it seems to have grown at least as complex as other languages that drove many of us to Python in the first place. And just as in those other languages, this has inevitably fostered a growing culture in which obscurity is a badge of honor.

That’s as contrary to Python’s original goals as it could be. Run an import this in any Python interactive session to see what I mean—the creed I’ve quoted from repeatedly in this book in contexts where it was clearly violated. On many levels, its core ideals of explicitness, simplicity, and lack of redundancy have been either naively forgotten or carelessly abandoned.

The end result is a language and community that could in part be described today in some of the same terms I used in the Perl sidebar of Chapter 1. While Python still has much to offer, this trend threatens to negate much of its perceived advantage, as the next section explains.

On “Optional” Language Features

I included a quote near the start of the prior chapter about metaclasses not being of interest to 99% of Python programmers, to underscore their perceived obscurity. That statement is not quite accurate, though, and not just numerically so. The quote’s author is a noted Python contributor and friend from the early days of Python, and I don’t mean to pick on anyone unfairly. Moreover, I’ve often made such statements about language feature obscurity myself—in the various editions of this very book, in fact.

The problem, though, is that such statements really apply only to people who work alone and only ever use code that they’ve written themselves. As soon as an “optional” advanced language feature is used by anyone in an organization, it is no longer optional—it is effectively imposed oneveryone in the organization. The same holds true for externally developed software you use in your systems—if the software’s author uses an advanced or extraneous language feature, it’s no longer entirely optional for you, because you have to understand the feature to reuse or change the code.

This observation applies to all the advanced topics covered in this book, including those listed as “magic” hooks near the beginning of the prior chapter, and many others:

Generators, decorators, slots, properties, descriptors, metaclasses, context managers, closures, super, namespace packages, Unicode, function annotations, relative imports, keyword-only arguments, class and static methods, and even obscure applications of comprehensions and operator overloading

If any person or program you need to work with uses such tools, they automatically become part of your required knowledge base too.

To see just how daunting this can be, one need only consider Chapter 40’s new-style inheritance procedure—a horrifically convoluted model that can make descriptors and metaclasses prerequisite to understanding even basic name resolution. Chapter 32’s super similarly ups the intellectual ante—imposing an obscenely implicit and artificial MRO algorithm on readers of any code that uses this tool.

The net effect of such over-engineering is to either escalate learning requirements radically, or foster a user base that only partially understands the tools they employ. This is obviously less than ideal for those hoping to use Python in simpler ways, and contradictory to the scripting motif.

Against Disquieting Improvements

This observation also applies to the many redundant features we’ve seen, such as Chapter 7’s str.format method and Chapter 34’s with statement—tools borrowed from other languages, and overlapping with others long present in Python. When programmers use multiple ways to achieve the same goal, all become required knowledge.

Let’s be honest: Python has grown rife with redundancy in recent years. As I suggested in the preface—and as you’ve now seen first-hand—today’s Python world comes replete with all the functional duplications and expansions chronicled in Table 41-1, among others we’ve seen in this book.

Table 41-1. A sampling of redundancy and feature explosion in Python

If you care about Python, you should take a moment to browse this table. It reflects a virtual explosion in functionality and toolbox size—59 concepts that are all fair game for newcomers. Most of its categories began with just one original member in Python; many were expanded in part to imitate other languages; and only the last few can be simplified by pretending that the latest Python is the only Python that matters to its programmers.

I’ve stressed avoiding unwarranted complexity in this book, but in practice, both advanced and new tools tend to encourage their own adoption—often for no better reason than a programmer’s personal desire to demonstrate prowess. The net result is that much Python code today is littered with these complex and extraneous tools. That is, nothing is truly “optional” if nothing is truly optional.

Complexity Versus Power

This is why some Python old-timers (myself included) sometimes worry that Python seems to have grown larger and more complex over time. New features added by veterans, converts, and even amateurs may have raised the intellectual bar for newcomers. Although Python’s core ideas, like dynamic typing and built-in types, have remained essentially the same, its advanced additions can become required reading for any Python programmer. I chose to cover these topics here for this reason, despite their omission in early editions. It’s not possible to skip the advanced stuff if it’s in code you have to understand.

On the other hand, as mentioned in Chapter 1, to most observers Python is still noticeably simpler than most of its contemporaries, and perhaps only as complex as its many roles require. Though it’s acquired many of the same tools as Java, C#, and C++, they tend to be lighter weight in the context of a dynamically typed scripting language. For all its growth over the years, Python is still relatively easy to learn and use when compared to the alternatives, and new learners can often pick up advanced topics as needed.

And frankly, application programmers tend to spend most of their time dealing with libraries and extensions, not advanced and sometimes-arcane language features. For instance, the book Programming Python—a follow-up to this one—deals mostly with the marriage of Python to application libraries for tasks such as GUIs, databases, and the Web, not with esoteric language tools (though Unicode still forces itself onto many stages, and the odd generator expression and yield crop up along the way).

Moreover, the flipside of this growth is that Python has become more powerful. When used well, tools like decorators and metaclasses are not only arguably “cool,” but allow creative programmers to build more flexible and useful APIs for other programmers to use. As we’ve seen, they can also provide good solutions to problems of encapsulation and maintenance.

Simplicity Versus Elitism

Whether this justifies the potential expansion of required Python knowledge is up to you to decide. For better or worse, a person’s skill level often decides this issue by default—more advanced programmers like more advanced tools and tend to forget about their impact on other camps. Fortunately, though, this isn’t an absolute; good programmers also understand that simplicity is good engineering, and advanced tools should be used only when warranted. This is true in any programming language, but especially in one like Python that is frequently exposed to new or novice programmers as an extension tool.

And if you’re still not buying this, keep in mind that many people using Python are not comfortable with even basic OOP. Trust me on this; I’ve met thousands of them. Although Python was never a trivial subject, the reports from the software trenches are very clear on this point: unwarranted added complexity is never a welcome feature, especially when it is driven by the personal preferences of an unrepresentative few. Whether intended or not, this is often understandably perceived as elitism—a mindset that is both unproductive and rude, and has no place in a tool as widely used as Python.

This is also a social issue, of course, and pertains as much to individual programmers as to language designers. In the “real world” where open source software is measured, though, Python-based systems that require their users to master the nuances of metaclasses, descriptors, and the like should probably scale their market expectations accordingly. Hopefully, if this book has done its job, you’ll find the importance of simplicity in programming to be one of its most important and lasting takeaways.

Closing Thoughts

So there you have it—some observations from someone who has been using, teaching, and advocating Python for two decades, and still wishes nothing but the best for its future. None of these concerns are entirely new, of course. Indeed, the growth of this very book over the years seems testament to the effect of Python’s own growth—if not an ironic eulogy to its original conception as a tool that would simplify programming and be accessible to both experts and nonspecialists alike. Judging by language heft alone, that dream seems to have been either neglected or abandoned entirely.

That said, Python’s present rise in popularity seems to show no signs of abating—a powerful counterargument to complexity concerns. Today’s Python world may be understandably less concerned with its original and perhaps idealistic goals than with applying its present form in their work. Python gets many a job done in the practical world of complex programming requirements, and this is still ample cause to recommend it for many tasks. Original goals aside, mass appeal does qualify as one form of success, though one whose significance will have to await the verdict of time.

If you’re interested in musing further over Python’s evolution and learning curve, I wrote a more in-depth article in 2012 on such things: Answer Me These Questions Three..., available online at http://learning-python.com/pyquestions3.html. These are important pragmatic questions that are crucial to Python’s future, and deserve more attention than I’ve given here. But these are highly subjective issues; this is not a philosophy text; and this book has already exceeded its page-count targets.

More importantly, in an open source project like Python the answers to such questions must be formed anew by each wave of newcomers. I hope the wave you ride in will have as much common sense as fun while plotting Python’s future.

Where to Go From Here

And that’s a wrap, folks. You’ve officially reached the end of this book. Now that you know Python inside and out, your next step, should you choose to take it, is to explore the libraries, techniques, and tools available in the application domains in which you work.

Because Python is so widely used, you’ll find ample resources for using it in almost any application you can think of—from GUIs, the Web, and databases to numeric programming, robotics, and system administration. See Chapter 1 and your favorite web browser for pointers to popular tools and topics.

This is where Python starts to become truly fun, but this is also where this book’s story ends, and others’ begin. For pointers on where to turn after this book, see the recommended follow-up texts mentioned in the preface. I hope to see you in an applications programming domain soon.

Good luck with your journey. And of course, “Always look on the bright side of Life!”

Encore: Print Your Own Completion Certificate!

And one last thing: in lieu of exercises for this part of the book, I’m going to post a bonus script here for you to study and run on your own. I can’t provide completion certificates for readers of this book (and the certificates would be worthless if I could), but I can include an arguably cheesy Python script that does—the following file, certificate.py, is a Python 2.X and 3.X script that creates a simple book completion certificate in both text and HTML file forms, and pops them up in a web browser on your machine by default.

#!/usr/bin/python

"""

File certificate.py: a Python 2.X and 3.X script.

Generate a bare-bones class completion certificate: printed,

and saved in text and html files displayed in a web browser.

"""

from __future__ import print_function # 2.X compatibility

import time, sys, webbrowser

if sys.version_info[0] == 2: # 2.X compatibility

input = raw_input

import cgi

htmlescape = cgi.escape

else:

import html

htmlescape = html.escape

maxline = 60 # For seperator lines

browser = True # Display in a browser

saveto = 'Certificate.txt' # Output filenames

template = """

%s

===> Official Certificate <===

Date: %s

This certifies that:

\t%s

has survived the massive tome:

\t%s

and is now entitled to all privileges thereof, including

the right to proceed on to learning how to develop Web

sites, desktop GUIs, scientific models, and assorted apps,

with the possible assistance of follow-up applications

books such as Programming Python (shameless plug intended).

--Mark Lutz, Instructor

(Note: certificate void where obtained by skipping ahead.)

%s

"""

# Interact, setup

for c in 'Congratulations!'.upper():

print(c, end=' ')

sys.stdout.flush() # Else some shells wait for \n

time.sleep(0.25)

print()

date = time.asctime()

name = input('Enter your name: ').strip() or 'An unknown reader'

sept = '*' * maxline

book = 'Learning Python 5th Edition'

# Make text file version

file = open(saveto, 'w')

text = template % (sept, date, name, book, sept)

print(text, file=file)

file.close()

# Make html file version

htmlto = saveto.replace('.txt', '.html')

file = open(htmlto, 'w')

tags = text.replace(sept, '<hr>') # Insert a few tags

tags = tags.replace('===>', '<h1 align=center>')

tags = tags.replace('<===', '</h1>')

tags = tags.split('\n') # Line-by-line mods

tags = ['<p>' if line == ''

else line for line in tags]

tags = ['<i>%s</i>' % htmlescape(line) if line[:1] == '\t'

else line for line in tags]

tags = '\n'.join(tags)

link = '<i><a href="http://www.rmi.net/~lutz">Book support site</a></i>\n'

foot = '<table>\n<td><img src="ora-lp.jpg" hspace=5>\n<td>%s</table>\n' % link

tags = '<html><body bgcolor=beige>' + tags + foot + '</body></html>'

print(tags, file=file)

file.close()

# Display results

print('[File: %s]' % saveto, end='')

print('\n' * 2, open(saveto).read())

if browser:

webbrowser.open(saveto, new=True)

webbrowser.open(htmlto, new=False)

if sys.platform.startswith('win'):

input('[Press Enter]') # Keep window open if clicked on Windows

Run this script on your own, and study its code for a summary of some of the ideas we’ve covered in this book. Fetch it from this book’s website described in the preface if you wish. You won’t find any descriptors, decorators, metaclasses, or super calls in this code, but it’s typical Python nonetheless.

When run, it generates the web page captured in the fully gratuitous Figure 41-1. This could be much more grandiose, of course; see the Web for pointers to Python support for PDFs and other document tools such as Sphinx surveyed in Chapter 15. But hey: if you’ve made it to the end of this book, you deserve another joke or two...

Figure 41-1. Web page created and opened by certificate.py.