Programming Google App Engine with Python (2015)

Chapter 2. Creating an Application

The App Engine development model is as simple as it gets:

1. Create the application.

2. Test the application on your own computer by using the web server software included with the App Engine development kit.

3. Deploy the finished application to App Engine.

In this chapter, we’ll walk through the process of creating a new application, testing it with the development server, registering a new project ID with Google Cloud Platform, setting up a domain name, and uploading the app to App Engine. We’ll look at some of the features of the Python software development kit (SDK) and the Cloud Console. We’ll also discuss the workflow for developing and deploying an app.

We will take this opportunity to demonstrate a common pattern in web applications: managing user preferences data. This pattern uses several App Engine services and features.

Setting Up the Cloud SDK

To develop an App Engine app in Python, you need several unsurprising things on your local computer:

§ Python, version 2.7.x

§ Your favorite programming text editor or IDE for Python

§ The Google Cloud SDK

The Google Cloud SDK is a collection of tools and libraries for developing, testing, and deploying software for the Cloud Platform, including App Engine. The centerpiece of the suite is the gcloud command, a multifunction tool which you use to install and update components, perform deployment and maintenance tasks, and otherwise interact with Cloud Platform as an administrator. The App Engine component provides tools for running your app locally on your computer for testing, and for deploying the app. The Cloud SDK works on any platform that can run Python, including Windows, Mac OS X, and Linux.

The development server and app maintenance features are all available through command-line tools. You can use these tools from a command prompt or otherwise integrate them into your development environment as you see fit. For example, it’s common to automate the deployment process with scripts that run the deployment tool and manage application versions and testing.

On Windows and Mac OS X, you can install a “launcher” application that makes it especially easy to create, edit, test, and upload an app, using a simple graphical interface. Paired with a good programming text editor (such as Notepad++ for Windows, or Sublime Text for Mac OS X), the launcher provides a fast and intuitive Python development experience.

Some professional Python IDEs, such as JetBrains PyCharm, have built-in support for creating App Engine applications, and provide another alternative to the command line for invoking the App Engine SDK tools.

In the next few sections, we’ll install the Cloud SDK and components related to Python app development.

Installing Python

The App Engine SDK for the Python runtime environment is compatible with any computer that runs Python 2.7. If you are using Mac OS X or Linux, or if you have used Python previously, you may already have Python on your system. You can test whether Python is installed on your system and check which version is installed by running the following command at a command prompt (in Windows, Command Prompt; in Mac OS X, Terminal):

python -V

(That’s a capital “V.”) If Python is installed, it prints its version number, like so:

Python 2.7.5

You can download and install Python 2.7 for your platform from the Python website.

Be sure to get Python version 2.7 (such as 2.7.8) from the Downloads section of the site. As of this writing, the latest major version of Python is 3.4, and the latest 2.x-compatible release is 2.7.1

TIP

App Engine Python does not yet have a dedicated Python 3 runtime environment. Python 3 includes several new language and library features that are not backward compatible with earlier versions. As we’ll see in a minute, you specify the runtime environment for your application using configuration, so your app always runs with the same major version of Python, even as new major versions become available.

Installing the Cloud SDK

There are several easy ways to install the Cloud SDK, depending on your operating system and personal preferences.

For Windows, visit the Cloud SDK website, select the Windows installation instructions, then click the “Download” button:

§ https://cloud.google.com/sdk/

Run GoogleCloudSDKInstaller.exe and follow the instructions. By default, the installer puts the SDK in C:\Program Files\Google\Cloud SDK, and offers to create appropriate shortcuts on the desktop. One of these shortcuts, the Google Cloud SDK Shell, opens a command prompt with its command lookup path amended to include the Cloud SDK commands.

For Mac OS X or Linux, open a command prompt and run the following command:

curl https://sdk.cloud.google.com | bash

This downloads the Cloud SDK archive, unpacks it, and runs an interactive installation routine. If you’d rather invoke these steps manually, you can download the archive from the Cloud SDK website. The installation routine is this shell script: ./google-cloud-sdk/install.sh.

Either way you do it, the result is a google-cloud-sdk directory, which you can put in a convenient place. The installation script attempts to add the Cloud SDK commands to your command prompt’s search path by amending your environment. If you’re using a shell other than bash, you may need to finish this step manually. You can always invoke the Cloud SDK commands using their full path in the google-cloud-sdk directory.

Close and reopen your command prompt to pick up the changes to the command search path.

On all platforms, confirm that the SDK was installed successfully by running the gcloud command like so:

gcloud -h

The command will print some help text describing its various features. If instead the command prompt reports that the command is not found, double-check that the SDK is on the command path. In Windows, this is set up by the “SDK Shell,” or the cloud_env.bat script in the main SDK folder. In Mac OS X and Linux, make sure you have restarted your command prompt to pick up the environment changes to your shell startup scripts, and that the google-cloud-sdk/bin directory is on your search path.

The Cloud SDK politely keeps all of its stuff in the google-cloud-sdk directory. To uninstall it, simply delete the directory, and undo the command search path modifications.

Authenticating with the Cloud SDK

Before doing anything else, run this command:

gcloud auth login

This opens a browser window for the Google authentication sequence. Sign in using your Google account (or register for a free account if you don’t have one yet), then authorize the Google Cloud SDK to access Cloud services on your behalf.

The gcloud command remembers the authentication credentials it receives and uses them with subsequent commands that you run from your computer. To revoke these credentials (to “sign out” of gcloud), run this command:

gcloud auth revoke

You can use gcloud auth login to sign in with multiple accounts. Credentials for all accounts are remembered until revoked, and you can switch between them without signing in again. To add an account, simply gcloud auth login again with the new account. To list all accounts with stored credentials and see which account is currently active:

gcloud auth list

To switch the active account to another one with stored credentials:

gcloud config set account your.email@gmail.com

Installing the App Engine SDK

The Cloud SDK installation process already asked you if you wanted to install an App Engine SDK. If you answered in the affirmative and requested the App Engine SDK for “Python and PHP,” then you’re all set. (The Python SDK and the PHP SDK are the same component.)

If you have not yet installed the App Engine SDK for Python, you can do so with this command:

gcloud components update gae-python

You can test that the App Engine SDK is installed by running the following command:

dev_appserver.py --help

You will use the dev_appserver.py command to start a local development server running a test version of your app. With the --help argument, the command simply prints help text and exits.

Windows users, if when you run this command, a dialog box opens with the message “Windows cannot open this file… To open this file, Windows needs to know what program created it,” you must tell Windows to use Python to open the file. In the dialog box, choose “Select the program from a list,” and click OK. Click Browse, then locate your Python installation (such as C:\Python27). Select python from this folder, then click Open. Select “Always use the selected program to open this kind of file.” Click OK. A window will open and attempt to run the command, then immediately close. You can now run the command from the Command Prompt.

You use the gcloud components command to install, update, and manage the various components of the Cloud SDK. To see a list of all available components and which ones are installed, run this command:

gcloud components list

To update all components that need updating:

gcloud components update

Developing the Application

It’s time to write our first App Engine application!

So what is an App Engine app? An App Engine app is software that responds to web requests. It does so by calling request handlers, routines that accept request parameters and return responses. App Engine determines which request handler to use for a given request from the request’s URL, using a configuration file included with the app that maps URLs to handlers.

An app can also include static files, such as images, CSS stylesheets, and browser JavaScript. App Engine serves these files directly to clients in response to requests for corresponding URLs without invoking any code. The app’s configuration specifies which of its files are static, and which URLs to use for those files.

The application configuration includes metadata about the app, such as its project ID and version number. When you deploy the app to App Engine, all of the app’s files, including the code, configuration files, and static files, are uploaded and associated with the project ID and version number mentioned in the configuration. An app can also have configuration files specific to the services, such as for datastore indexes, task queues, and scheduled tasks. These files are associated with the app in general, not a specific version of the app.

In the next few sections, we create the files needed for a simple application, and look at how to use the tools and libraries included with the SDK.

The User Preferences Pattern

The application we create in this section is a simple clock. When a user visits the site, the app displays the current time of day according to the server’s system clock. By default, the app shows the current time in the Coordinated Universal Time (UTC) time zone. The user can customize the time zone by signing in using Google Accounts and setting a preference.

This app demonstrates three App Engine features:

§ The datastore, primary storage for data that is persistent, reliable, and scalable

§ The memory cache (or memcache), secondary storage that is faster than the datastore, but is not necessarily persistent in the long term

§ Google Accounts, the ability to use Google’s user account system for authenticating and identifying users

Google Accounts works similarly to most user account systems. If the user is not signed in to the clock application, she sees a generic view with default settings (the UTC time zone) and a link to sign in or create a new account. If the user chooses to sign in or register, the application directs her to a sign-in form managed by Google Accounts. Signing in or creating an account redirects the user back to the application.

Of course, you can implement your own account mechanism instead of using Google Accounts. Using Google Accounts has advantages and disadvantages—the chief advantage being that you don’t have to implement your own account mechanism. If a user of your app already has a Google account, the user can sign in with that account without creating a new account for your app.

If the user accesses the application while signed in, the app loads the user’s preferences data and uses it to render the page. The app retrieves the preferences data in two steps. First, it attempts to get the data from the fast secondary storage, the memory cache. If the data is not present in the memory cache, the app attempts to retrieve it from the primary storage (the datastore), and if successful, it puts it into the memory cache to be found by future requests.

This means that for most requests, the application can get the user’s preferences from memcache without accessing the datastore. While reading from the datastore is reasonably fast, reading from the memcache is much faster and avoids the cost of a datastore call. The difference is substantial when the same data must be accessed every time the user visits a page.

Our clock application has two request handlers. One handler displays the current time of day, along with links for signing in and out. It also displays a web form for adjusting the time zone when the user is signed in. The second request handler processes the time zone form when it is submitted. When the user submits the preferences form, the app saves the changes and redirects the browser back to the main page.

The application gets the current time from the application server’s system clock. It’s worth noting that App Engine makes no guarantees that the system clocks of all its web servers are synchronized. Because two requests for this app may be handled by different servers, different requests may see different clocks. The server clock is not consistent enough as a source of time data for a real-world application, but it’s good enough for this example.

A Simple App

The simplest Python application for App Engine is a single directory with two files: a configuration file named app.yaml, and a file of Python code for a request handler. The directory containing the app.yaml file is the application root directory. You’ll refer to this directory often when using the tools.

Create a directory named clock to contain the project. Using your favorite text editor, create a file inside this directory named app.yaml similar to Example 2-1.

Example 2-1. The app.yaml configuration file for a simple application, using the Python 2.7 runtime environment

application: clock

version: 1

runtime: python27

api_version: 1

threadsafe: yes

handlers:

- url: .*

script: main.application

libraries:

- name: webapp2

version: "2.5.2"

This configuration file is in a format called YAML, an open format for configuration files and network messages. You don’t need to know much about the format beyond what you see here.

In this example, the configuration file tells App Engine that this is version 1 of an application called clock, which uses version 1 (api_version) of the Python 2.7 runtime environment. Every request for this application (every URL that matches the regular expression .*, which is all of them) is to be handled by an application object defined in the application variable of a Python module named main.

Next, create a file named main.py similar to Example 2-2, in the same directory as app.yaml.

Example 2-2. A simple Python web application, using the webapp2 framework

import datetime

import webapp2

class MainPage(webapp2.RequestHandler):

def get(self):

message = '<p>The time is: %s</p>' % datetime.datetime.now()

self.response.out.write(message)

application = webapp2.WSGIApplication([('/', MainPage)],

debug=True)

This simple Python web application uses a web application framework called “webapp2,” which is included with App Engine. This framework conforms to a common standard for Python web application frameworks known as the Web Server Gateway Interface (WSGI). You don’t need to know much about WSGI, except that it’s a Python standard, there are many useful frameworks to choose from, and it’s easy to port a WSGI application to other application hosting environments using various adapters (such as a WSGI-to-CGI adapter). webapp2 is a simple example of a WSGI framework. Django, a popular open source web app framework for Python that’s also included with App Engine, is another.

We’ll walk through this example in a moment, but first, let’s get it running. From a command prompt, run the dev_appserver.py command, specifying the path to the project directory (clock) as an argument:

dev_appserver.py clock

TIP

If your current working directory is the clock directory you just created, you can run the command using a dot (.) as the path to the project:

dev_appserver.py .

Test your application by visiting the server’s URL in a web browser:

§ http://localhost:8080/

The browser displays a page similar to Figure 2-1.

Figure 2-1. The first version of the clock application viewed in a browser

You can leave the web server running while you develop your application. The web server notices when you make changes to your files, and reloads them automatically as needed.

The server starts up and prints several messages to the console. You can safely ignore warnings that say things like “Could not initialize images API.” These are expected if you have followed the installation steps so far. The last message should look something like this:

INFO ... Starting module "default" running at: http://localhost:8080

INFO ... Starting admin server at: http://localhost:8000

This message indicates the server started successfully. If you do not see this message, check the other messages for hints, and double-check that the syntax of your app.yaml file is correct.

Introducing the webapp Framework

App Engine’s Python 2.7 runtime environment uses WSGI as the interface between your application and the server instance running the application. Typically, you would not write the code that implements this interface. Instead, you would use a framework, a suite of libraries and tools that form an easy way to think about building web applications and perform common web tasks.

There are dozens of web frameworks written in Python, and several are mature, well documented, and have active developer communities. Django, Flask, web2py, and Pyramid are examples of well-established Python web frameworks that work well with App Engine. We’ll discuss how to use Django with App Engine in Chapter 18.

The webapp2 framework is intended to be small and easy to use. It doesn’t have the features of more established frameworks, but it’s good enough for small projects. For simplicity, most of the Python examples in this book use the webapp2 framework. We’ll introduce some of its features here.

Let’s take a closer look at our simple web application, line by line:

import datetime

import webapp2

This loads the libraries we intend to use in the main module. We use datetime to get the system time for our clock. The webapp2 framework is in the webapp2 module:

class MainPage(webapp2.RequestHandler):

def get(self):

message = '<p>The time is: %s</p>' % datetime.datetime.now()

self.response.out.write(message)

webapp2 applications consist of one or more request handlers, units of code mapped to URLs or URL patterns that are executed when a client (or other process) requests a URL. As we saw earlier, the first URL mapping takes place in app.yaml, which associates the request URL with its WSGI application object in a Python module. The webapp2 application maps this to a RequestHandler class.

To produce the response, webapp2 instantiates the class and then calls a method of the class that corresponds to the HTTP method of the request. When you type a URL into your browser’s address bar, the browser uses the HTTP GET method with the request, so webapp2 calls the get()method of the request handler. Similarly, when you submit a web form, the browser uses the HTTP POST method, which would attempt to call a post() method.

The code can access the request data and produce the response data, using attributes of the instance. In this case, we prepared a response string (message), then used the output stream of the response attribute to write the message. You can also use the response attribute to set response headers, such as to change the content type. (Here, we leave the content type at its default of text/html.)

application = webapp2.WSGIApplication([('/', MainPage)],

debug=True)

The application module global variable contains the object that represents the WSGI application. This value is created when the main module is imported for the first time, and stays in memory for the lifetime of the application instance. (App Engine creates and destroys application instances as needed to serve your app’s traffic. More on that later.) App Engine knows which module and variable to use based on the mapping in the app.yaml file.

The application object is an instance of the WSGIApplication class provided by the webapp2 module. The constructor is called with two values. The first is a list of URL pattern and RequestHandler class pairs. When the application is called to handle a request, the URL is tested against each pattern in the order it appears in the list. The first to match wins. The URL pattern is a regular expression.

In this case, our application simply maps the root URL path (/) to MainPage. If the application is asked to handle any other URL path (any path that doesn’t match), webapp2 serves an HTTP 404 error page. Notice that the app.yaml file maps all URL paths to this application, effectively putting webapp2 in charge of serving 404 errors. (If a URL does not match any pattern in app.yaml, App Engine serves its own 404 error.)

The WSGIApplication constructor is also given a debug=True parameter. This tells webapp2 to print detailed error messages to the browser when things go wrong. webapp2 knows to only use this in the development server, and to disable this feature when it is running on App Engine, so you can just leave it turned on.

A single WSGIApplication instance can handle multiple URLs, routing the request to different RequestHandler classes based on the URL pattern. But we’ve already seen that the app.yaml file maps URL patterns to handler scripts. So which URL patterns should appear in app.yaml, and which should appear in the WSGIApplication? Many web frameworks include their own URL dispatcher logic, and it’s common to route all dynamic URLs to the framework’s dispatcher in app.yaml. With webapp2, the answer mostly depends on how you’d like to organize your code. For the clock application, we will create a second request handler as a separate script to take advantage of a feature of app.yaml for user authentication, but we could also put this logic in main.py and route the URL with the WSGIApplication object.

Templates, Users, and Google Accounts

So far, our clock shows the same display for every user. To allow users to customize the display and save their preferences for future sessions, we need a way to identify the user making a request. An easy way to do this is with Google Accounts, aka the Users service.

Before we make the user interface of our app more elaborate, let’s introduce a templating system to manage our HTML. Mixing markup and code for the browser in your server-side code gets messy fast. It’s nearly always better to use a library that can represent the user interface code separately from your app code, using templates. The app code calls the templating system to fill in the blanks with dynamic data and render the result.

For this example, we’ll use the Jinja2 templating system. Jinja2 is an open source templating system written in Python, based on the templating system included with the Django web application framework. App Engine will provide this library to your app if you request it in the app.yaml file.

Edit app.yaml, and add these lines to the libraries: section near the bottom:

libraries:

# ...

- name: jinja2

version: "2.6"

- name: markupsafe

version: "0.15"

The libraries: section of app.yaml tells App Engine to add certain Python libraries to the runtime environment when it runs the app. We used this feature to request version 2.5.2 of webapp2. These new lines add Jinja2 (and its related library markupsafe) to the environment. App Engine supports a short but useful list of third-party libraries in this way, so you do not have to add the library to your application code.

To run your app locally, you must have these libraries installed. For historical reasons, webapp2 is provided for us by the App Engine SDK, but Jinja2 is not, so we must install it in the local development environment. App Engine does not include every library in the SDK because it would have to include every supported version of every library, and that could get large.

There are several ways to install Python packages. Your Python installation may already have the easy_install command, and if it doesn’t, you can get it by downloading and installing setuptools. A newer more featureful alternative is pip. Both easy_install and pip get packages from the Python Package Index (PyPi) and install them in your Python installation’s site-packages collection.

To install Jinja2 (and its helper package markupsafe) using pip:

pip install jinja2 markupsafe

In Mac OS X or Linux, if you’re installing these packages in the system’s main Python installation, you must run this command via sudo and enter an administrative password when prompted:

sudo pip install jinja2 markupsafe

Windows users, there is a pip.exe command you can add to your command path (in C:\Python27\Scripts), but you can also just invoke it with the python command:

python -m pip install jinja2 markupsafe

Let’s add something to our app’s home page that indicates whether the user is signed in, and provides links for signing in and signing out of the application. Edit main.py to resemble Example 2-3.

Example 2-3. A version of main.py that invites the user to sign in with Google Accounts, using a Jinja2 template

import datetime

import jinja2

import os

import webapp2

from google.appengine.api import users

template_env = jinja2.Environment(

loader=jinja2.FileSystemLoader(os.getcwd()))

class MainPage(webapp2.RequestHandler):

def get(self):

current_time = datetime.datetime.now()

user = users.get_current_user()

login_url = users.create_login_url(self.request.path)

logout_url = users.create_logout_url(self.request.path)

template = template_env.get_template('home.html')

context = {

'current_time': current_time,

'user': user,

'login_url': login_url,

'logout_url': logout_url,

}

self.response.out.write(template.render(context))

application = webapp2.WSGIApplication([('/', MainPage)],

debug=True)

Next, create a new file in the same directory named home.html, and edit it to resemble Example 2-4. This is the Jinja2 template.

Example 2-4. The HTML template for the home page, using the Jinja2 template system

<html>

<head>

<title>The Time Is...</title>

</head>

<body>

{% if user %}

<p>

Welcome, {{ user.email() }}!

You can <a href="{{ logout_url }}">sign out</a>.

</p>

{% else %}

<p>

Welcome!

<a href="{{ login_url }}">Sign in or register</a> to customize.

</p>

{% endif %}

<p>The time is: {{ current_time }}</p>

</body>

</html>

Reload the page in your browser. The new page resembles Figure 2-2.

We’ve added a few new things to main.py:

import jinja2

import os

# ...

from google.appengine.api import users

You import the Jinja2 library the same way you would with a typical installation on your computer. The libraries: section of app.yaml puts Jinja2 on the library search path when running on App Engine.

Figure 2-2. The clock app with a link to Google Accounts when the user is not signed in

We also import the os module to use in the next part, as well as the API for the Users service:

template_env = jinja2.Environment(

loader=jinja2.FileSystemLoader(os.getcwd()))

One way to configure Jinja2 is with an Environment object. This object maintains aspects of the template system that are common across your app. In this case, we use the Environment to declare that our template files are loaded from the filesystem, using the FileSystemLoader.

We store the Jinja2 Environment object in a module global variable because we only need to create this object once in the lifetime of the application instance. As with the WSGIApplication object, the constructor is called when the module is imported, and the object stays resident in memory.

TIP

Remember that we’ve turned on concurrent requests using the threadsafe: true line in app.yaml. This tells App Engine to use one instance to process multiple requests simultaneously. These requests will share global module variables, and may interleave instructions. This is fine for most common read-only uses of global variables, such as configuration data, compiled regular expressions, or the Jinja2 Environment object.

The os.getcwd() value passed to the FileSystemLoader constructor tells it to find templates in the current working directory. When the request handler is called, the current working directory is the application root directory. If you move your templates into a subdirectory (and that’s probably a good idea), this value needs to be modified accordingly:

current_time = datetime.datetime.now()

user = users.get_current_user()

login_url = users.create_login_url(self.request.path)

logout_url = users.create_logout_url(self.request.path)

The request handler code calls the Users service API by using functions in the users module from the google.appengine.api package. users.get_current_user() returns an object of class users.User that represents the user making the request if the user is signed in, or None(Python’s null value) if the user is not signed in. You can use this value to access the user’s email address, which our application does from within the template.

To allow a user to sign in or sign out, you direct the user’s browser to the Google Accounts system “login” or “logout” URLs. The app gets these URLs using the users.create_login_url() and users.create_logout_url() functions, respectively. These functions take a URL path for your application as an argument. Once the user has signed in or signed out successfully, Google Accounts redirects the user back to your app using that URL path. For this app, we direct the user to sign in or sign out by presenting her with links to click. (In other situations, redirecting the user might be more appropriate.)

template = template_env.get_template('home.html')

context = {

'current_time': current_time,

'user': user,

'login_url': login_url,

'logout_url': logout_url

}

self.response.out.write(template.render(context))

Here, we load the home.html template, set the dynamic data in the template’s “context,” render the template with the context values into the text of the page, and finally write it to the response.

Within the template, we use a conditional section to display a different welcome message depending on whether the user is signed in or not. {% if user %} is true if the context value we set to 'user' is considered true in Python, which it would be if users.get_current_user()returned a User object. The {% else %} and {% endif %} directives delimit the sections of the template to render, based on the condition. {{ user.email() }} calls the email() method of the object, and interpolates its return value into the template as a string. Similarly, {{ logout_url }}, {{ login_url }}, and {{ current_time }} interpolate the generated URLs we set in the context.

TIP

For more information about Jinja2 template syntax and features, see the Jinja2 website.

If you click the “Sign in or register” link with the app running in the development server, the link goes to the development server’s simulated version of the Google Accounts sign-in screen, as shown in Figure 2-3. At this screen, you can enter any email address, and the development server will proceed as if you are signed in with an account that has that address.

If this app were running on App Engine, the login and logout URLs would go to the actual Google Accounts locations. Once signed in or out, Google Accounts redirects back to the given URL path for the live application.

Click “Sign in or register,” then click the Login button on the simulated Google Accounts screen, using the default test email address (test@example.com). The clock app now looks like Figure 2-4. To sign out again, click the “sign out” link.

Figure 2-3. The development server’s simulated Google Accounts sign-in screen

Using Python Virtual Environments

When we installed the Jinja2 library earlier, we added it directly to the collection of libraries available to Python on our local computer. This is convenient for writing software that’s meant to be run on your local computer, but not so convenient when writing software that will be run elsewhere. It can be tricky to keep track of all of the packages your software depends on, and ensure that those packages are installed—or can be installed—in the environment where the software is to run. And if two projects depend on different versions of a package, your computer’s site packages are no help, because they can only contain one at a time.

The App Engine development server is smart about isolating your app under test from packages you have installed locally that won’t be available in the live runtime environment. If we didn’t specify jinja2 in the libraries: region of app.yaml, the import jinja2 statement would have failed in the development server even if we had the jinja2 package installed. Similarly, importing a module that’s installed locally but would not otherwise be available on App Engine will fail locally.

Figure 2-4. The clock app, with the user signed in

This is great, but it isn’t a complete solution. App Engine lets you request a specific version of a library in app.yaml, and when your app is running on App Engine, exactly that version will be loaded. In the development server, the requested version is ignored, and it just imports whatever you have installed. If your app needs Django 1.3 but you have Django 1.5 installed, the development server will use Django 1.5, no matter what app.yaml says. If you’re developing multiple projects on your computer and two projects need different versions of the same library, site packages are insufficient.

The solution is to use a virtual environment. A virtual environment is a Python environment that is isolated from your main Python installation’s site packages. The environment starts out clean, as if you installed Python for the first time. You can add libraries to the virtual environment, and these will not affect the main Python installation. And you can install different versions of the same libraries in different environments. These environments are “virtual” because they share the same Python interpreter and standard library as your main installation, and use tools to manipulate the Python library load path to isolate your app.

Virtual environments are a good practice for Python development in general, and App Engine in particular. Let’s take a quick look at how to use them.

To get started, install the virtualenv library in your main Python installation:

pip install virtualenv

As before, Mac OS X and Linux users will need sudo pip install virtualenv to install this to the system’s Python, and Windows users may prefer python -m pip install virtualenv.

Installing this package adds the virtualenv command to your system. In Mac OS X and Linux, this is typically at /usr/local/bin/virtualenv. In Windows, it’s C:\Python27\Scripts\virtualenv.exe, or you can use python -m virtualenv.

A virtual environment lives in a new directory. To create a new virtual environment, change the current working directory to where you want this new environment directory to live, then run the virtualenv command with a name for the new environment:

virtualenv myapp_env

The command creates the directory and populates it with tools and its own site-packages directory. To use the environment, you activate it. The process is only slightly different between Mac OS X/Linux and Windows.

In Mac OS X or Linux, you use the shell’s source directive for this, like so:

source ./myapp_env/bin/activate

Similarly, in Windows, you activate a virtual environment with the activate.bat script that virtualenv created in the environment’s Scripts\ directory:

myapp_env\Scripts\activate.bat

The command prompt changes to include the name of the active environment, like so:

(myapp_env)~/

While the environment is active, the environment’s bin/ directory (or Scripts\ directory in Windows) is at the beginning of your command path. This includes special versions of python, easy_install, and pip that know how to use the virtual environment. If you use any of these commands at the prompt, you’ll be using these environment-specific versions.

Let’s do so now. Use pip to install Jinja2, like we did earlier:

pip install jinja2 markupsafe

Mac and Linux users, notice that sudo is not necessary as it was before, because we’re modifying our own environment, not the system-wide Python installation.

You can now start the development server from the command line as usual, and it will use the virtual environment for its libraries:

dev_appserver.py clock

In addition to taking package names on the command line, pip can read a list of packages to install from a requirements file. This is a simple text file that lists the packages to install. It can also request specific versions of the packages. For example, say you had a file named requirements.txtwith the following contents:

jinja2==2.6

markupsafe==0.15

To install the modules listed in this file, pass the filename to pip install using the -r flag:

pip install -r requirements.txt

Using a requirements file that matches the libraries: section of app.yaml makes it easy for you and other members of your team to set up a virtual environment that looks exactly like the App Engine runtime environment for the app. Store this file in your source control repository with the rest of your code.

To deactivate the environment and restore your command path, run the deactivate command:

deactivate

To use a third-party library that is not provided by the runtime environment, you must add the library to your application directory along with your application files. This only works with “pure Python” libraries, and not libraries that depend on natively compiled extensions. You can use pipto install these libraries as well, but notice the key difference. Here, we used pip to install libraries in a local virtual environment to emulate what App Engine provides. When you install a library into your app directory, you are adding the library’s Python code files to the app itself. These files are uploaded to App Engine along with your app code.

To install a library into your app directory, use the -t flag with a path to a subdirectory that will contain it. It’s a good idea to reserve a lib/ directory in the app root for this purpose:

pip install -t lib WTForms WTForms-Appengine

This creates the lib/ directory if needed, downloads the library packages, then installs them. Once completed, lib/ contains directories ending in -info/ that are only used during installation. You can delete these to reduce the number of files that get uploaded. The installed libraries reside in separate directories (such as lib/wtforms).

To use these libraries, you must add the lib/ directory to the module lookup path. Put this at the top of each module that contains request handler code:

import sys

sys.path.append('lib')

With this directory on the lookup path, you can import these libraries in the usual way (such as import wtforms).

Datastore Models and Web Forms

Now that we know who the user is, we can ask her for her preferred time zone, remember her preference, and use it on future visits.

First, we need a way to remember the user’s preferences so future requests can access them. The Google Cloud Datastore provides reliable, scalable storage for this purpose. The Python API includes a data modeling interface that maps Python objects to datastore entities. We can use it to write a UserPrefs class.

Create a new file named models.py, as shown in Example 2-5.

Example 2-5. The file models.py, with a class for storing user preferences in the datastore

from google.appengine.api import users

from google.appengine.ext import ndb

class UserPrefs(ndb.Model):

tz_offset = ndb.FloatProperty(default=0.0)

user = ndb.UserProperty(auto_current_user_add=True)

def get_userprefs(user_id=None):

if notuser_id:

user = users.get_current_user()

if notuser:

return None

user_id = user.user_id()

key = ndb.Key('UserPrefs', user_id)

userprefs = key.get()

if notuserprefs:

userprefs = UserPrefs(id=user_id)

return userprefs

The Python data modeling interface is provided by the module ndb in the package google.appengine.ext. A data model is a class whose base class is ndb.Model. The model subclass defines the structure of the data in each object by using class properties. This structure is enforced byndb.Model when values are assigned to instance properties. For our UserPrefs class, we define two properties: tz_offset, a floating-point number of hours to offset UTC, and user, a User object returned by the Google Accounts API.

Every datastore entity has a primary key. Unlike a primary key in a relational database table, an entity key is permanent and can only be set when the entity is created. A key is unique across all entities in the system, and consists of several parts, including the entity’s kind (in this case,UserPrefs). An app can set one component of the key to an arbitrary value, known in the API as the key name.

The clock application uses the user’s unique ID, provided by the user_id() method of the User object, as the key name of a UserPrefs entity. This allows the app to fetch the entity by key, as it knows the user’s ID from the Google Accounts API. Fetching the entity by key is faster than performing a datastore query.

In models.py, we define a function named get_userprefs() that gets the UserPrefs object for the user. After determining the user ID, the function constructs a datastore key for an entity of the kind UserPrefs with a key name equivalent to the user ID. If the entity exists in the datastore, the function returns the UserPrefs object.

If the entity does not exist in the datastore, the function creates a new UserPrefs object with default settings and a key name that corresponds to the user. The new object is not saved to the datastore automatically. The caller must invoke the put() method on the UserPrefs instance to save it.

Now that we have a mechanism for getting a UserPrefs object, we can make two upgrades to the main page. If the user is signed in, we can get the user’s preferences (if any) and adjust the clock’s time zone.

Edit main.py. With the other import statements, import the models module we just created:

import models

In the request handler code, call the models.get_userprefs() function, and use the return value to adjust the current_time value. Also, add the userprefs value to the template context:

class MainPage(webapp2.RequestHandler):

def get(self):

# ...

userprefs = models.get_userprefs()

if userprefs:

current_time += datetime.timedelta(

0, 0, 0, 0, 0, userprefs.tz_offset)

template = template_env.get_template('home.html')

context = {

# ...

'userprefs': userprefs,

}

self.response.out.write(template.render(context))

Let’s also add a web form to the template so the user can set a time zone preference. You’ll need to edit home.html, adding the following near the bottom of the template, above the </body>:

{% if user %}

<form action="/prefs" method="post">

<label for="tz_offset">

Timezone offset from UTC (can be negative):

</label>

<input name="tz_offset" id="tz_offset" type="text"

size="4" value="{{ userprefs.tz_offset }}" />

<input type="submit" value="Set" />

</form>

{% endif %}

To enable the preferences form, we need a new request handler to parse the form data and update the datastore. Let’s implement this as a new request handler module. (We’ll see why in a moment.)

Create a file named prefs.py with the contents shown in Example 2-6.

Example 2-6. A new handler module, prefs.py, for the preferences form

import webapp2

import models

class PrefsPage(webapp2.RequestHandler):

def post(self):

userprefs = models.get_userprefs()

try:

tz_offset = float(self.request.get('tz_offset'))

userprefs.tz_offset = tz_offset

userprefs.put()

except ValueError:

# User entered a value that wasn't a float. Ignore for now.

pass

self.redirect('/')

application = webapp2.WSGIApplication([('/prefs', PrefsPage)],

debug=True)

This request handler handles HTTP POST requests to the URL /prefs, which is the URL (“action”) and HTTP method used by the form. Because it’s an HTTP POST action, the code goes in the post() method (instead of the get() method used in main.py). The handler code calls theget_userprefs() function from models.py to get the UserPrefs object for the current user, which is either a new unsaved object with default values, or the object for an existing entity. The handler parses the tz_offset parameter from the form data as a float, sets the property of theUserPrefs object, then saves the object to the datastore by calling its put() method. The put() method creates the object if it doesn’t exist, or updates the existing object.

If the user enters something other than a floating-point number in the form field, we don’t do anything. It’d be appropriate to return an error message, but we’ll leave this as is to keep the example simple.

The form handler redirects the user’s browser to the / URL. In webapp2, the self.redirect() method takes care of setting the appropriate response headers for redirecting the browser.

Finally, as shown in Example 2-7, edit app.yaml to map the handler module to the URL /prefs in the handlers: section.

Example 2-7. A new version of app.yaml mapping the URL /prefs, with login required

application: clock

version: 1

runtime: python27

api_version: 1

threadsafe: true

handlers:

- url: /prefs

script: prefs.application

login: required

- url: .*

script: main.application

libraries:

- name: webapp2

version: "2.5.2"

- name: jinja2

version: "2.6"

- name: markupsafe

version: "0.15"

The login: required line says that the user must be signed in to Google Accounts to access the /prefs URL. If the user accesses the URL while not signed in, App Engine automatically directs the user to the Google Accounts sign-in page, then redirects her back to this URL afterward. This makes it easy to require sign-in for sections of your site, and to ensure that the user is signed in before the request handler is called.

Be sure to put the /prefs URL mapping before the .* mapping. URL patterns are tried in order, and the first pattern to match determines the handler used for the request. Because the pattern .* matches all URLs, /prefs must come first or it will be ignored.

Reload the page to see the customizable clock in action. Try changing the time zone by submitting the form. Also try signing out, then signing in again using the same email address, and again with a different email address. The app remembers the time zone preference for each user.

The Development Server Console

The development server has a handy feature for inspecting and debugging your application while testing on your local machine: a web-based console. With your development server running, visit the following URL in a browser to access the console:

§ http://localhost:8000/

(This is the same as the server URL, with a different port number: 8000.)

When the development server console opens for the first time, the Instances panel is shown. At first, only one instance is running locally. The development server is multithreaded, and can simulate an app running with multiple instances on App Engine.

In the left navigation, select datastore viewer. This lets you inspect and manipulate the local simulated instance of Cloud Datastore being used by your app. If you completed and tested the clock app described in this chapter, there should be at least one entity in the viewer, showing its field data and other information. Figure 2-5 shows the datastore viewer in this state.

The Datastore Viewer lets you list and inspect entities by kind, edit entities, and create new ones. You can edit the values for existing properties, but you cannot delete properties or add new ones, nor can you change the type of the value. For new entities, the console makes a guess as to which properties belong on the entity based on existing entities of that kind, and displays a form to fill in those properties. Similarly, you can only create new entities of existing kinds, and cannot create new kinds from the console.

The development server console includes features for managing other aspects of the simulated environment, such as task queues and messaging. We’ll look at these when we discuss the corresponding services.

Figure 2-5. The development server console with the datastore viewer selected

Caching with Memcache

The code that gets user preferences data in Example 2-5 calls the key.get() method to fetch the UserPrefs entity every time a signed-in user visits the site. User preferences are often read and seldom changed, so calling the datastore with every request is more expensive than it needs to be. We can mitigate the cost of reading from primary storage by using a caching layer, secondary storage that’s faster to read than the datastore, but less permanent.

We can use the memory cache service (memcache) as secondary storage for user preferences data. Using memcache with the datastore this way is so common for App Engine apps, the ndb library can do this automatically, and does it by default. We do not need to change our code at all to take advantage of this feature.

To see our app use memcache, open the development server console, then select Memcache Viewer. If you have already tried storing a time zone preference, the console reports that the cache has one item in it. Click the Flush Cache button, and confirm that you want to delete everything from the cache. In a new tab, visit the app in the development server. If you already have a preference stored in the datastore, the app loads it again, and stores it in memcache for future quick reference. (If you don’t have a preference stored, store one now.) Go back to the memcache viewer and reload. Memcache once again contains an item, the cached version of the UserPrefs object.

The ndb library knows to delete the object in the cache whenever it updates the object in the datastore. This keeps fresh data in the cache. However, this behavior is not guaranteed. Memcache and the datastore are separate services, and it is possible for the update to the datastore to succeed and the delete from memcache to fail. In this case, memcache has stale data. To mitigate the problem this may cause, memcache only retains values for a limited amount of time. You can adjust the maximum amount of time it will retain a value. You can also disable ndb’s use of memcache completely.

Later, in Chapter 9, we’ll discuss ndb’s use of memcache in detail. In Chapter 12, we’ll see how to call the memcache service directly, such as for storing the results of computation or network access.

The Python Interactive Console

An especially useful feature exclusive to the Python development server is the Interactive Console. This feature lets you type arbitrary Python code directly into a web form and see the results displayed in the browser. You can use this to write ad hoc Python code to test and manipulate the datastore, memcache, and global data within the local development server.

Here’s an example: run your clock application, sign in with an email address, and then set a time zone preference, such as -8. Now open the development server console, then select Interactive Console. In the lefthand text box, enter the following, where -8 is the time zone preference you used:

import models

q = models.UserPrefs.gql("WHERE tz_offset = -8")

for prefs inq:

print prefs.user

Click the Execute button. The code runs, and the email address you used appears below.

Code run in the development server console behaves just like application code. If you perform a datastore query that needs a custom index, the development server adds configuration for that index to the application’s index.yaml configuration file. Datastore index configuration is discussed inChapter 7.

Registering the Application

Before you can upload your application to App Engine and share it with the world, you must first register a project ID. It’s time to introduce the Cloud Console.

To access the Cloud Console, visit the following URL in your browser, signing in with your Google account if necessary:

§ https://console.developers.google.com/

Bookmark or memorize this URL, as you’ll be back here frequently once your app is live. The Cloud Console is home base for all of your cloud projects.

The home screen lists all of your current projects (if any), and gives you access to your Cloud-related account settings and billing information.2

It costs nothing to create a project. You can have up to 25 free projects per Google account, and you can delete projects later. You can have an unlimited number of paid applications. Note that you will be reserving a unique ID when you create a project, and IDs for deleted projects cannot be reclaimed.

In the Projects section, click the Create Project button. When prompted, enter a project name. The Console generates a unique project ID for you, but you can change it to something more memorable if you like. As we’re about to see, the ID is used in test URLs and a few other places. If you intend to register a domain name for your web application, you will not need to show your project ID to your users.3 You cannot change the ID after the project has been created (but you can delete the project and create a new one). Click Create to create the project.

Uploading the Application

In a traditional web application environment, releasing an application to the world can be a laborious process. Getting the latest software and configuration to multiple web servers and backend services in the right order and at the right time to minimize downtime and prevent breakage is often a difficult and delicate process. With App Engine, deployment is as simple as uploading the files with a single click or command. You can upload and test multiple versions of your application, and set any uploaded version to be the current public version.

Edit your app.yaml configuration file, and replace the application: value with the project ID you registered in the previous step:

application: saucy-boomerang-123

From a command prompt, run the appcfg.py command as follows, substituting the path to your application directory for clock:

appcfg.py update clock

As with dev_appserver.py, clock is just the path to the directory. If the current working directory is the clock/ directory, you can use the relative path, a dot (.).

The appcfg.py tool uses your Google credentials to communicate with Google. These are the credentials you set up earlier with the gcloud auth login command.

TIP

If you intend to use the gcloud command to access other Cloud Platform services, you can tell it to use your new project by default using this command:

gcloud config set project saucy-boomerang-123

To see all of the gcloud configuration fields that you can set in this way, use the following command:

gcloud config list

The upload process determines the project ID and version number from the app.yaml configuration file, then uploads and installs the files and configuration as the given version of the app. The app starts running on App Engine immediately.

Testing the App

Every App Engine project gets a free domain name consisting of the project ID followed by .appspot.com. For example, if your project ID is saucy-boomerang-123, you can access the app at this URL:

http://saucy-boomerang-123.appspot.com/

Visit your app’s URL in a browser, and confirm that it works as expected.

Next, go back to the Cloud Console and select your project. The Overview section now displays graphs representing your test traffic. Figure 2-6 shows an example.

Figure 2-6. The Cloud Console overview screen for a small app (this book’s website)

Navigate to Compute, then App Engine, and finally Dashboard. This dashboard shows more detailed graphs than the project overview and is specific to the App Engine part of Cloud Platform. You should see a small spike in the requests-per-second chart, referring to your test traffic. The scale of the chart goes up to the highest point in the chart, so the spike reaches the top of the graph, even though you have only accessed the application a few times.

Next, open Monitoring in the sidebar, then select the Logs panel. All requests served by App Engine are logged with details about the request and the app’s response. These records also include all messages logged by the application code during the request. Click a record to expand it for more information. You can scroll up and down to attempt to load more records in either direction on the timeline. You can also filter records by log level (such as to only show requests during which your app logged errors), time, label, or a regular expression match.

Take a moment to browse the Console. Throughout this book, we discuss how an application consumes system resources, and how you can optimize an app for speed and cost effectiveness. You will use the Cloud Console to track resource consumption and diagnose problems.

TIP

Google improves the Cloud Console continuously, and menus and buttons may have changed since the time this book was written. In particular, as features graduate from App Engine to Cloud Platform as a whole, their Console panels may relocate in the navigation bar.

Enabling Billing

When you created the project in Cloud Console, you were prompted to set up a billing account, or associate an existing billing account with the project. Google uses this account’s payment information when the project accrues charges.

Google App Engine costs nothing to get started. Simply by creating a project, Google grants you a limited amount of application resources so you can create an app, try the features of the platform, and get your app working and serving live traffic. When you’re ready, you can set a spending limit to make more resources available. This limit applies to all resources managed by App Engine. Other Cloud Platform services, such as Compute Engine, are billed separately.

The default budget is zero dollars, and you will not be billed for App Engine resources until you increase it. You can use the Cloud Console to set the budget. To do so, expand Compute in the sidebar nav, then App Engine, then Settings. Adjust the daily budget, then save your preferences.

1 If you’re developing an old App Engine Python application using Python 2.5, you can continue to do so. However, you will need Python 2.7 to run the Cloud SDK tools. Your Python 2.5 code will work with the Python 2.7 interpreter locally. Upgrading to the Python 2.7 runtime environment is strongly recommended.

2 The descriptions and screenshots of the Google Developers Console in this book are current as of the time this material was produced. Google improves the Console continuously, and some descriptions may be out of date by the time you read this. Such is life.

3 As of this writing, there is one obscure feature that does not yet support custom domains: incoming XMPP messages. I’m not aware of another case where end users must interact with a project ID when the app has a custom domain.