Node.js in Action (2014)

Part 2. Web application development with Node

Chapter 9. Advanced Express

This chapter covers

· Implementing authentication

· URL routing

· Creating a REST API

· Handling errors

In this chapter, you’ll learn a number of advanced Express techniques that will enable you to take more advantage of the framework’s functionality.

To demonstrate these techniques, you’ll create a simple application that allows people to register and post public messages that are displayed, in reverse chronological order, for visitors to see. This type of application is known as a “shoutbox” application. Figure 9.1 shows the front and user registration pages. Figure 9.2 shows the login and post pages.

Figure 9.1. The front and registration pages of the shoutbox application

Figure 9.2. The login and post pages of the shoutbox application

For this application, you’ll add logic to do the following:

· Authenticate users

· Implement validation and pagination

· Provide a public representational state transfer (REST) API to send and receive messages

Let’s dive in by leveraging Express for user authentication.

9.1. Authenticating users

In this section you’ll start working on the shoutbox application by creating an authentication system for it from scratch. Within this section you’ll implement the following:

· Logic to store and authenticate registered users

· Registration functionality

· Login functionality

· Middleware to load user information, on request, for logged-in users

For user authentication, you’ll need some way to store the data. For this application you’ll be using Redis, which you learned about in section 5.3.1. It’s a quick install and has a minimal learning curve, which makes it a good candidate since we’re focusing on application logic, not the database layer. The database interaction within this chapter translates well to nearly every database available, so if you’re feeling adventurous you may want to replace Redis with your favorite database. Let’s create a User model.

9.1.1. Saving and loading users

In this section you’ll follow a series of steps to implement user loading, saving, and authentication. You’ll do the following:

· Define application dependencies using a package.json file

· Create a user model

· Add logic to load and save user data using Redis

· Secure user passwords using bcrypt

· Add logic to authenticate attempts to log in

Bcrypt is a salted hashing function that’s available as a third-party module designed specifically for hashing passwords. Bcrypt is great for passwords because as computers get faster and faster, bcrypt can be made slower to effectively eliminate brute-force attacks.

Creating a package.json file

To create an application skeleton with support for EJS and sessions, start a command-line session, change to a development directory, and enter express -e -s shoutbox. You used the -e flag in the previous chapter to enable EJS support in app.js. The -s flag similarly enables sessions support.

With the application skeleton created, change to the shoutbox directory. Next, modify the package.json file, which specifies dependencies, to include a couple of additional modules. Change the package.json file so it looks like the contents of the next listing.

Listing 9.1. A package.json file with additional bcrypt and Redis dependencies

{

"name": "shoutbox",

"version": "0.0.1",

"private": true,

"scripts": {

"start": "node app"

"dependencies": {

"express": "3.x",

"ejs": "*",

"bcrypt": "0.7.3",

"redis": "0.7.2"

}

To install the dependencies, enter npm install. This will install them to ./node _modules.

Finally, execute the following command to create an empty EJS template file that you’ll define later. As this template file is included by other template files, you’ll get errors if you don’t precreate it:

touch views/menu.ejs

With the application skeleton set up and dependencies installed, you can now define the application’s user model.

Creating a user model

You now need to create a lib directory and, within that, a file named user.js. You’ll put the code for the user model in this file.

Listing 9.2 specifies the first logic you’ll want to add. In this code, the redis and bcrypt dependencies are required, and then a Redis connection is opened with redis.createClient(). The User function accepts an object and merges this object’s properties into its own. For example,new User({ name: 'Tobi' }) creates an object and sets the object’s name property to Tobi.

Listing 9.2. Starting to create a user model

Saving a user into Redis

The next functionality you’ll need is the ability to save a user, storing their data with Redis. The save() method shown in listing 9.3 checks if the user already has an ID, and if so it invokes the update() method, indexing the user ID by name, and populating a Redis hash with the object’s properties. Otherwise, if the user doesn’t have an ID, they’re considered a new user, the user:ids value is incremented, which gives the user a unique ID, and the password is hashed before saving into Redis with the same update() method.

Add the code in the following listing to lib/user.js.

Listing 9.3. The user model’s save implementation

Securing user passwords

When the user is first created, it’ll need to have a .pass property set to the user’s password. The user-saving logic will then replace the .pass property with a hash generated using the password.

The hash will be salted. Per-user salting helps to protect against rainbow table attacks: the salt acts as a private key for the hashing mechanism. You can use bcrypt to generate a 12-character salt for the hash with genSalt().

Rainbow table attacks

Rainbow table attacks use precomputed tables to break hashed passwords. You can read more about it in the Wikipedia article: http://en.wikipedia.org/wiki/Rainbow_table.

After the salt is generated, bcrypt.hash() is called, which hashes the .pass property and the salt. This final hash value then replaces the .pass property before .update() stores it in Redis, ensuring that plain-text passwords aren’t saved, only the hash.

The following listing, which you’ll add to lib/user.js, defines a function that creates the salted hash and stores it in the user’s .pass property.

Listing 9.4. Adding bcrypt encryption support to the user model

That’s all there is to it.

Testing the user-saving logic

To try it out, start the Redis server by entering redis-server on the command line. Then add the code in listing 9.5, which will create an example user, to the bottom of lib/user.js. You can then run node lib/user on the command line to create the example user.

Listing 9.5. Testing the user model

You should see output indicating that the user has been created: user id 1, for example. After testing the user model, remove the code in listing 9.5 from lib/user.js.

When you use the redis-cli tool that comes with Redis, you can use the HGETALL command to fetch each key and value of the hash, as the following command-line session demonstrates.

Listing 9.6. Using the redis-cli tool to examine stored data

Having defined logic to save a user, you’ll now need to add logic to retrieve user information.

Other Redis commands you can run in the redis-cli tool

For more information about Redis commands, see the Redis command reference at http://redis.io/commands.

Retrieving user data

When a user attempts to log in to a web application, they’ll usually enter a username and password into a form, and this data is then submitted to the application for authentication. Once the login form is submitted, you’ll need a method for fetching the user via name.

This logic is defined in the following listing as User.getByName(). The function first does an ID lookup with User.getId() and then passes the ID that it finds to User.get(), which gets the Redis hash data for that user. Add the following logic to lib/user.js.

Listing 9.7. Fetching a user from Redis

Having retrieved the hashed password, you can now proceed with authenticating the user.

Authenticating user logins

The final component needed for user authentication is a method, defined in the following listing, that takes advantage of the functions defined earlier for user data retrieval. Add this logic to lib/user.js.

Listing 9.8. Authenticating a user’s name and password

The authentication logic begins by fetching the user by name. If the user isn’t found, the callback function is immediately invoked. Otherwise, the user’s stored salt and the password submitted are hashed to produce what should be identical to the stored user.pass hash. If the submitted and stored hashes don’t match, the user has entered invalid credentials. When looking up a key that doesn’t exist, Redis will give you an empty hash, which is why the check for !user.id is used instead of !user.

Now that you’re able to authenticate users, you’ll need a way for users to register.

9.1.2. Registering new users

To allow users to create new accounts and then sign in, you’ll need both registration and login capabilities.

In this section, you’ll do the following to implement registration:

· Map registration and login routes to URL paths

· Add route logic to display a registration form

· Add logic to store user data submitted from the form

The form will look like figure 9.3.

Figure 9.3. User registration form

This form will be displayed when a user visits /register with a web browser. Later you’ll create a similar form that will allow users to log in.

Adding registration routes

To get the registration form to show up, you’ll first want to create a route to render the form and return it to the user’s browser for display.

Listing 9.9 shows how you should alter app.js, using Node’s module system to import a module defining registration route behavior from the routes directory and associating HTTP methods and URL paths to route functions. This forms a sort of “front controller.” As you can see, there will be both GET and POST register routes.

Listing 9.9. Adding registration routes

Next, to define the route logic, create an empty file in the routes directory called register.js. Start defining registration route behavior by exporting the following function from routes/register.js—a route that renders the registration template:

exports.form = function(req, res){

res.render('register', { title: 'Register' });

};

This route uses an Embedded JavaScript (EJS) template, which you’ll create next, to define the registration form HTML.

Creating a registration form

To define the registration form’s HTML, create a file in the views directory called register.ejs. You can define this form using the HTML/EJS detailed in the following listing.

Listing 9.10. A view template that provides a registration form

Note the use of include messages, which literally includes another template: messages.ejs. This template, which you’ll define next, is used to communicate with the user.

Relaying feedback to users

During user registration, and in many other parts of a typical application, it can be necessary to relay feedback to the user. A user, for example, may attempt to register with a username that someone else is already using. In this case, you’ll need to let the user know they must choose another name.

In your application, the messages.ejs template will be used to display errors. A number of templates throughout the application will include the messages.ejs template.

To create the messages template, create a file in the views directory called messages.ejs and put the logic in the following snippet into it. The template logic checks if the locals.messages variable is set, and, if so, the template cycles through the variable displaying message objects. Each message object has a type property (allowing you to use messages for non-error notifications if need be) and a string property (the message text). Application logic can queue an error for display by adding to the res.locals.messages array. After messages are displayed,removeMessages is called to empty the messages queue:

<% if (locals.messages) { %>

<% messages.forEach(function(message) { %>

<% }) %>

<% removeMessages() %>

<% } %>

Figure 9.4 shows how the registration form will look when displaying an error message.

Figure 9.4. Registration form error reporting

Adding a message to res.locals.messages is a simple way to communicate with the user, but as res.locals doesn’t persist across redirects, you need to make it more robust by storing messages between requests using sessions.

Storing transient messages in sessions

A common web application design pattern is the Post/Redirect/Get (PRG) pattern. In this pattern, a user requests a form, the form data is submitted as an HTTP POST request, and the user is then redirected to another web page. Where the user is redirected to depends on whether the form data was considered valid by the application. If the form data isn’t considered valid, the application redirects the user back to the form page. If the form data is valid, the user is redirected to a new web page. The PRG pattern is primarily used to prevent duplicate form submissions.

In Express, when a user is redirected, the contents of res.locals are reset. If you’re storing messages to the user in res.locals, the messages are lost before they can be displayed. By storing messages in a session variable, however, you can work around this. Messages can then be displayed on the final redirect page.

To accommodate the ability to queue messages to the user in a session variable, you need to add an additional module to your application. Create a file named ./lib/messages.js and add the following code:

var express = require('express');

var res = express.response;

res.message = function(msg, type){

type = type || 'info';

var sess = this.req.session;

sess.messages = sess.messages || [];

sess.messages.push({ type: type, string: msg });

};

The res.message function provides a way to add messages to a session variable from any Express request. The express.response object is the prototype that Express uses for the response objects. Adding properties to this object means they’ll then be available to all middleware and routes alike. In the preceding snippet, express.response is assigned to a variable named res to make it easier to add properties on the object and to improve readability.

To made it even easier to add messages, add the code in the following snippet. The res.error function allows you to easily add a message of type error to the message queue. It leverages the res.message function you previously defined in the module:

res.error = function(msg){

return this.message(msg, 'error');

};

The last step is to expose these messages to the templates for output. If you don’t do this, you’d have to pass req.session.messages to every res.render() call in the application, which isn’t exactly ideal.

To address this, you’ll create middleware that will populate res.locals.messages with the contents of res.session.messages on each request, effectively exposing the messages to any templates that are rendered. So far, ./lib/messages.js extends the response prototype, but it doesn’t export anything. Adding the following snippet to this file, however, will export the middleware you need:

module.exports = function(req, res, next){

res.locals.messages = req.session.messages || [];

res.locals.removeMessages = function(){

req.session.messages = [];

};

next();

};

First, a messages template variable is defined to store the session’s messages—it’s an array that may or may not exist from the previous request (remember that these are session-persisted messages). Next, you’ll need a way to remove the messages from the session; otherwise they’ll build up, because nothing is clearing them.

Now, all you need to do to integrate this new feature is to require() the file in app.js. You should mount this middleware below the session middleware because it depends on req.session being defined. Note that because this middleware was designed not to accept options and doesn’t return a second function, you can call app.use(messages) instead of app.use(messages()). For future-proofing, it’s typically best for third-party middleware to use app.use(messages()) regardless of whether or not it accepts options:

...

var register = require('./routes/register');

var messages = require('./lib/messages');

...

app.use(express.methodOverride());

app.use(express.cookieParser('your secret here'));

app.use(express.session());

app.use(messages);

...

Now you’re able to access messages and removeMessages() within any view, so messages.ejs should work perfectly when included in any template.

With the display of the registration form completed and a way to relay any necessary feedback to the user worked out, let’s move on to handling registration submissions.

Implementing user registration

Now that the registration form is defined and you’ve added a way to relay feedback to the user, you need to create the route function that will handle HTTP POST requests to /register. This function will be called submit.

As discussed in chapter 7, when form data is submitted, the bodyParser() middleware will populate req.body with the submitted data. The registration form uses the object notation user[name], which translates to req.body.user.name once parsed by Connect. Likewise,req.body.user.pass is used for the password field.

You need only a small amount of code in the submission route to handle validation, such as ensuring the username isn’t already taken, and to save the new user, as listing 9.11 shows.

Once registration is complete, the user.id is assigned to the user’s session, which you’ll later check to verify that the user is authenticated. If validation fails, a message is exposed to templates as the messages variable, via res.locals.messages, and the user is redirected back to the registration form.

To add this functionality, add the contents of the following listing to routes/register.js.

Listing 9.11. Creating a user with submitted data

You can now fire up the application, visit /register, and register a user. The next thing you’ll need is a way for returning registered users to authenticate, via the /login form.

9.1.3. Logging in registered users

Adding login functionality is even simpler than registration because the bulk of the necessary logic is already in User.authenticate(), the general-purpose authentication method defined earlier.

In this section you’ll add the following:

· Route logic to display a login form

· Logic to authenticate user data submitted from the form

The form will look like figure 9.5.

Figure 9.5. User login form

Let’s start by modifying app.js so login routes are required and the route paths are established:

...

var login = require('./routes/login');

...

app.get('/login', login.form);

app.post('/login', login.submit);

app.get('/logout', login.logout);

...

Next, you’ll add functionality to display a login form.

Displaying a login form

The first step in implementing a login form is creating a file for login- and logout-related routes: routes/login.js. The route logic you’ll need to add to display the login form is nearly identical to the logic used earlier to display the registration form; the only differences are the name of the template displayed and the page title:

exports.form = function(req, res){

res.render('login', { title: 'Login' });

};

The EJS login form that you’ll define in ./views/login.ejs, shown in listing 9.12, is extremely similar to register.ejs as well; the only differences are the instruction text and the route that data is submitted to.

Listing 9.12. A view template for a login form

Now that you’ve added the route and template needed to display the login form, the next step is to add logic to handle login attempts.

Authenticating logins

To handle login attempts, you need to add route logic that will check the submitted username and password and, if they’re correct, set a session variable to the user’s ID and redirect the user to the home page. The following listing contains this logic, and you should add it to routes/login.js.

Listing 9.13. A route to handle logins

In listing 9.13, if the user is authenticated using User.authenticate(), req.session .uid is assigned in the same way as in the POST /register route: the session will persist this value, which you can use later to retrieve the User or other associated user data. If a match isn’t found, an error is set and the form is redisplayed.

Users may also prefer to explicitly log out, so you should provide a link for this somewhere in the application. In app.js, you assigned app.get('/logout', login.logout), so in ./routes/login.js the following function will remove the session, which is detected by the session()middleware, causing the session to be assigned for subsequent requests:

exports.logout = function(req, res){

req.session.destroy(function(err) {

if (err) throw err;

res.redirect('/');

})

};

Now that the registration and login pages have been created, the next thing you need to add is a menu so users can reach them. Let’s create one.

Creating a menu for authenticated and anonymous users

In this section, you’ll create a menu for both anonymous and authenticated users, allowing them to sign in, register, submit entries, and log out. Figure 9.6 shows the menu for an anonymous user.

Figure 9.6. User login and registration menu used to access the forms you created

When the user is authenticated, you’ll display a different menu showing their username, as well as a link to a page for posting messages to the shoutbox and a link allowing the user to log out. This menu is shown in figure 9.7.

Figure 9.7. Menu when the user is authenticated

Each EJS template you’ve created, representing an application page, has contained the code <% include menu %> after the <body> tag. This includes the ./views/menu.ejs template, which you’ll create next with the contents of the following listing.

Listing 9.14. Anonymous and authenticated user menu template

In this application, you can assume that if a user variable is exposed to the template, that a user is authenticated, because you won’t be exposing the variable otherwise; you’ll see this next. That means that when this variable is present, you can display the username along with the entry submission and logout links. When an anonymous user is visiting, the site login and register links are displayed.

You may be wondering where this user local variable comes from—you haven’t written it yet. Next you’ll write some code to load the logged-in user’s data for each request and make this data available to templates.

9.1.4. User-loading middleware

A common task when you work with a web application is loading user information from a database, typically represented as a JavaScript object. Having this data readily available makes interacting with the user simpler. For this chapter’s application you’ll load the user data on every request, using middleware.

This middleware script will be placed in ./lib/middleware/user.js, requiring the User model from the directory above (./lib). The middleware function is first exported, and then it checks the session for the user ID. When the user ID is present, a user is authenticated, so it’s safe to fetch the user data from Redis.

Because Node is single-threaded, there’s no thread-local storage. In the case of an HTTP server, the request and response variables are the only contextual objects available. High-level frameworks could build upon Node to provide additional objects to store things like the authenticated user, but Express made the choice to stick with the original objects that Node provides. As a result, contextual data is typically stored on the request object, as shown in listing 9.15 where the user is stored as req.user; subsequent middleware and routes can access it using the same property.

You may wonder what the assignment to res.locals.user is for. res.locals is the request-level object that Express provides to expose data to templates, much like app.locals. It’s also a function that can be used to merge existing objects into itself.

Listing 9.15. Middleware that loads a logged-in user’s data

To use this new middleware, first delete all lines in app.js containing the text “user.” You can then require the module as usual, and then pass it to app.use(). In this application, user is used above the router, so only the routes and middleware following user will have access toreq.user. If you’re using middleware that loads data, as this middleware does, you may want to move the express.static middleware above it; otherwise each time a static file is served, a needless round trip to the database will have taken place to fetch the user.

The following listing shows how you can enable this middleware in app.js.

Listing 9.16. Enabling user-loading middleware

If you fire up the application again and visit either the /login or /register pages in your browser, you should see the menu. If you’d like to style the menu, add the following lines of CSS to public/stylesheets/style.css.

Listing 9.17. CSS that can be added to style.css to style application menus

#menu {

position: absolute;

top: 15px;

right: 20px;

font-size: 12px;

color: #888;

}

#menu .name:after {

content: ' -';

}

#menu a {

text-decoration: none;

margin-left: 5px;

color: black;

}

With the menu in place, you should be able to register yourself as a user. Once you’ve registered a user, you should see the authenticated user menu with the Post link.

In the next section, you’ll learn advanced routing techniques while adding the functionality for posting shoutbox messages.

9.2. Advanced routing techniques

The primary function of Express routes is to pair a URL pattern with response logic. Routes can also, however, pair a URL pattern with middleware. This allows you to use middleware to provide reusable functionality to certain routes.

In this section, you’ll do the following:

· Validate user-submitted content using route-specific middleware

· Implement route-specific validation

· Implement pagination

Let’s explore some of the various ways you can leverage route-specific middleware.

9.2.1. Validating user content submission

To give you something to apply validation to, let’s finally add the ability to post to the shoutbox application. To add the ability to post, you’ll need to do a few things:

· Create an entry model

· Add entry-related routes

· Create an entry form

· Add logic to create entries using submitted form data

You’ll start by creating an entry model.

Creating an entry model

Create a file to contain the entry model definition at lib/entry.js. Add the code contained in the following listing to this file. The entry model will be quite similar to the user model created earlier, except it will save data in a Redis list.

Listing 9.18. A model for entries

With the basic model fleshed out, you now need to add a function called getRange, using the contents of the following listing. This function will allow you to retrieve entries.

Listing 9.19. Logic to retrieve a range of entries

With a model created, you can now add routes to list and create entries.

Adding entry-related routes

Before you add entry-related routes to the application, you’ll need to make some modifications to app.js. First, add the following require statement to the top of your app.js file:

var entries = require('./routes/entries');

Next, also in app.js, change the line containing the text app.get('/' to the following to make any requests to the path / return the entry listing:

app.get('/', entries.list);

You can now begin adding routing logic.

Adding front-page display of entries

Start by creating the file routes/entries.js and add the code in the following listing to require the entry model and export a function for rendering a list of entries.

Listing 9.20. Listing entries

With route logic defined for listing entries, you now need to add an EJS template to display them. In the views directory, create a file named entries.ejs and put the following EJS in it.

Listing 9.21. Modified entries.ejs including pagination

<!DOCTYPE html>

<html>

<head>

</head>

<body>

<% include menu %>

<% entries.forEach(function(entry) { %>

<p>Posted by <%= entry.username %></p>

</div>

<% }) %>

</body>

</html>

Now, when you run the application, the front page will display a list of entries. As no entries have yet been created, however, let’s move on to adding the necessary components to create some.

Creating an entry form

Now you have the ability to list entries, but no way to add them. You’ll add this capability next, starting by adding the following lines to the routing section of app.js:

app.get('/post', entries.form);

app.post('/post', entries.submit);

Next, add the following route to routes/entries.js. This route logic will render a template containing a form:

exports.form = function(req, res){

res.render('post', { title: 'Post' });

};

Next, use the EJS template in the following listing to create a template for the form and save it to views/post.ejs.

Listing 9.22. A form into which post data can be entered

With form display taken care of, let’s move on to creating entries from the submitted form data.

Implementing entry creation

To add the capability to create entries from submitted form data, add the logic in the next listing to the file routes/entries.js. This logic will add entries when form data is submitted.

Listing 9.23. Add an entry using submitted form data

exports.submit = function(req, res, next){

var data = req.body.entry;

var entry = new Entry({

"username": res.locals.user.name,

"title": data.title,

"body": data.body

});

entry.save(function(err) {

if (err) return next(err);

res.redirect('/');

});

};

Now when you use a browser to access /post on your application, you’ll be able to add entries if you’re logged in.

With that taken care of, let’s move on to route-specific middleware and how you can use it to validate form data.

9.2.2. Route-specific middleware

Suppose you want the entry text field in the post entry form to be required. The first way you might think of to address this problem is to simply add it straight in your route callback, as shown in the following snippet. This approach isn’t ideal, however, because it tightly ties the validation logic to this particular form. In many cases validation logic can be abstracted into reusable components, making development easier, faster, and more declarative:

...

exports.submit = function(req, res, next){

var data = req.body.entry;

if (!data.title) {

res.error("Title is required.");

res.redirect('back');

return;

}

if (data.title.length < 4) {

res.error("Title must be longer than 4 characters.");

res.redirect('back');

return;

}

...

Express routes can optionally accept middleware of their own, applied only when that route is matched, before the final route callback. The route callbacks themselves that you’ve been using throughout the chapter aren’t treated specially. These are the same as any other middleware, even the ones you’re about to create for validation!

Let’s get started with route-specific middleware by looking at a simple, but inflexible, way to implement validation as route-specific middleware.

Form validation using route-specific middleware

The first possibility is to write a few simple, yet specific, middleware components to perform validation. Extending the POST /post route with this middleware might look something like the following:

app.post('/post',

requireEntryTitle,

requireEntryTitleLengthAbove(4),

entries.submit

);

Note in the previous snippet that the route definition, which normally has only a path and routing logic as arguments, has two additional arguments specifying validation middleware.

The two example middleware components in the following listing illustrate how the original validations can be abstracted out. But they’re still not very modular and only work for the single field entry[title].

Listing 9.24. Two more potential, but imperfect, attempts at validation middleware

function requireEntryTitle(req, res, next) {

var title = req.body.entry.title;

if (title) {

next();

} else {

res.error("Title is required.");

res.redirect('back');

}

function requireEntryTitleLengthAbove(len) {

return function(req, res, next) {

var title = req.body.entry.title;

if (title.length > len) {

next();

} else {

res.error("Title must be longer than " + len);

res.redirect('back');

}

A more viable solution would be to abstract the validators and pass the target field name. Let’s take a look at approaching it this way.

Building flexible validation middleware

You can pass the field name, as shown in the following snippet. This allows you to reuse validation logic, lessening the amount of code you need to write.

app.post('/post',

validate.required('entry[title]'),

validate.lengthAbove('entry[title]', 4),

entries.submit);

Swap the line app.post('/post', entries.submit); in the routing section of app.js with this snippet. It’s worth noting that the Express community has created many similar libraries for public consumption, but understanding how validation middleware works, and how to author your own, is invaluable.

So let’s get on with it. Create a file named ./lib/middleware/validate.js using the program code in the next listing. In it you’ll export several middleware components—in this case, validate.required() and validate.lengthAbove(). The implementation details here aren’t important; the point of this example is that a small amount of effort can go a long way if the code is common within the application.

Listing 9.25. Validation middleware implementation

To make this middleware available to your application, add the following line at the top of app.js:

var validate = require('./lib/middleware/validate');

If you try the application now, you’ll find that the validation will be in effect. This validation API could be made even more fluent, but we’ll leave that for you to investigate.

9.2.3. Implementing pagination

Pagination is another great candidate for route-specific middleware. In this section, you’ll write a small middleware function that will make it easy to paginate any resource you have available.

Designing a pager API

The API for the page() middleware you’ll create will look like the following snippet, where Entry.count is a function that will look up the total count of entries, and 5 is the number to display per page, defaulting to 10. In apps.js, change the line containing app.get('/' to the contents of the following snippet:

app.get('/', page(Entry.count, 5), entries.list);

To make the application ready for the pagination middleware, add the lines in the following snippet to the top of app.js. This will require the pagination middleware you’ll be creating and the entry model:

...

var page = require('./lib/middleware/page');

var Entry = require('./lib/entry');

...

Next, you need to implement Entry.count(). With Redis, this is simple. Open up lib/entry.js and add the following function, which utilizes the LLEN command to get the list’s cardinality (the number of elements):

Entry.count = function(fn){

db.llen('entries', fn);

};

You’re now ready to implement the middleware itself.

Implementing pagination middleware

For pagination, you’ll use the query-string ?page=N value to determine the current page. Add the following middleware function to ./lib/middleware/page.js.

Listing 9.26. Pagination middleware

The middleware in listing 9.26 grabs the value assigned to ?page=N; for example, ?page=1. It then fetches the total number of results and exposes the page object with some precomputed values to any views that may later be rendered. These values are computed outside of the template to allow for a cleaner template containing less logic.

Using the pager in a route

Now you need to update the entries.list route. All you have to change is the original Entry.getRange(0, -1) to use the range that the page() middleware defined, as the following code shows:

exports.list = function(req, res, next){

var page = req.page;

Entry.getRange(page.from, page.to, function(err, entries){

if (err) return next(err);

...

What’s req.param() all about?

req.param() is similar to PHP’s $_REQUEST associative array. It allows you to check the query string, route, or body. For example, ?page=1, /:page with the value /1, or even posting JSON with {"page":1} would all be equivalent. If you were to access req.query.page directly, only the query-string value would be used.

Creating a template for pagination links

Next, you need a template to implement the pager itself. Add the following listing to ./views/pager.ejs, which is a simple pager that consists of Previous and Next buttons.

Listing 9.27. An EJS template for rendering paging buttons

Including pagination links in a template

Now that you’re all set up with the pager middleware and pager template, you can use EJS’s include directive to add the template to the entry listing template ./views /entries.ejs.

Listing 9.28. Modified entries.ejs including pagination

<!DOCTYPE html>

<html>

<head>

</head>

<body>

<% include menu %>

<% entries.forEach(function(entry) { %>

<p>Posted by <%= entry.username %></p>

</div>

<% }) %>

<% include pager %>

</body>

</html>

Enabling clean pagination URLs

You might be wondering how to implement paging using only the pathname, such as /entries/2, instead of a URL parameter, such as ?page=2. Fortunately, only two changes need to be made to the pagination implementation to make this possible:

1. Change the route path to accept a page number.

2. Modify the page template.

The first step is to change the entries listing route path to accept a page number. You could do this by calling app.get() with the string /:page, but you’ll want to consider / equivalent to /0, so make it optional using the string /:page?. In route paths, strings like :page are called routeparameters, or params for short.

With the parameter being optional, both /15 and / are valid, and the page() middleware defaults the page to 1. Because this route is top-level—/5 and not /entries/5, for example—the :page parameter may potentially consume routes such as /upload. The simple solution is to move this route definition down below the others so that it’s the last route defined. This way, more specific routes will be matched before ever reaching this route.

To implement this, the first step is to remove the existing route path in app.js for /. Remove the following line:

app.get('/', page(Entry.count, 5), entries.list);

Next, you’ll want to add the following route path to app.js. Add this after all of the other route definitions:

app.get('/:page?', page(Entry.count, 5), entries.list);

The only other change necessary is to the pager template. The query string needs to be removed so the value becomes part of the path rather than a URL parameter. Change views/pager.ejs to the following:

<% if (page.count > 1) { %>

<% if (page.number) { %>

<% } %>

<% if (page.number < page.count - 1) { %>

<% if (page.number) { %>

<% } %>

<% } %>

</div>

Now if you start up your application, you’ll notice paging URLs are clean.

9.3. Creating a public REST API

In this section, you’ll implement a RESTful public API for the shoutbox application, so that third-party applications can access and add to publication data. The idea of REST is that application data can be queried and changed using verbs and nouns, represented by HTTP methods and URLs, respectively. A REST request will typically return data in a machine-readable form, such as JSON or XML.

To implement an API, you’ll do the following:

· Design an API that allows users to show, list, remove, and post entries

· Add Basic authentication

· Implement routing

· Provide JSON and XML responses

Various techniques can be used to authenticate and sign API requests, but implementing the more complex solutions are beyond the scope of this book. To illustrate how you could integrate authentication, you’ll use the basicAuth() middleware bundled by Connect.

9.3.1. Designing the API

Before proceeding with the implementation, it’s a good idea to rough out the routes that will be involved. For this application, you’ll prefix the RESTful API with the /api path, but this is a design choice you can alter. For example, you may wish to use a subdomain such ashttp://api.myapplication.com.

The following snippet illustrates why it can be a good choice to move the callback functions into separate Node modules, versus defining them inline with the app.VERB() calls. A single list of routes gives you a clear picture of what you and the rest of your team has implemented, and where the implementation callback lives:

app.get('/api/user/:id', api.user);

app.get('/api/entries/:page?', api.entries);

app.post('/api/entry', api.add);

9.3.2. Adding Basic authentication

As previously mentioned, there are many ways to approach API security and restrictions that fall outside the scope of this book. But it’s worth illustrating the process with Basic authentication.

The api.auth middleware will abstract this process, because the implementation will live in the soon-to-be-created ./routes/api.js module. If you recall from chapter 6, app.use() can be passed a pathname. This is the mount point, meaning that request pathnames beginning with /api and any HTTP verb will cause this middleware to be invoked.

The line app.use('/api', api.auth), as shown in the following snippet, should be placed before the middleware that loads user data. This is so that you can later modify the user-loading middleware to load data for authenticated API users:

...

var api = require('./routes/api');

...

app.use('/api', api.auth);

app.use(user);

...

Next, create the ./routes/api.js file, and require both express and the user model, as shown in the following snippet. As mentioned in chapter 7, the basicAuth() middleware accepts a function to perform the authentication, taking the function signature (username, password, callback). Your User.authentication function is a perfect fit:

var express = require('express');

var User = require('../lib/user');

exports.auth = express.basicAuth(User.authenticate);

Authentication is ready to roll. Let’s move on to implementing the API routes.

9.3.3. Implementing routing

The first route you’ll implement is GET /api/user/:id. The logic for this route will have to first fetch the user by ID, responding with a 404 Not Found code if the user doesn’t exist. If the user exists, the user data will be passed to res.send() to be serialized, and the application will respond with a JSON representation of this data. Add the logic in the following snippet to routes/api.js:

exports.user = function(req, res, next){

User.get(req.params.id, function(err, user){

if (err) return next(err);

if (!user.id) return res.send(404);

res.json(user);

});

};

Next, add the following route path to app.js:

app.get('/api/user/:id', api.user);

You’re now ready to test it.

Testing user data retrieval

Fire up the application and test it out with the cURL command-line tool. The following snippet shows how you can test the application’s REST authentication. Credentials are provided in the URL tobi:ferret, which cURL uses to produce the Authorization header field:

$ curl http://tobi:ferret@127.0.0.1:3000/api/user/1 -v

The following listing shows the result of a successful test.

Listing 9.29. Testing output

Removing sensitive user data

As you can see by the JSON response, both the user’s password and salt are provided in the response. To alter this, you can implement .toJSON() on the User.prototype in lib/user.js:

User.prototype.toJSON = function(){

return {

id: this.id,

name: this.name

}

};

If .toJSON exists on an object, it will be used by JSON.stringify calls to get the JSON format. If the cURL request shown earlier was to be issued again, you’d now receive only the ID and name properties:

{

"id": "1",

"name": "tobi"

}

The next thing you’ll add to the API is the ability to create entries.

Adding entries

The processes for adding an entry via the HTML form and through an API are nearly identical, so you’ll likely want to reuse the previously implemented entries.submit() route logic.

When adding entries, however, the route logic stores the name of the user, adding the entry in addition to the other details. For this reason, you’ll need to modify the user-loading middleware to populate res.locals.user with the user data loaded by the basicAuth middleware. ThebasicAuth middleware stores this data in a property of the request object: req.remoteUser. Adding a check for this in the user-loading middleware is straightforward: simply change the module.exports definition in lib/middleware/user.js as follows to make the user-loading middleware work with the API:

...

module.exports = function(req, res, next){

if (req.remoteUser) {

res.locals.user = req.remoteUser;

}

var uid = req.session.uid;

if (!uid) return next();

User.get(uid, function(err, user){

if (err) return next(err);

req.user = res.locals.user = user;

next();

});

};

With this change made, you’ll now be able to add entries via the API.

One more change you’ll want to implement, however, is an API-friendly response, rather than redirection to the application’s homepage. To add this functionality, change the entry.save call in routes/entries.js to the following:

...

entry.save(function(err) {

if (err) return next(err);

if (req.remoteUser) {

res.json({message: 'Entry added.'});

} else {

res.redirect('/');

}

});

...

Finally, to activate the entry-adding API in your application, add the contents of the following snippet to the routing section of api.js:

app.post('/api/entry', entries.submit);

By using the following cURL command, you can test adding an entry via the API. Here the title and body data is sent using the same field names that are in the HTML form:

$ curl -F entry[title]='Ho ho ho' -F entry[body]='Santa loves you'

http://tobi:ferret@127.0.0.1:3000/api/entry

Now that you’ve added the ability to create entries, you need to add the ability to retrieve entry data.

Adding entry listing support

The next API route you’ll implement is GET /api/entries/:page?. The route implementation is nearly identical to the existing entry listing route in ./routes/entries.js. You’ll want to use the already defined page() middleware to provide you with the req.page object used for pagination, as you did previously.

Because the routing logic will be accessing entries, you’ll require the Entry model at the top of routes/api.js using the following line:

var Entry = require('../lib/entry');

Next, you’ll add the line in the following snippet to the app.js routing section:

app.get('/api/entries/:page?', page(Entry.count), api.entries);

Now add the routing logic in the following snippet to routes/api.js. The difference between this route logic and the similar logic in routes/entries.js reflects the fact that you’re no longer rendering a template, but JSON instead:

exports.entries = function(req, res, next){

var page = req.page;

Entry.getRange(page.from, page.to, function(err, entries){

if (err) return next(err);

res.json(entries);

});

};

The following cURL command will request entry data from the API:

$ curl http://tobi:ferret@127.0.0.1:3000/api/entries

This cURL command should result in output similar to the following JSON:

[

{

"username": "rick",

"title": "Cats can't read minds",

"body": "I think you're wrong about the cat thing."

{

"username": "mike",

"title": "I think my cat can read my mind",

"body": "I think cat can hear my thoughts."

...

With basic API implementation covered, let’s move on to look at how APIs can support multiple response formats.

9.3.4. Enabling content negotiation

Content negotiation is what enables a client to specify what formats it’s willing to accept, and which it would prefer. In this section, you’ll provide JSON and XML representations of the API content so that the API consumers can decide what they want.

HTTP provides the content negotiation mechanism via the Accept header field. For example, a client that prefers HTML, but that is willing to accept plain text, could set the following request header:

Accept: text/plain; q=0.5, text/html

The qvalue or quality value (q=0.5 in this example) indicates that even though text/html is specified second, it’s favored by 50 percent over text/plain. Express parses this information and provides a normalized req.accepted array:

[{ value: 'text/html', quality: 1 },

{ value: 'text/plain', quality: 0.5 }]

Express also provides the res.format() method, which accepts an array of MIME types and callbacks. Express will determine what the client is willing to accept and what you’re willing to provide, and it’ll invoke the appropriate callback.

Implementing content negotiation

Implementing content negotiation for the GET /api/entries route might look something like listing 9.30. JSON is supported as it was before—you serialize the entries as JSON with res.send(). The XML callback iterates the entries and writes to the socket as it does so. Note that there’s no need to set the Content-Type explicitly; res .format() will set it to the associated type automatically.

Listing 9.30. Implementing content negotiation

If you set a default response format callback, this will execute if a user hasn’t requested a format you’ve explicitly handled.

The res.format() method also accepts an extension name that maps to an associated MIME type. For example, json and xml can be used in place of application/json and application/xml, as the following snippet shows:

...

res.format({

json: function(){

res.send(entries);

xml: function(){

res.write('<entries>\n');

entries.forEach(function(entry){

res.write(' <entry>\n');

res.write(' <title>' + entry.title + '</title>\n');

res.write(' <body>' + entry.body + '</body>\n');

res.write(' <username>' + entry.username + '</username>\n');

res.write(' </entry>\n');

});

res.end('</entries>');

}

})

...

Responding with XML

Writing a bunch of custom logic in the route in order to respond with XML may not be the cleanest way to go, so let’s use the view system to clean this up.

Create a template named ./views/entries/xml.ejs with the following EJS iterating the entries to generate <entry> tags.

Listing 9.31. Using an EJS template to generate XML

The XML callback can now be replaced with a single res.render() call, passing the entries array, as shown in the following code:

...

xml: function(){

res.render('entries/xml', { entries: entries });

}

})

...

You’re now ready to test the XML version of the API. Enter the following in the command line to see the XML output:

curl -i -H 'Accept: application/xml'

http://tobi:ferret@127.0.0.1:3000/api/entries

9.4. Error handling

So far, neither the application itself nor the API respond with error or 404 Not Found pages. This means that if a resource isn’t found, or if the connection to the database goes down, Express will respond with its default of 404 or 500, respectively. As you can see in figure 9.8, this isn’t user friendly, so let’s customize it. In this section, you’ll implement both 404 and error middleware, which will be used to respond with HTML, JSON, or plain text as accepted by the client.

Figure 9.8. A standard Connect 404 error message

Let’s get started with the missing resources by implementing the 404 middleware.

9.4.1. Handling 404 errors

As previously mentioned, the default behavior when Connect exhausts all middleware without a response is to respond with 404 and a small plain-text string. It looks something like the following response for an entry that doesn’t exist:

$ curl http://tobi:ferret@127.0.0.1:3000/api/not/a/real/path -i

-H "Accept: application/json"

HTTP/1.1 404 Not Found

Content-Type: text/plain

Connection: keep-alive

Transfer-Encoding: chunked

Cannot GET /api/not/a/real/path

Depending on your needs, this may be acceptable, but ideally a JSON API will respond with a JSON response, as the following snippet shows:

$ curl http://tobi:ferret@127.0.0.1:3000/api/not/a/real/path

-i -H "Accept: application/json"

HTTP/1.1 404 Not Found

Content-Type: application/json; charset=utf-8

Content-Length: 37

Connection: keep-alive

{ "message": "Resource not found" }

Implementing the 404 middleware is nothing special; neither Connect nor Express special-case this functionality. A 404 middleware function is a regular middleware function that’s used below any other. If it’s reached, you can safely assume nothing else decided to respond, so you can go ahead and render a template or respond in any other way you prefer.

Figure 9.9 shows the HTML response for a 404 error you’ll create.

Figure 9.9. A 404 error message that’s easier on the eyes than a standard Connect 404 message

Adding a route to return the error response

Open up ./routes/index.js. So far this file only contains the original exports.index function that express(1) generated. Feel free to get rid of that, because it was replaced with entries.list.

The implementation of the error response function will depend on your application’s needs. In the following snippet, you’ll use the res.format() content negotiation method to provide text/html, application/json, and text/plain responses to the client, depending on which they prefer. The response method res.status(code) is identical to setting Node’s res.statusCode = code property, but because it’s a method, it’s chainable, as you can see by the immediate .format() call in the following code.

Listing 9.32. Not Found route logic

exports.notfound = function(req, res){

res.status(404).format({

html: function(){

res.render('404');

json: function(){

res.send({ message: 'Resource not found' });

xml: function() {

res.write('<error>\n');

res.write(' <message>Resource not found</message>\n');

res.end('</error>\n');

text: function(){

res.send('Resource not found\n');

}

});

};

Creating the error page template

You haven’t created the 404 template yet, so create a new file named ./views/404.ejs containing the following EJS snippet. The design of the template is entirely up to you.

Listing 9.33. Sample 404 page

<!DOCTYPE html>

<html>

<head>

<title>404 Not Found</title>

</head>

<body>

<% include menu %>

<h1>404 Not Found</h1>

<p>Sorry we can't find that!</p>

</body>

</html>

Enabling the middleware

Add the routes.notfound middleware below the others, and you can now handle 404 errors as you wish:

...

app.use(app.router);

app.use(routes.notfound);

...

Now that you can handle 404s in style, let’s implement a custom error-handling middleware component to provide a better experience for users when an error occurs.

9.4.2. Handling errors

Up until now you’ve been passing errors to next(). But by default Connect will respond with the canned 500 Internal Server Error response, much like the bland default 404 response. Typically, it’s not a good idea to leak error details to a client, as it poses a potential security issue, but this default response isn’t helpful for the consumers of your API or visitors viewing it from a browser.

In this section, you’ll create a generic 5xx template that will be used to respond to clients when an error occurs. It will provide HTML for clients who accept HTML, and JSON for those accepting JSON, such as the API consumers.

The middleware function can live wherever you like, but for now place it in ./routes/index.js alongside the 404 function. The key difference with the exports .error middleware here is that it accepts four parameters. As we discussed in chapter 6, error-handling middleware must have no more and no fewer than four parameters.

Using a conditional route to test error pages

If your application is robust, it may be difficult to trigger an error on demand. For this reason, it can be handy to create conditional routes. These routes are only enabled via a configuration flag, environment variable, or perhaps an environment type, such as when you’re in development.

The following snippet from app.js illustrates how you can add a /dev/error route to the application only when the ERROR_ROUTE environment variable is specified, creating a faux error with an arbitrary err.type property. Add this code to the routing section of app.js:

if (process.env.ERROR_ROUTE) {

app.get('/dev/error', function(req, res, next){

var err = new Error('database connection failed');

err.type = 'database';

next(err);

});

}

Once this is in place, you can fire up the application with this optional route by executing the following command. Visit /dev/error in the browser if you’re curious, but you’ll be using it in a moment to test the error handler:

$ ERROR_ROUTE=1 node app

Implementing the error handler

To implement the error handler in ./routes/index.js, listing 9.34 starts off with a call to console.error(err.stack). This is possibly the most important line in this function. It ensures that when an error propagates through Connect, reaching this function, you’ll know about it. The error message and stack trace will be written to the stderr stream for later review.

Listing 9.34. Error handler with content negotiation

In order to provide a more meaningful response to the user, without exposing too much information about a given error, you might want to check the properties of the error and respond accordingly. Here the err.type property you added on the /dev/error route is checked in order to customize the message, and you then respond with HTML, JSON, or plain-text representations, much like the 404 handler.

Application error alerts

This unified error handler is a great place to perform additional error-related tasks, like alerting your team that something has gone wrong. Try it out yourself: choose one of the third-party email modules and write an error-handling middleware component that will alert you via email, and then invoke next(err) to pass the error to the remaining error-handling middleware.

Creating the error page template

The EJS template for the res.render('5xx') call will live in ./views/5xx.ejs, as shown in the following listing.

Listing 9.35. Sample 500 error page

<!DOCTYPE html>

<html>

<head>

</head>

<body>

<% include menu %>

<h1><%= status %> Error</h1>

<p>

Try refreshing the page, if this problem

persists then we're already working on it!

</p>

</body>

</html>

Enabling the middleware

By editing app.js and placing the routes.error middleware below the others—even below routes.notfound—you’ll ensure that all errors Connect can see, even potential errors in routes.notfound, will hit this middleware component:

...

app.use(app.router);

app.use(routes.notfound);

app.use(routes.error);

});

Fire up the application with the ERROR_ROUTE enabled again, and take a look at the new error page in figure 9.10.

Figure 9.10. An error page

You’ve now created a fully functioning shoutbox application and have learned some essential Express development techniques in the process.

9.5. Summary

In this chapter, you built a simple web application that employs many aspects of Express’s functionality that we didn’t touch on in the previous chapter. The techniques you’ve learned in this chapter should help you go further in your web application development efforts.

You first created a general-purpose user authentication and registration system that uses sessions to store the IDs of logged-in users and any messages the system wants displayed to the users.

You then leveraged the authentication system, through the use of middleware, to create a REST API. The REST API exposes selected application data to developers and, through the use of content negotiation, makes the data available in either JSON or XML.

Having spent the last two chapters honing your web application development skills, you’re ready to focus on a subject useful for all types of Node development: automated testing.