Basic API Concerns - Full Stack Web Development with Backbone.js (2014)

Full Stack Web Development with Backbone.js (2014)

Chapter 8. Basic API Concerns

In the last chapter, you used an API that was “canned” to learn about populating a collection with fetch(). While Backbone.js is backend agnostic, you could work with any programming language or data store to build a “real” application stack. However, with the advances of Node and JavaScript, some of the concepts that apply to Backbone at the client equally apply when building APIs or connecting to data stores at the server.

Even if your use case does not require an isomorphic application design, where logic for templating and data validation can be shared, the concepts of this chapter should help you understand more enhancements of Backbone models and collections to “write” state to a remote server.

To persist state with Backbone, we delve deeper into concepts from API development. This is why this chapter on general persistence, and the next chapter on authentication, move the development of Munich Cinema more and more toward full stack JavaScript application development.

Our discussion on fetching movies is extended to storing votes on movies. In general, changing application state also requires discussion on authenticating users. But to simplify the discussion in this chapter, we let users vote on movies without authentication. Then, in the next chapter, authentication will be discussed.

To store state from the client at a server, we first replace the mock API with a Movies API based on Restify and JavaScript promises based on Bluebird. We then connect the Backbone application to the extended API and discuss further enhancements of Backbone models and collections.

Therefore, this chapter explores the following topics:

§ Building a RESTful API with Restify for voting on movies

§ Using a Proxy middleware to connect a separate API process

§ Data sources

Backend Services

While the user interface of a web application is a service that heavily depends on static files (JavaScript and CSS), the design of a backend service (API) is heavily shaped by the type of data stores. In the classical “LAMP” (Linux-Apache-MySQL-PHP) applications stacks, relational database such as MySQL or Postgres influence how data is stored and queried.

Also, in the Munich Cinema application stack, it would be possible to build a Movies service around a relational database based on an object-relational-mapper (ORM). You could either use a different programming language, or proceed (as we will do later) with JavaScript.

TIP

If you want to use JavaScript with a relational database, you might want to look at the Bookshelf ORM. Written by Tim Griesser, the project borrows a number of abstractions, such as models and collections, from the Backbone data layer.

However, discussing the concepts and ideas of backend services would quickly explode the scope of this book. What we can discuss are a number of concepts that influence the setup of Backbone models and collections. Also, it is important to discuss ideas around separating frontend and API service. Even if you will work with an isomorphic JavaScript app, concepts for connecting a data store will be important.

For operations on data stores with JavaScript, such as querying or writing, we face the same time dependency problems as we discussed in Asynchronous Effects.

Hopefully, the ideas in this chapter can help you understand current projects and discussions to try out some nonrelational data stores too, such as Redis, ArangoDB, or MongoDB.

Proxies

When you are developing an API, you often will require working with multiple processes—for example, for running services for the user interface, the API, and a data store.

In this chapter, we will mainly work with two processes. The frontend process takes all incoming HTTP requests, but requests starting with /api will be proxied to the API process.

NOTE

Working with client and API in isolation is also possible with CORS or “cross origin resource sharing” policies. Unfortunately, older web browsers do not support that W3C recommendation. Piotr Sarnacki’s blog post “Client and API Isolation” includes a good discussion on working with a CORS setup.

To proxy requests from one process to another with Node, you can install the proxy middleware with:

$ npm install proxy-middleware --save

You can add the proxy middleware in ./server.js as follows:

var http = require('http');

var express = require('express');

var app = express();

var logger = require('morgan'); // to log requests

var url = require('url');

var proxy = require('proxy-middleware');

app.use(logger({ immediate: true, format: 'dev' }));

// The proxy is added with this line:

app.use('/api', proxy(url.parse('http://0.0.0.0:5001/api/')));

app.use(express.static(__dirname + '/static'));

var port = 5000;

http.createServer(app).listen(port, function() {

console.log('Frontend listening at %s', port);

});

With the line:

app.use('/api', proxy(url.parse('http://0.0.0.0:5001/api/')));

you proxy requests from the server process 0.0.0.0:5000/api to the API process running at 0.0.0.0:5001/api. How this API process looks, and how Backbone collections and models can store state, will be clear in a second.

TIP

For some setups, you might want to proxy a RESTful interface of a database directly. For example, the ArangoDB multipurpose data store provides an easy way to mount RESTful APIs with a package called Foxx.

To make starting your development environment easy, you could extend your Gruntfile to manage the frontend and backend services. As explained in Grunt, you can register tasks to run the two processes automatically as follows:

grunt.registerTask('runFrontend', function () {

grunt.util.spawn({

cmd: 'node',

args: ['./node_modules/.bin/nodemon', 'server.js'],

opts: {

stdio: 'inherit'

}

}, function () {

grunt.fail.fatal(new Error("nodemon quit"));

});

});

grunt.registerTask('runAPI', function () {

grunt.util.spawn({

cmd: 'node',

args: ['./node_modules/.bin/nodemon', 'api.js'],

opts: {

stdio: 'inherit'

}

}, function () {

grunt.fail.fatal(new Error("nodemon quit"));

});

});

// Include both processes in the server task:

grunt.registerTask('server', ['compile', 'runFrontend', 'runAPI', 'watch']);

Before running grunt server, let’s build the Movie services in ./api.js.

Building a Movies Service

When building an API with Node.js, the Restify library by Mark Cavage is a good choice. Restify is very similar to Express.js but keeps it simpler when it comes to building RESTful APIs.

To continue with the API example from Chapter 7, we replace the setup of canned with a small API variation based on Restify.

To set up the Restify server, you can type the following command:

$ npm install restify --save

The specification on our first API endpoints would be:

GET /api/genres

Return all movie genres

GET /api/movies

Return all movies

As a first step, we can serve both resources from the movies.json file and set up a basic Restify server in ./api.js:

// Our main API server is powered by restify

var restify = require('restify');

var _ = require('underscore');

var movies = require('./movies.json');

// Similar to Express.js, we create a server with Restify

var server = restify.createServer({ name: 'movies' })

// Adding middleware to process HTTP bodies

server

.use(restify.fullResponse())

.use(restify.bodyParser())

// The main API route for movies

server.get('/api/movies', function (req, res, next) {

res.send(movies);

})

// The API route to extract a genres of movies

server.get('/api/genres', function (req, res, next) {

var genres = _.chain(movies)

.map(function(movie) {

return movie.genres

})

.flatten()

.uniq()

.value();

res.send(genres);

})

var port = process.env.PORT || 5000;

server.listen(port, function () {

console.log('%s listening at %s', server.name, server.url)

})

We can run this server with:

$ node api.js

Note that we are not simply sending JSON for the genres. Rather, we derive the data with some methods from Underscore.js. That this setup works can be checked with a curl request to genres:

$ curl 0.0.0.0:5000/api/genres

["Drama","Comedy","Action","Adventure","Fantasy","Family","Crime","Animation",

"Mystery","Thriller","Sci-Fi","Western","Biography","History"]

Wrapping a Data Store

Our next goal is to bring the idea of “data store” into the API. The main purpose of a data stores is to improve querying and indexing of data.

For example, if our server managed hundreds or thousands of movies, we would add some parameters for pagination (e.g., by adding skip and limit parameters to address a certain page of movies). The processing of these parameters in the backend goes beyond the scope of this book, but inBasic Sync and Fetch, you saw how query parameters could be passed to an API.

What you will learn, however, in this section are basic concepts of dynamic HTTP responses with JavaScript. Let’s adapt the route for /api/movies:

var restify = require('restify');

var server = restify.createServer({ name: 'movies' });

// We'll play with a simple in-memory data store

var DS = require('./DS.js');

var ds = new DS();

server

.use(restify.fullResponse())

.use(restify.bodyParser())

// This is our first API endpoint

server.get('/api/movies', function (req, res, next) {

// We return all movies from the database

// ... and we'll see how in a second how with:

// return ds.allMovies();

// .then(function(m) { res.send(m); })

// .catch(function(err) { res.send(500, err) });

});

// This is our second API endpoint

server.get("/api/movies/:key", function(req, res, next) {

res.send(400, "pending");

})

var port = process.env.PORT || 5000;

server.listen(port, function () {

console.log('%s listening at %s', server.name, server.url)

})

Right now, the important lines are the ones that set up a database service:

var DS = require('./DS.js');

var ds = new DS();

How the data store DS.js looks will be the main topic of this chapter.

When working with the nonblocking environment of JavaScript on the server, we face similar problems as we discussed in Asynchronous Effects on the client. As previously, we will introduce JavaScript promises to deal with pending (unresolved) states from asynchronous operations.

NOTE

Although promises will be natively supported with the upcoming JavaScript ECMA6 specification, it should be noted that some developers prefer working with callbacks only. Callbacks and promises should not be combined.

With a JavaScript promise, we abstract away the problems that uncertain runtimes for code execution bring. Promises are also part of some object-data mappers in Node.js, such as Bookshelf or Serverbone.

There are currently a number of options for libraries to wrap asynchronous code execution with a promise on the server. We will take a closer look at the Bluebird library by Petka Antonov.

Conceptually, a promise from the Movies data store works as follows. When an HTTP request GET /api/movies hits the server, the Movies data store will be called to provide the movie’s data. Since loading data from a data store can take some time, the function ds.allMovies()returns a promise, which can resolve either to a success or failure, and you can switch the HTTP response depending on the promise outcome with then() and catch():

server.get('/api/movies', function (req, res, next) {

return ds.allMovies()

.then(function(m) { res.send(m); })

.catch(function(err) { res.send(500, err) });

});

With this construct, data is sent to the client when the promise resolves, or an error is sent if there is a problem. For the purposes of illustration, the preceding response (which is written without promises) would look as follows with callbacks:

server.get('/api/movies', function (req, res, next) {

ds.allMoviesCB(function(err, movies) {

if (err) {

res.send(500, err);

}

res.send(movies);

});

});

An important difference to the promise style is, and this is again a matter of opinion, how exceptions are treated. With promises, you can mute an exception and control errors implicitly. While with callbacks, the Node process can “explode” and might not return a response to the client.

Next up, let’s look at the implementation of a very basic data store in DS.js. First, we install Bluebird with npm:

$ npm install bluebird --save

Our data store first reads a file with JSON data. In Node.js, reading a file is also an asynchronous operation and a perfect use case for a promise.

To start, we bring in the dependencies into the data store DS.js, which are:

// We require the filesystem library first

var fs = require('fs');

var fileName = "./movies.json";

var _ = require('underscore');

// Next, we require the Promise library

var Promise = require('bluebird');

An important command from Bluebird is promisifyAll. With this command, you can obtain new functions with a suffix Async that wraps the original function in a promise. Because we will need operations from the filesystem, we do the following:

// We need to wrap the methods from the filesystem with:

Promise.promisifyAll(fs);

Next, we read the movies once into memory:

var Movies;

// prepare Data

var Movies = fs.readFileAsync(fileName, "utf8")

.then(JSON.parse)

The idea here is to use the promise version of fs.readFile and parse the file into JSON objects. An important property of promises is chaining, and you will see how to add further operations later.

With this basic setup, we can build the “core” of a simple data store:

// map only ID and title of a movie

function _mapAttributes(movie) {

return {

id: movie.id,

title: movie.title

};

}

// map all movies attributes

function _mapAllAttributes(movie) {

return {

title: movie.title,

description: movie.description,

showtime: movie.showtime,

rating: movie.rating,

genres: movie.genres,

_key: sha1(movie.title),

};

}

// We will later export the data store to a module

var DS = function() {};

DS.prototype.allMovies = function() {

allMovies: function() {

return Movies.map(_mapAttributes)

}

}

This is the most simple operation: listing all movies. With this, we map all movies from the Movies promise to a simplified JSON structure. In _mapAttributes, we just pick a simple primary key id and the title from the full movies data set. This mapping is an important part of a data “schema,” and we will come back to the role of the id attribute in a second.

To get something running, you can export this basic module with:

module.exports = DS;

To see the basic schema in action, you can start a new API process with:

$ node api.js

And from the command line, we invoke the following test with curl:

$ curl 0.0.0.0:5001/api/movies | jshon

The output from the curl command is piped into jshon, a small command-line JSON beautifier.

We should obtain a result similar to:

[

{

"id": 13,

"title": "The Twilight Saga: Eclipse"

},

{

"id": 14,

"title": "Mission: Impossible – Ghost Protocol"

}

]

Similarly, you can build an API endpoint that finds a single movie resource. First, let’s add a find function to DS.js:

function _find(movies, key) {

var match = _.find(movies, function(movie) {

return movie.id === parseInt(key)

});

if (!match) {

throw Promise.RejectionError("ID not found");

} else {

return match;

}

}

DS.prototype.find = function(key) {

return Movies.then(function(movies) {

return _find(movies, key);

})

.then(_mapAllAttributes);

}

Note that the promise is rejected when no movie is found with a Promise.RejectionError. This operation “error” can be caught from the api.js as follows:

server.get('/api/movies/:key', function(req, res, next) {

return ds.find(req.params.key)

.then(function(m) { res.send(m); })

.error(function (e) {

res.send(404, {err: e.message});

})

.catch(function(err) { res.send(500, err) });

});

To check that this works, you can do:

$ curl 0.0.0.0:5001/api/movies/12 | jshon

{

"title": "The Artist",

"description": "A silent movie star meets a young dancer, but

the arrival of talking pictures sends their careers in

opposite directions.",

"showtime": 1388770080,

"id": 12,

"rating": 2,

"genres": [

"Drama",

"Comedy"

],

"director": "Michel Hazanavicius",

"year": 2009

}

With these basic endpoints, let’s look again at the role of the data “schema,” the blueprint that defines how valid data should look like.

The role of the id attribute is especially important. Depending on the data store that you use, there are different ways to represent the primary key:

§ In many databases, particularly relational databases, primary keys often are in form of 1,2,3,…

§ Many document stores use more complex IDs, because data can be distributed among different nodes, and strategies must be taken to prevent conflicts of IDs. So, we would be able to resolve IDs such as “7658095015.”

To understand how we can control the mapping of primary keys to models in Backbone collection, let’s experiment a bit. What if the primary key of the data store is not called id but _key instead? Let’s simulate this scenario as follows in DS.js:

function _mapAttributes(movie) {

return {

id: movie.id,

title: movie.title,

_key: sha1(movie.title),

};

};

Now, we will use a SHA1 calculation to generate a custom primary key. For this, you can use the “SHA1” in the Node package and install it with:

$ npm install sha1 --save

When we now run a test with curl, we should see something like the following:

[

{

"_key": "3aa4e093a294c5a8ebef09a18a0d172d2c37a03b",

"title": "X-Men: First Class"

},

{

"_key": "6ee51f26282403baac57bd8affd19d9e67ab4252",

"title": "Django Unchained"

}

]

Because the _key attribute should be used to look up models, we adapt the show route to find method as follows:

var match = _.find(movies, function(movie) {

return sha1(movie.title) === key

});

Let’s play with this data store in the Backbone application for Munich Cinema. First, because the primary key of the data is not anymore mapped to id, but to _key, we must tell our Backbone model about the different setup.

Therefore, you must add the _key mapping in app/models/movie.js, as follows:

var Movie = Backbone.Model.extend({

idAttribute: '_key'

});

return Movie;

The idAttribute tells a Backbone model to use a different mapping for fetching and updating models.

To test that this works, you can wire up fetching the details of a single movie in the router with:

routes: {

'details/:key': 'showDetails'

},

showDetails: function(key) {

var movie = new Movie({_key: key});

this.listenTo(movie, 'all', function(ev) { console.log(ev) });

movie.fetch();

}

If we load this route in the browser, you should see the request to the movies API using the _key attribute, as shown in Figure 8-1.

By using the idAttribute in a model, Backbone uses a different key when fetching a model

Figure 8-1. By using the idAttribute in a model, Backbone uses a different key when fetching a model

Let’s continue with learning about persisting application state to the API—for example, to store votes on movies.

Persistence

Our goal is to let users provide a rating with “stars” for movies. For this, we track the average rating of a movie, as well as the votes of a user on a movie.

For voting, we define the following API endpoint:

PUT /movies/:key

parameters: vote

The idea is, after users vote on a movie, we calculate a new score on the server and respond with the average score for a movie.

Again, we start by setting up some logic on the backend for voting on movies. For the API in api.js, you can add the following route:

server.put("/api/movies/:key", function(req, res, next) {

return ds.voteMovie(req.params.key, req.body.vote)

.then(function(m) { res.send(m); })

.error(function (e) {

res.send(404, {err: e.message});

})

.catch(function(err) { res.send(500, err) });

});

And the new operation voteMovie() in the data store ./DS.js looks like the following outline:

DS.prototype.voteMovie = function(id, vote, voter) {

var that = this;

return Movies

.then(function() {

return that.voteExists(id, 0)

})

.then(function(result) {

return that.addVote(vote, id, voter)

})

.then(function() {

return that.computeScore(id)

})

.then(function(score) {

return that.updateScore(id, score);

})

.then(function() {

return that.showMovie(id);

});

}

The basic promise chain is the following: check that users only vote once on a movie, add the vote, and compute a new score. As ranking algorithms can vary, the implementation of computeScore and updateScore is not treated in the book context. But you might want to take a look at the GitHub repo of the book for some ideas.

The new route can be tested via the API endpoints for the application simply with curl:

$ curl -X PUT \

-d "{'vote': 1}" \

0.0.0.0:5001/api/movies/26fc6f540d3a319b0d650df59e0df6ffa05a3224

And, as you can see, the rating attribute increases:

{

"rating": 2,

"description": "To control the oceans, Lord Cutler Beckett ..."

"showtime": 1388733380,

"title": "Pirates of the Caribbean: At World's End",

"director": "Gore Verbinski",

"_key": "26fc6f540d3a319b0d650df59e0df6ffa05a3224",

"year": 2007,

"genres": [

"Action",

"Adventure",

"Fantasy"

],

"length": 169

}

Now, to persist votes form the Backbone application, you can use the save function from Backbone. Similar to fetch, you can pass extra parameters via an options hash.

One simple way to send a vote to the API would be with the following Ajax request in app/models/movie.js:

voteMovie: function(stars) {

var that = this;

this.save({ type: "PUT",

url: "/movies/" + this.id,

contentType: 'application/json',

data: JSON.stringify({vote: stars})

})

.then(function(movie) {

that.set({rating: stars, score: movie.score, rank: movie.rank});

})

.fail(function(err) {

console.log(err);

});

}

When you now go to the browser and require this movie with:

var Movies = require('collections/movie');

movies = new Movies();

movie = movies.get(4);

movie.voteMovie(3);

the movie rating will be saved. In this simple example of saving a model, we don’t require advanced validation logic that can be automatically triggered with Backbone. We will see an example of validating attributes before saving the next chapter, when we discuss the signup of a new user.

Conclusion

This chapter covered the basics of developing an API for a Backbone application. We used JavaScript and Restify to set up an API process, and we built a basic data store based on JavaScript promises. Next, we discussed the importance of mapping the primary key on the server to an ID attribute on the client side. Finally, we saw how to save the new state of a Backbone model to the server.

A voting application without protection of votes does not make much sense, however. We want users to sign up and sign in before they are allowed to submit votes. Also, the validation logic is important when storing data on the server. We’ll work all this out in the next chapter, which covers how to use sessions to deal with multiple users and votes.