Rapid Prototyping with JS: Agile JavaScript Development (2014)
III. BACK-END PROTOTYPING
8. BONUS: Webapplog Articles
Summary: articles on the essence of asynchronocity in Node.js, TDD with Mocha; introduction to Express.js, Monk, Wintersmith, Derby frameworks/libraries.
“Don’t worry about failure; you only have to be right once.” — Drew Houston
For your convenience we included some of the Node.js posts from Webapplog.com — a publicly accessible blog about web development — in this chapter.
8.1 Asynchronicity in Node
8.1.1 Non-Blocking I/O
One of the biggest advantages of using Node.js over Python or Ruby is that Node has a non-blocking I/O mechanism. To illustrate this, let me use an example of a line in a Starbucks coffeeshop. Let’s pretend that each person standing in line for a drink is a task, and everything behind the counter — cashier, register, barista — is a server or server application. Whether we order a cup of regular drip coffee, like Pike Place, or hot tea, like Earl Grey, the barista makes it. The whole line waits while that drink is made, and each person is charged the appropriate amount.
Of course, we know the aforementioned drinks (a.k.a., time-consuming bottlenecks) are easy to make; just pour the liquid and it’s done. But what about those fancy choco-mocha-frappe-latte-soy-decafs? What if everybody in line decides to order these time-consuming drinks? The line will be held up, and in turn, grow longer and longer. The manager of the coffeeshop will have to add more registers and put more baristas to work (or even stand behind the register him/herself).
This is not good, right? But this is how virtually all server-side technologies work, except Node.js, which is like a real Starbucks. When you order something, the barista yells the order to the other employee, and you leave the register. Another person gives their order while you wait for your state-of-the-art eye-opener in a paper cup. The line moves, the processes are executed asynchronously and without blocking the queue.
This is why Node.js blows everything else away (except maybe low-level C++) in terms of performance and scalability. With Node.js, you just don’t need that many CPUs and servers to handle the load.
8.1.2 Asynchronous Way of Coding
Asynchronicity requires a different way of thinking for programmers familiar with Python, PHP, C or Ruby. It’s easy to introduce a bug unintentionally by forgetting to end the execution of the code with a proper return expression.
Here is a simple example illustrating this scenario:
1 var test = function (callback) {
2 return callback();
3 console.log('test') //shouldn't be printed
4 }
5
6 var test2 = function(callback){
7 callback();
8 console.log('test2') //printed 3rd
9 }
10
11 test(function(){
12 console.log('callback1') //printed first
13 test2(function(){
14 console.log('callback2') //printed 2nd
15 })
16 });
If we don’t use return callback() and just use callback() our string test2 will be printed (test is not printed).
1 callback1
2 callback2
3 tes2
For fun I’ve added a setTimeout() delay for the callback2 string, and now the order has changed:
1 var test = function (callback) {
2 return callback();
3 console.log('test') //shouldn't be printed
4 }
5
6 var test2 = function(callback){
7 callback();
8 console.log('test2') //printed 2nd
9 }
10
11 test(function(){
12 console.log('callback1') //printed first
13 test2(function(){
14 setTimeout(function(){
15 console.log('callback2') //printed 3rd
16 },100)
17 })
18 });
Prints:
1 callback1
2 tes2
3 callback2
The last example illustrates that the two functions are independent of each other and run in parallel. The faster function will finish sooner than the slower one. Going back to our Starbucks examples, you might get your drink faster than the other person who was in front of you in the line. Better for people, and better for programs! :-)
8.2 MongoDB Migration with Monk
Recently one of our top users complained that his Storify account was inaccessible. We’ve checked the production database, and it appears that the account might have been compromised and maliciously deleted by somebody using the user’s account credentials. Thanks to a great MongoHQ service, we had a backup database in less than 15 minutes. There were two options to proceed with the migration:
1. Mongo shell script
2. Node.js program
Because Storify user account deletion involves deletion of all related objects — identities, relationships (followers, subscriptions), likes, stories — we’ve decided to proceed with the latter option. It worked perfectly, and here is a simplified version which you can use as a boilerplate for MongoDB migration (also at gist.github.com/4516139).
Let’s load all of the modules we need: Monk, Progress, Async, and MongoDB:
1 var async = require('async');
2 var ProgressBar = require('progress');
3 var monk = require('monk');
4 var ObjectId=require('mongodb').ObjectID;
By the way, Monk, made by LeanBoost, is a “tiny layer that provides simple yet substantial usability improvements for MongoDB usage within Node.js”.
Monk takes connection string in the following format:
1 username:password@dbhost:port/database
So we can create the following objects:
1 var dest = monk('localhost:27017/storify_localhost');
2 var backup = monk('localhost:27017/storify_backup');
We need to know the object ID which we want to restore:
1 var userId = ObjectId(YOUR-OBJECT-ID);
This is a handy restore() function which we can reuse to restore objects from related collections by specifying the query (for more on MongoDB queries, go to the post Querying 20M-Record MongoDB Collection). To call it, just pass a name of the collection as a string, e.g., "stories" and a query which associates objects from this collection with your main object, e.g., {userId:user.id}. The progress bar is needed to show us nice visuals in the terminal:
1 var restore = function(collection, query, callback){
2 console.info('restoring from ' + collection);
3 var q = query;
4 backup.get(collection).count(q, function(e, n) {
5 console.log('found '+n+' '+collection);
6 if (e) console.error(e);
7 var bar = new ProgressBar('[:bar] :current/:total'
8 + ':percent :etas'
9 , { total: n-1, width: 40 })
10 var tick = function(e) {
11 if (e) {
12 console.error(e);
13 bar.tick();
14 }
15 else {
16 bar.tick();
17 }
18 if (bar.complete) {
19 console.log();
20 console.log('restoring '+collection+' is completed');
21 callback();
22 }
23 };
24 if (n>0){
25 console.log('adding '+ n+ ' '+collection);
26 backup.get(collection).find(q, {
27 stream: true
28 }).each(function(element) {
29 dest.get(collection).insert(element, tick);
30 });
31 } else {
32 callback();
33 }
34 });
35 }
Now we can use async to call the restore() function mentioned above:
1 async.series({
2 restoreUser: function(callback){ // import user element
3 backup.get('users').find({_id:userId}, {
4 stream: true, limit: 1
5 }).each(function(user) {
6 dest.get('users').insert(user, function(e){
7 if (e) {
8 console.log(e);
9 }
10 else {
11 console.log('resored user: '+ user.username);
12 }
13 callback();
14 });
15 });
16 },
17
18 restoreIdentity: function(callback){
19 restore('identities',{
20 userid:userId
21 }, callback);
22 },
23
24 restoreStories: function(callback){
25 restore('stories', {authorid:userId}, callback);
26 }
27
28 }, function(e) {
29 console.log();
30 console.log('restoring is completed!');
31 process.exit(1);
32 });
The full code is available at gist.github.com/4516139 and here:
1 var async = require('async');
2 var ProgressBar = require('progress');
3 var monk = require('monk');
4 var ms = require('ms');
5 var ObjectId=require('mongodb').ObjectID;
6
7 var dest = monk('localhost:27017/storify_localhost');
8 var backup = monk('localhost:27017/storify_backup');
9
10 var userId = ObjectId(YOUR-OBJECT-ID);
11 // monk should have auto casting but we need it for queries
12
13 var restore = function(collection, query, callback){
14 console.info('restoring from ' + collection);
15 var q = query;
16 backup.get(collection).count(q, function(e, n) {
17 console.log('found '+n+' '+collection);
18 if (e) console.error(e);
19 var bar = new ProgressBar(
20 '[:bar] :current/:total :percent :etas',
21 { total: n-1, width: 40 })
22 var tick = function(e) {
23 if (e) {
24 console.error(e);
25 bar.tick();
26 }
27 else {
28 bar.tick();
29 }
30 if (bar.complete) {
31 console.log();
32 console.log('restoring '+collection+' is completed');
33 callback();
34 }
35 };
36 if (n>0){
37 console.log('adding '+ n+ ' '+collection);
38 backup.get(collection).find(q, { stream: true })
39 .each(function(element) {
40 dest.get(collection).insert(element, tick);
41 });
42 } else {
43 callback();
44 }
45 });
46 }
47
48 async.series({
49 restoreUser: function(callback){ // import user element
50 backup.get('users').find({_id:userId}, {
51 stream: true,
52 limit: 1 })
53 .each(function(user) {
54 dest.get('users').insert(user, function(e){
55 if (e) {
56 console.log(e);
57 }
58 else {
59 console.log('resored user: '+ user.username);
60 }
61 callback();
62 });
63 });
64 },
65
66 restoreIdentity: function(callback){
67 restore('identities',{
68 userid:userId
69 }, callback);
70 },
71
72 restoreStories: function(callback){
73 restore('stories', {authorid:userId}, callback);
74 }
75
76 }, function(e) {
77 console.log();
78 console.log('restoring is completed!');
79 process.exit(1);
80 });
To launch it, run npm install/npm update and change the hard-coded database values.
8.3 TDD in Node.js with Mocha
8.3.1 Who Needs Test-Driven Development?
Imagine that you need to implement a complex feature on top of an existing interface, e.g., a ‘like’ button on a comment. Without tests, you’ll have to manually create a user, log in, create a post, create a different user, log in with a different user and like the post. Tiresome? What if you’ll need to do it 10 or 20 times to find and fix some nasty bug? What if your feature breaks existing functionality, but you notice it six months after the release because there was no test?!
Don’t waste time writing tests for throwaway scripts, but please adapt the habit of Test-Driven Development for the main code base. With a little time spent in the beginning, you and your team will save time later and have confidence when rolling out new releases. Test Driven Development is a really, really, really good thing.
8.3.2 Quick Start Guide
Follow this quick guide to set up your Test-Driven Development process in Node.js with Mocha.
Install Mocha globally by executing this command:
1 $ sudo npm install -g mocha
We’ll also use two libraries, Superagent and expect.js by LearnBoost. To install them, fire up NPM commands in your project folder like this:
1 $ npm install superagent
2 $ npm install expect.js
Open a new file with .js extension and type:
1 var request = require('superagent');
2 var expect = require('expect.js');
So far we’ve included two libraries. The structure of the test suite going to look like this:
1 describe('Suite one', function(){
2 it(function(done){
3 ...
4 });
5 it(function(done){
6 ...
7 });
8 });
9 describe('Suite two', function(){
10 it(function(done){
11 ...
12 });
13 });
Inside of this closure, we can write a request to our server, which should be running at localhost:8080:
1 ...
2 it (function(done){
3 request.post('localhost:8080').end(function(res){
4 //TODO check that response is okay
5 });
6 });
7 ...
Expect will give us handy functions to check any condition we can think of:
1 ...
2 expect(res).to.exist;
3 expect(res.status).to.equal(200);
4 expect(res.body).to.contain('world');
5 ...
Lastly, we need to add the done() call to notify Mocha that the asynchronous test has finished its work. And the full code of our first test looks like this:
1 var request = require('superagent');
2 var expect = require('expect.js');
3
4 describe('Suite one', function(){
5 it (function(done){
6 request.post('localhost:8080').end(function(res){
7 expect(res).to.exist;
8 expect(res.status).to.equal(200);
9 expect(res.body).to.contain('world');
10 done();
11 });
12 });
If we want to get fancy, we can add before and beforeEach hooks which will, according to their names, execute once before the test (or suite) or each time before the test (or suite):
1 before(function(){
2 //TODO seed the database
3 });
4 describe('suite one ',function(){
5 beforeEach(function(){
6 //todo log in test user
7 });
8 it('test one', function(done){
9 ...
10 });
11 });
Note that before and beforeEach can be placed inside or outside of the describe construction.
To run our test, simply execute:
1 $ mocha test.js
To use a different report type:
1 $ mocha test.js -R list
2 $ mocah test.js -R spec
8.4 Wintersmith — Static Site Generator
For this book’s one-page website —rapidprototypingwithjs.com — I used Wintersmith to learn something new and to ship fast. Wintersmith is a Node.js static site generator. It greatly impressed me with its flexibility and ease of development. In addition, I could stick to my favorite tools such as Markdown, Jade and Underscore.
Why Static Site Generators
Here is a good article on why using a static site generator is a good idea in general: An Introduction to Static Site Generators. It basically boils down to a few main things:
Templates
You can use template engines such as Jade. Jade uses whitespaces to structure nested elements, and its syntax is similar to Ruby on Rail’s Haml markup.
Markdown
I’ve copied markdown text from my book’s Introduction chapter and used it without any modifications. Wintersmith comes with a marked parser by default. More on why Markdown is great in my old post: Markdown Goodness.
Simple Deployment
Everything is HTML, CSS and JavaScript so you just upload the files with an FTP client, e.g., Transmit (by Panic) or Cyberduck.
Basic Hosting
Due to the fact that any static web server will work well, there is no need for Heroku or Nodejitsu PaaS solutions, or even PHP/MySQL hosting.
Performance
There are no database calls, no server-side API calls, and no CPU/RAM overhead.
Flexibility
Wintersmith allows for different plugins for contents and templates, and you can even write your own plugins.
8.4.1 Getting Started with Wintersmith
There is a quick getting started guide on github.com/jnordberg/wintersmith.
To install Wintersmith globally, run NPM with -g and sudo:
1 $ sudo npm install wintersmith -g
Then run to use the default blog template:
1 $ wintersmith new <path>
or for an empty site:
1 $ wintersmith new <path> -template basic
or use a shortcut:
1 $ wintersmith new <path> -T basic
Similar to Ruby on Rails scaffolding, Wintersmith will generate a basic skeleton with contents and templates folders. To preview a website, run these commands:
1 $ cd <path>
2 $ wintersmith preview
3 $ open http://localhost:8080
Most of the changes will be updates automatically in the preview mode, except for the config.json file.
Images, CSS, JavaScript and other files go into the contents folder. The Wintersmith generator has the following logic:
1. looks for *.md files in the contents folder
2. reads metadata such as the template name
3. processes *.jade templates per metadata in *.md files
When you’re done with your static site, just run:
1 $ wintersmith build
8.4.2 Other Static Site Generators
Here are some of the other Node.js static site generators:
· DocPad
· Blacksmith
· Scotch
· Wheat
· Petrify
A more detailed overview of these static site generators is available in the post Node.js Based Static Site Generators.
For other languages and frameworks like Rails and PHP, take a look at Static Site Generators by GitHub Watcher Count and the “mother of all site generator lists”.
8.5 Intro to Express.js: Simple REST API app with Monk and MongoDB
8.5.1 REST API app with Express.js and Monk
This app is a start of a mongoui project — a phpMyAdmin counterpart for MongoDB written in Node.js. The goal is to provide a module with a nice web admin user interface. It will be something like Parse.com, Firebase.com, MongoHQ or MongoLab has, but without tying it to any particular service. Why do we have to type db.users.findOne({'_id':ObjectId('...')}) anytime we want to look up the user information? The alternative MongoHub Mac app is nice (and free) but clunky to use and not web-based.
Ruby enthusiasts like to compare Express to the Sinatra framework. It’s similarly flexible in the way developers can build their apps. Application routes are set up in a similar manner, i.e., app.get('/products/:id', showProduct);. Currently Express.js is at version number 3.1. In addition to Express, we’ll use the Monk module.
We’ll use Node Package Manager, which usually comes with a Node.js installation. If you don’t have it already, you can get it at npmjs.org.
Create a new folder and NPM configuration file, package.json, in it with the following content:
1 {
2 "name": "mongoui",
3 "version": "0.0.1",
4 "engines": {
5 "node": ">= v0.6"
6 },
7 "dependencies": {
8 "mongodb":"1.2.14",
9 "monk": "0.7.1",
10 "express": "3.1.0"
11 }
12 }
Now run npm install to download and install modules into the node_module folder. If everything went okay you’ll see bunch of folders in node_modules folders. All of the code for our application will be in one file, index.js, to keep it simple stupid:
1 var mongo = require('mongodb');
2 var express = require('express');
3 var monk = require('monk');
4 var db = monk('localhost:27017/test');
5 var app = new express();
6
7 app.use(express.static(__dirname + '/public'));
8 app.get('/',function(req,res){
9 db.driver.admin.listDatabases(function(e,dbs){
10 res.json(dbs);
11 });
12 });
13 app.get('/collections',function(req,res){
14 db.driver.collectionNames(function(e,names){
15 res.json(names);
16 })
17 });
18 app.get('/collections/:name',function(req,res){
19 var collection = db.get(req.params.name);
20 collection.find({},{limit:20},function(e,docs){
21 res.json(docs);
22 })
23 });
24 app.listen(3000)
Let’s break down the code piece by piece. Module declaration:
1 var mongo = require('mongodb');
2 var express = require('express');
3 var monk = require('monk');
Database and Express application instantiation:
1 var db = monk('localhost:27017/test');
2 var app = new express();
Tell Express application to load and server static files (if there are any) from the public folder:
1 app.use(express.static(__dirname + '/public'));
Home page, a.k.a. root route, set up:
1 app.get('/',function(req,res){
2 db.driver.admin.listDatabases(function(e,dbs){
3 res.json(dbs);
4 });
5 });
get() function just takes two parameters: string and function. The string can have slashes and colons — for example, product/:id. The function must have two parameters: request and response. Request has all of the information like query string parameters, session and headers, and response is an object to which we output the results. In this case, we do it by calling the res.json() function.
db.driver.admin.listDatabases(), as you might guess, gives us a list of databases in an asynchronous manner.
Two other routes are set up in a similar manner with the get() function:
1 app.get('/collections',function(req,res){
2 db.driver.collectionNames(function(e,names){
3 res.json(names);
4 })
5 });
6 app.get('/collections/:name',function(req,res){
7 var collection = db.get(req.params.name);
8 collection.find({},{limit:20},function(e,docs){
9 res.json(docs);
10 })
11 });
Express conveniently supports other HTTP verbs like post and update. In the case of setting up a post route, we write this:
1 app.post('product/:id',function(req,res) {...});
Express also has support for middleware. Middleware is just a request function handler with three parameters: request, response, and next. For example:
1 app.post('product/:id',
2 authenticateUser,
3 validateProduct,
4 addProduct
5 );
6
7 function authenticateUser(req,res, next) {
8 //check req.session for authentication
9 next();
10 }
11
12 function validateProduct (req, res, next) {
13 //validate submitted data
14 next();
15 }
16
17 function addProduct (req, res) {
18 //save data to database
19 }
validateProduct and authenticateProduct are middlewares. They are usually put into separate file (or files) in big projects.
Another way to set up middleware in the Express application is to utilize the use() function. For example, earlier we did this for static assets:
1 app.use(express.static(__dirname + '/public'));
We can also do it for error handlers:
1 app.use(errorHandler);
Assuming you have mongoDB installed, this app will connect to it (localhost:27017) and display the collection name and items in collections. To start the mongo server:
1 $ mongod
to run the app (keep the mongod terminal window open):
1 $ node .
or
1 $ node index.js
To see the app working, open localhost:3000 in Chrome with the JSONViewer extension (to render JSON nicely).
8.6 Intro to Express.js: Parameters, Error Handling and Other Middleware
8.6.1 Request Handlers
Express.js is a node.js framework that, among other things, provides a way to organize routes. Each route is defined via a method call on an application object with a URL patter as a first parameter (RegExp is also supported) — for example:
1 app.get('api/v1/stories/', function(res, req){
2 ...
3 })
or, for a POST method:
1 app.post('/api/v1/stories'function(req,res){
2 ...
3 })
Needless to say, DELETE and PUT methods are supported as well.
The callbacks that we pass to the get() or post() methods are called request handlers because they take requests (req), process them, and write to response (res) objects. For example:
1 app.get('/about', function(req,res){
2 res.send('About Us: ...');
3 });
We can have multiple request handlers — hence the name middleware. They accept a third parameter, next, calling which (next()) will switch the execution flow to the next handler:
1 app.get('/api/v1/stories/:id', function(req,res, next) {
2 //do authorization
3 //if not authorized or there is an error
4 // return next(error);
5 //if authorized and no errors
6 return next();
7 }), function(req,res, next) {
8 //extract id and fetch the object from the database
9 //assuming no errors, save story in the request object
10 req.story = story;
11 return next();
12 }), function(req,res) {
13 //output the result of the database search
14 res.send(res.story);
15 });
The ID of a story in URL patter is a query string parameter which we need for finding matching items in the database.
8.6.2 Parameters Middleware
Parameters are values passed in a query string of a URL of the request. If we didn’t have Express.js or a similar library and had to use just the core Node.js modules, we’d have to extract parameters from HTTP.request object via some require('querystring').parse(url) orrequire('url').parse(url, true) functions trickery.
Thanks to Connect framework and the people at VisionMedia, Express.js already has support for parameters, error handling and many other important features in the form of middlewares. This is how we can plug param middleware in our app:
1 app.param('id', function(req,res, next, id){
2 //do something with id
3 //store id or other info in req object
4 //call next when done
5 next();
6 });
7
8 app.get('/api/v1/stories/:id',function(req,res){
9 //param middleware will be execute before and
10 //we expect req object already have needed info
11 //output something
12 res.send(data);
13 });
For example:
1 app.param('id', function(req,res, next, id){
2 req.db.get('stories').findOne({_id:id}, function (e, story){
3 if (e) return next(e);
4 if (!story) return next(new Error('Nothing is found'));
5 req.story = story;
6 next();
7 });
8 });
9
10 app.get('/api/v1/stories/:id',function(req,res){
11 res.send(req.story);
12 });
Or we can use multiple request handlers, but the concept remains the same: we can expect to have the req.story object or an error thrown prior to the execution of this code, so we abstract the common code/logic of getting parameters and their respective objects:
1 app.get('/api/v1/stories/:id', function(req,res, next) {
2 //do authorization
3 }),
4 //we have an object in req.story so no work is needed here
5 function(req,res) {
6 //output the result of the database search
7 res.send(story);
8 });
Authorization and input sanitation are also good candidates for residing in the middlewares.
The function param() is especially cool because we can combine different keys, e.g.:
1 app.get('/api/v1/stories/:storyId/elements/:elementId',
2 function(req,res){
3 res.send(req.element);
4 }
5 );
8.6.3 Error Handling
Error handling is typically used across the whole application, so it’s best to implement it as a middleware. It has the same parameters plus one more, error:
1 app.use(function(err, req, res, next) {
2 //do logging and user-friendly error message display
3 res.send(500);
4 })
In fact, the response can be anything:
JSON string
1 app.use(function(err, req, res, next) {
2 //do logging and user-friendly error message display
3 res.send(500, {status:500,
4 message: 'internal error',
5 type:'internal'}
6 );
7 })
Text message
1 app.use(function(err, req, res, next) {
2 //do logging and user-friendly error message display
3 res.send(500, 'internal server error');
4 })
Error page
1 app.use(function(err, req, res, next) {
2 //do logging and user-friendly error message display
3 //assuming that template engine is plugged in
4 res.render('500');
5 })
Redirect to error page
1 app.use(function(err, req, res, next) {
2 //do logging and user-friendly error message display
3 res.redirect('/public/500.html');
4 })
Error HTTP response status (401, 400, 500, etc.)
1 app.use(function(err, req, res, next) {
2 //do logging and user-friendly error message display
3 res.end(500);
4 })
By the way, logging should also be abstracted in a middleware!
To trigger an error from within your request handlers and middleware, you can just call:
1 next(error);
or
1 next(new Error('Something went wrong :-(');
You can also have multiple error handlers and use named instead of anonymous functions, as its shows in the Express.js Error handling guide.
8.6.4 Other Middleware
In addition to extracting parameters, it can be used for many things, like authorization, error handling, sessions, output, and others.
res.json() is one of them. It conveniently outputs the JavaScript/Node.js object as a JSON. For example:
1 app.get('/api/v1/stories/:id', function(req,res){
2 res.json(req.story);
3 });
is equivalent to (if req.story is an Array and Object):
1 app.get('/api/v1/stories/:id', function(req,res){
2 res.send(req.story);
3 });
or
1 app.get('api/v1/stories/:id',function(req,res){
2 res.set({
3 'Content-Type': 'application/json'
4 });
5 res.send(req.story);
6 });
8.6.5 Abstraction
Middleware is flexible. You can use anonymous or named functions, but the best thing is to abstract request handlers into external modules based on the functionality:
1 var stories = require.('./routes/stories');
2 var elements = require.('./routes/elements');
3 var users = require.('./routes/users');
4 ...
5 app.get('/stories/,stories.find);
6 app.get('/stories/:storyId/elements/:elementId', elements.find);
7 app.put('/users/:userId',users.update);
routes/stories.js:
1 module.exports.find = function(req,res, next) {
2 };
routes/elements.js:
1 module.exports.find = function(req,res,next){
2 };
routes/users.js:
1 module.exports.update = function(req,res,next){
2 };
You can use some functional programming tricks, like this:
1 function requiredParamHandler(param){
2 //do something with a param, e.g.,
3 //check that it's present in a query string
4 return function (req,res, next) {
5 //use param, e.g., if token is valid proceed with next();
6 next();
7 });
8 }
9
10 app.get('/api/v1/stories/:id',
11 requiredParamHandler('token'),
12 story.show
13 );
14
15 var story = {
16 show: function (req, res, next) {
17 //do some logic, e.g., restrict fields to output
18 return res.send();
19 }
20 }
As you can see, middleware is a powerful concept for keeping code organized. The best practice is to keep the router lean and thin by moving all of the logic into corresponding external modules/files. This way, important server configuration parameters will be neatly in one place when you need them! :-)
8.7 JSON REST API server with Node.js and MongoDB using Mongoskin and Express.js
This tutorial will walk you through writing test using the Mocha and Super Agent libraries and then use them in a test-driven development manner to build a Node.js free JSON REST API server utilizing Express.js framework and Mongoskin library for MongoDB. In this REST API server, we’ll perform create, update, remove and delete (CRUD) operations and harness Express.js middleware concept with app.param() and app.use() methods.
8.7.1 Test Coverage
Before anything else let’s write functional tests that make HTTP requests to our soon-to-be-created REST API server. If you know how to use Mocha or just want to jump straight to the Express.js app implementation feel free to do so. You can use CURL terminal commands for testing too.
Assuming we already have Node.js, NPM and MongoDB installed, let’s create a new folder (or if you wrote the tests use that folder):
1 mkdir rest-api
2 cd rest-api
We’ll use Mocha, Expect.js and Super Agent libraries. To install them run these command from the project folder:
1 $ npm install mocha
2 $ npm install expect.js
3 $ npm install superagent
Now let’s create express.test.js file in the same folder which will have six suites:
· creating a new object
· retrieving an object by its ID
· retrieving the whole collection
· updating an object by its ID
· checking an updated object by its ID
· removing an object by its ID
HTTP requests are just a breeze with Super Agent’s chained functions which we’ll put inside of each test suite. Here is the full source code for the express.test.js file:
1 var superagent = require('superagent')
2 var expect = require('expect.js')
3
4 describe('express rest api server', function(){
5 var id
6
7 it('post object', function(done){
8 superagent.post('http://localhost:3000/collections/test')
9 .send({ name: 'John'
10 , email: 'john@rpjs.co'
11 })
12 .end(function(e,res){
13 // console.log(res.body)
14 expect(e).to.eql(null)
15 expect(res.body.length).to.eql(1)
16 expect(res.body[0]._id.length).to.eql(24)
17 id = res.body[0]._id
18 done()
19 })
20 })
21
22 it('retrieves an object', function(done){
23 superagent.get('http://localhost:3000/collections/test/'+id)
24 .end(function(e, res){
25 // console.log(res.body)
26 expect(e).to.eql(null)
27 expect(typeof res.body).to.eql('object')
28 expect(res.body._id.length).to.eql(24)
29 expect(res.body._id).to.eql(id)
30 done()
31 })
32 })
33
34 it('retrieves a collection', function(done){
35 superagent.get('http://localhost:3000/collections/test')
36 .end(function(e, res){
37 // console.log(res.body)
38 expect(e).to.eql(null)
39 expect(res.body.length).to.be.above(1)
40 expect(res.body.map(function (item){
41 return item._id
42 })).to.contain(id)
43 done()
44 })
45 })
46
47 it('updates an object', function(done){
48 superagent.put('http://localhost:3000/collections/test/'+id)
49 .send({name: 'Peter'
50 , email: 'peter@yahoo.com'})
51 .end(function(e, res){
52 // console.log(res.body)
53 expect(e).to.eql(null)
54 expect(typeof res.body).to.eql('object')
55 expect(res.body.msg).to.eql('success')
56 done()
57 })
58 })
59
60 it('checks an updated object', function(done){
61 superagent.get('http://localhost:3000/collections/test/'+id)
62 .end(function(e, res){
63 // console.log(res.body)
64 expect(e).to.eql(null)
65 expect(typeof res.body).to.eql('object')
66 expect(res.body._id.length).to.eql(24)
67 expect(res.body._id).to.eql(id)
68 expect(res.body.name).to.eql('Peter')
69 done()
70 })
71 })
72
73 it('removes an object', function(done){
74 superagent.del('http://localhost:3000/collections/test/'+id)
75 .end(function(e, res){
76 // console.log(res.body)
77 expect(e).to.eql(null)
78 expect(typeof res.body).to.eql('object')
79 expect(res.body.msg).to.eql('success')
80 done()
81 })
82 })
83 })
To run the tests we can use the $ mocha express.test.js command.
8.7.2 Dependencies
In this tutorial we’ll utilize Mongoskin, a MongoDB library which is a better alternative to the plain good old native MongoDB driver for Node.js. In additition Mongoskin is more light-weight than Mongoose and schema-less. For more insight please check out Mongoskin comparison blurb.
Express.js is a wrapper for the core Node.js HTTP module objects. The Express.js framework is build on top of Connect middleware and provided tons of convenience. Some people compare the framework to Ruby’s Sinatra in terms of how it’s non-opinionated and configurable.
If you’ve create a rest-api folder in the previous section Test Coverage, simply run these commands to install modules for the application:
1 npm install express
2 npm install mongoskin
8.7.3 Implementation
First things first, so let’s define our dependencies:
1 var express = require('express')
2 , mongoskin = require('mongoskin')
After the version 3.x, Express streamlines the instantiation of its app instance, in a way that this line will give us a server object:
1 var app = express()
To extract params from the body of the requests we’ll use bodyParser() middleware which looks more like a configuration statement:
1 app.use(express.bodyParser())
Middleware (in this and other forms) is a powerful and convenient pattern in Express.js and Connect to organize and re-use code.
As with the bodyParser() method that saves us from the hurdles of parsing a body object of HTTP request, Mongoskin makes possible to connect to the MongoDB database in one effortless line of code:
1 var db = mongoskin.db('localhost:27017/test', {safe:true});
Note: If you wish to connect to a remote database, e.g., MongoHQ instance, substitute the string with your username, password, host and port values. Here is the format of the URI string: mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]
The app.param() method is another Express.js middleware. It basically says “do something every time there is this value in the URL pattern of the request handler.” In our case we select a particular collection when request pattern contains a sting collectionName prefixed with a colon (you’ll see it later in the routes):
1 app.param('collectionName',
2 function(req, res, next, collectionName) {
3 req.collection = db.collection(collectionName)
4 return next()
5 }
6 )
Merely to be user-friendly, let’s put a root route with a message:
1 app.get('/', function(req, res) {
2 res.send('please select a collection, e.g., /collections/messages')
3 })
Now the real work begins. Here is how we retrieve a list of items sorted by _id and which has a limit of 10:
1 app.get('/collections/:collectionName',
2 function(req, res) {
3 req.collection
4 .find({},
5 {limit:10, sort: [['_id',-1]]}
6 ).toArray(function(e, results){
7 if (e) return next(e)
8 res.send(results)
9 }
10 )
11 }
12 )
Have you noticed a :collectionName string in the URL pattern parameter? This and the previous app.param() middleware is what gives us the req.collection object which points to a specified collection in our database.
The object creating endpoint is slightly easier to grasp since we just pass the whole payload to the MongoDB (method a.k.a. free JSON REST API):
1 app.post('/collections/:collectionName', function(req, res) {
2 req.collection.insert(req.body, {}, function(e, results){
3 if (e) return next(e)
4 res.send(results)
5 })
6 })
Single object retrieval functions are faster than find(), but they use different interface (they return object directly instead of a cursor), so please be aware of that. In addition, we’re extracting the ID from :id part of the path with req.params.id Express.js magic:
1 app.get('/collections/:collectionName/:id', function(req, res) {
2 req.collection.findOne({_id: req.collection.id(req.params.id)},
3 function(e, result){
4 if (e) return next(e)
5 res.send(result)
6 }
7 )
8 })
PUT request handler gets more interesting because update() doesn’t return the augmented object, instead it returns us a count of affected objects.
Also {$set:req.body} is a special MongoDB operator (operators tend to start with a dollar sign) that sets values.
The second ` {safe:true, multi:false}` parameter is an object with options that tell MongoDB to wait for the execution before running the callback function and to process only one (first) item.
1 app.put('/collections/:collectionName/:id', function(req, res) {
2 req.collection.update({_id: req.collection.id(req.params.id)},
3 {$set:req.body},
4 {safe:true, multi:false},
5 function(e, result){
6 if (e) return next(e)
7 res.send((result===1)?{msg:'success'}:{msg:'error'})
8 }
9 )
10 })
Finally, the DELETE method which also output a custom JSON message:
1 app.del('/collections/:collectionName/:id', function(req, res) {
2 req.collection.remove({_id: req.collection.id(req.params.id)},
3 function(e, result){
4 if (e) return next(e)
5 res.send((result===1)?{msg:'success'}:{msg:'error'})
6 }
7 )
8 })
Note: The delete is an operator in JavaScript, so Express.js uses app.del instead.
The last line that actually starts the server on port 3000 in this case:
1 app.listen(3000)
Just in case something is not working quite well here is the full code of express.js file:
1 var express = require('express')
2 , mongoskin = require('mongoskin')
3
4 var app = express()
5 app.use(express.bodyParser())
6
7 var db = mongoskin.db('localhost:27017/test', {safe:true});
8
9 app.param('collectionName',
10 function(req, res, next, collectionName){
11 req.collection = db.collection(collectionName)
12 return next()
13 }
14 )
15 app.get('/', function(req, res) {
16 res.send('please select a collection, '
17 + 'e.g., /collections/messages')
18 })
19 app.get('/collections/:collectionName', function(req, res) {
20 req.collection.find({},{limit:10, sort: [['_id',-1]]})
21 .toArray(function(e, results){
22 if (e) return next(e)
23 res.send(results)
24 }
25 )
26 })
27
28 app.post('/collections/:collectionName', function(req, res) {
29 req.collection.insert(req.body, {}, function(e, results){
30 if (e) return next(e)
31 res.send(results)
32 })
33 })
34 app.get('/collections/:collectionName/:id', function(req, res) {
35 req.collection.findOne({_id: req.collection.id(req.params.id)},
36 function(e, result){
37 if (e) return next(e)
38 res.send(result)
39 }
40 )
41 })
42 app.put('/collections/:collectionName/:id', function(req, res) {
43 req.collection.update({_id: req.collection.id(req.params.id)},
44 {$set:req.body},
45 {safe:true, multi:false},
46 function(e, result){
47 if (e) return next(e)
48 res.send((result===1)?{msg:'success'}:{msg:'error'})
49 }
50 )
51 })
52 app.del('/collections/:collectionName/:id', function(req, res) {
53 req.collection.remove({_id: req.collection.id(req.params.id)},
54 function(e, result){
55 if (e) return next(e)
56 res.send((result===1)?{msg:'success'}:{msg:'error'})
57 }
58 )
59 })
60
61 app.listen(3000)
Exit your editor and run this in your terminal:
1 $ node express.js
And in a different window (without closing the first one):
1 $ mocha express.test.js
If you really don’t like Mocha and/or BDD, CURL is always there for you. :-)
For example, CURL data to make a POST request:
1 $ curl -d "" http://localhost:3000
GET requests also work in the browser, for example http://localhost:3000/test.
In this tutorial our tests are longer than the app code itself so abandoning test-driven development might be tempting, but believe me the good habits of TDD will save you hours and hours during any serious development when the complexity of the applications you work one is big.
8.7.4 Conclusion
The Express.js and Mongoskin libraries are great when you need to build a simple REST API server in a few line of code. Later, if you need to expand the libraries they also provide a way to configure and organize your code.
NoSQL databases like MongoDB are good at free-REST APIs where we don’t have to define schemas and can throw any data and it’ll be saved.
The full code of both test and app files: https://gist.github.com/azat-co/6075685.
If you like to learn more about Express.js and other JavaScript libraries take a look at the series Intro to Express.js tutorials.
Note: *In this example I’m using semi-colon less style. Semi-colons in JavaScript are absolutely optional except in two cases: in the for loop and before expression/statement that starts with parenthesis (e.g., Immediately-Invoked Function Expression).
8.8 Node.js MVC: Express.js + Derby Hello World Tutorial
8.8.1 Node MVC Framework
Express.js is a popular node framework which uses the middleware concept to enhance the functionality of applications. Derby is a new sophisticated Model View Controller (MVC) framework which is designed to be used with Express as its middleware. Derby also comes with the support of Racer, data synchronization engine, and a Handlebars-like template engine, among many other features.
8.8.2 Derby Installation
Let’s set up a basic Derby application architecture without the use of scaffolding. Usually project generators are confusing when people just start to learn a new comprehensive framework. This is a bare minimum “Hello World” application tutorial that still illustrates the Derby skeleton and demonstrates live-templates with websockets.
Of course, we’ll need Node.js and NPM, which can be obtained at nodejs.org. To install Derby globally, run:
1 $ npm install -g derby
To check the installation:
1 $ derby -V
My version, as of April 2013, is 0.3.15. We should be good to go to creating our first app!
8.8.3 File Structure
This is the project folder structure:
1 project/
2 -package.json
3 -index.js
4 -derby-app.js
5 views/
6 derby-app.html
7 styles/
8 derby-app.less
8.8.4 Dependencies
Let’s include dependencies and other basic information in the package.json file:
1 {
2 "name": "DerbyTutorial",
3 "description": "",
4 "version": "0.0.0",
5 "main": "./server.js",
6 "dependencies": {
7 "derby": "*",
8 "express": "3.x"
9 },
10 "private": true
11 }
Now we can run npm install, which will download our dependencies into node_modules folder.
8.8.5 Views
Views must be in the views folder, and they must be either in index.html under a folder which has the same name as your derby app JavaScript file, i.e., views/derby-app/index.html, or be inside of a file which has the same name as your derby app JS file, i.e., derby-app.html.
In this example, the “Hello World” app, we’ll use <Body:> template and {message} variable. Derby uses mustach-handlebars-like syntax for reactive binding. index.html looks like this:
1 <Body:>
2 <input value="{message}"><h1>{message}</h1>
Same thing with Stylus/LESS files; in our example, index.css has just one line:
1 h1 {
2 color: blue;
3 }
To find out more about those wonderful CSS preprocessors, check out the documentation at Stylus and LESS.
8.8.6 Main Server
index.js is our main server file, and we begin it with an inclusion of dependencies with the require() function:
1 var http = require('http'),
2 express = require('express'),
3 derby = require('derby'),
4 derbyApp = require('./derby-app');
The last line is our derby application file derby-app.js.
Now we’re creating the Express.js application (v3.x has significant differences between 2.x) and an HTTP server:
1 var expressApp = new express(),
2 server = http.createServer(expressApp);
Derby uses the Racer data synchronization library, which we create like this:
1 var store = derby.createStore({
2 listen: server
3 });
To fetch some data from back-end to the front-end, we instantiate the model object:
1 var model = store.createModel();
Most importantly we need to pass the model and routes as middlewares to the Express.js app. We need to expose the public folder for socket.io to work properly.
1 expressApp.
2 use(store.modelMiddleware()).
3 use(express.static(__dirname + '/public')).
4 use(derbyApp.router()).
5 use(expressApp.router);
Now we can start the server on port 3001 (or any other):
1 server.listen(3001, function(){
2 model.set('message', 'Hello World!');
3 });
Full code of index.js file:
1 var http = require('http'),
2 express = require('express'),
3 derby = require('derby'),
4 derbyApp = require('./derby-app');
5
6 var expressApp = new express(),
7 server = http.createServer(expressApp);
8
9 var store = derby.createStore({
10 listen: server
11 });
12
13 var model = store.createModel();
14
15 expressApp.
16 use(store.modelMiddleware()).
17 use(express.static(__dirname + '/public')).
18 use(derbyApp.router()).
19 use(expressApp.router);
20
21 server.listen(3001, function(){
22 model.set('message', 'Hello World!');
23 });
8.8.7 Derby Application
Finally, the Derby app file, which contains code for both a front-end and a back-end. Front-end only code is inside of the app.ready() callback. To start, let’s require and create an app. Derby uses unusual construction (not the same familiar good old module.exports = app):
1 var derby = require('derby'),
2 app = derby.createApp(module);
To make socket.io magic work, we need to subscribe a model attribute to its visual representation — in other words, bind data and view. We can do it in the root route, and this is how we define it (patter is /, a.k.a. root):
1 app.get('/', function(page, model, params) {
2 model.subscribe('message', function() {
3 page.render();
4 });
5 });
Full code of derby-app.js file:
1 var derby = require('derby'),
2 app = derby.createApp(module);
3
4 app.get('/', function(page, model, params) {
5 model.subscribe('message', function() {
6 page.render();
7 });
8 });
8.8.8 Launching Hello World App
Now everything should be ready to boot our server. Execute node . or node index.js and open a browser at localhost:3001. You should be able to see something like this:
Derby + Express.js Hello World App
8.8.9 Passing Values to Back-End
Of course, the static data is not much, so we can slightly modify our app to make back-end and front-end pieces talks with each other.
In the server file index.js, add store.afterDb to listen to set events on the message attribute:
1 server.listen(3001, function(){
2 model.set('message', 'Hello World!');
3 store.afterDb('set','message', function(txn, doc, prevDoc, done){
4 console.log(txn)
5 done();
6 });
7 });
Full code of index.js after modifications:
1 var http = require('http'),
2 express = require('express'),
3 derby = require('derby'),
4 derbyApp = require('./derby-app');
5
6 var expressApp = new express(),
7 server = http.createServer(expressApp);
8
9 var store = derby.createStore({
10 listen: server
11 });
12
13 var model = store.createModel();
14
15 expressApp.
16 use(store.modelMiddleware()).
17 use(express.static(__dirname + '/public')).
18 use(derbyApp.router()).
19 use(expressApp.router);
20
21 server.listen(3001, function(){
22 model.set('message', 'Hello World!');
23 store.afterDb('set','message', function(txn, doc, prevDoc, done){
24 console.log(txn)
25 done();
26 });
27 });
In the Derby application file derby-app.js, add model.on() to app.ready():
1 app.ready(function(model){
2 model.on('set', 'message',function(path, object){
3 console.log('message has been changed: '+ object);
4 })
5 });
Full derby-app.js file after modifications:
1 var derby = require('derby'),
2 app = derby.createApp(module);
3
4 app.get('/', function(page, model, params) {
5 model.subscribe('message', function() {
6 page.render();
7 })
8 });
9
10 app.ready(function(model) {
11 model.on('set', 'message', function(path, object) {
12 console.log('message has been changed: ' + object);
13 })
14 });
Now we’ll see logs both in the terminal window and in the browser Developer Tools console. The end result should look like this in the browser:
Hello World App: Browser Console Logs
And like this in the terminal:
Hello World App: Terminal Console Logs
For more magic in the persistence area, check out Racer’s db property. With it you can set up an automatic synch between views and database!
The full code of all of the files in this Express.js + Derby Hello World app is available as a gist at gist.github.com/azat-co/5530311.