Learning JavaScript (2016)
Chapter 14. Asynchronous Programming
We got a hint of asynchronous programming in Chapter 1 when we responded to user interaction. Recall that user interaction is naturally asynchronous: you can’t control when a user clicks, touches, speaks, or types. But user input isn’t the only reason for asynchronous execution: the very nature of JavaScript makes it necessary for many things.
When a JavaScript application runs, it runs single-threaded. That is, JavaScript only ever does one thing at a time. Most modern computers are capable of doing multiple things at once (assuming they have multiple cores), and even computers with a single core are so fast that they can simulate doing multiple things at once by doing a tiny bit of task A, then a tiny bit of task B, then a tiny bit of task C, and so on until all the tasks are done (this is called preemptive multitasking). From the user’s perspective, tasks A, B, and C ran simultaneously, whether or not the tasks were actually running simultaneously on multiple cores.
So JavaScript’s single-threaded nature might strike you as a limiting factor, but it actually frees you from having to worry about some very thorny problems that are present in multithreaded programming. This freedom comes at a cost, though: it means that to write smoothly running software, you have to think asynchronously, and not just for user input. Thinking this way can be difficult at first, especially if you’re coming from a language where execution is normally synchronous.
JavaScript has had a mechanism for asynchronous execution from the earliest days of the language. However, as the popularity of JavaScript has grown—and the sophistication of the software being written for it—new constructs to manage asynchronous programming have been added to the language. As a matter of fact, we can think of JavaScript as having three distinct eras of asynchronous support: the callback era, the promise era, and the generator era. If it were a simple matter of generators being better than everything that came before them, we would explain how generators work and leave the rest in the past. It’s not that simple, though. Generators by themselves do not provide any sort of asynchronous support: they rely on either promises or a special type of callback to provide asynchronous behavior. Likewise, as useful as promises are, they rely on callbacks (and callbacks by themselves are still useful for things like events).
Aside from user input, the three primary things you’ll be using asynchronous techniques for are:
§ Network requests (Ajax calls, for instance)
§ Filesystem operations (reading/writing files, etc.)
§ Intentionally time-delayed functionality (an alarm, for example)
The Analogy
The analogy I like to use for both callbacks and promises is getting a table at a busy restaurant when you don’t have a reservation. So you don’t have to wait in line, one restaurant will take your mobile phone number and call you when your table is ready. This is like a callback: you’ve provided the host with something that allows them to let you know when your table is ready. The restaurant is busy doing things, and you can busy yourself with other things; nobody’s waiting on anyone else. Another restaurant might give you a pager that will buzz when your table is ready. This is more like a promise: something the host gives to you that will let you know when your table is ready.
As we move through callbacks and promises, keep these analogies in mind, especially if you are new to asynchronous programming.
Callbacks
Callbacks are the oldest asynchronous mechanism in JavaScript, and we’ve already seen them used to handle user input and timeouts. A callback is simply a function that you write that will be invoked at some point in the future. There’s nothing special about the function itself: it’s just a regular JavaScript function. Typically, you provide these callback functions to other functions, or set them as properties on objects (or, rarely, provide them in an array). Callbacks are very often (but not always) anonymous functions.
Let’s start with a simple example, using setTimeout, a built-in function that delays execution some number of milliseconds:
console.log("Before timeout: " + new Date());
function f() {
console.log("After timeout: " + new Date());
}
setTimeout(f, 60*1000); // one minute
console.log("I happen after setTimeout!");
console.log("Me too!");
If you run this at the console—unless you are a very slow typist indeed—you should see something like this:
Before timeout: Sun Aug 02 2015 17:11:32 GMT-0700 (Pacific Daylight Time)
I happen after setTimeout!
Me too!
After timeout: Sun Aug 02 2015 17:12:32 GMT-0700 (Pacific Daylight Time)
What beginners struggle with is the disconnect between the linear nature of the code we write and the actual execution of that code. There is a part of us that wants—that expects—a computer to do things in the exact order in which we write them. In other words, we want to see this:
Before timeout: Sun Aug 02 2015 17:11:32 GMT-0700 (Pacific Daylight Time)
After timeout: Sun Aug 02 2015 17:12:32 GMT-0700 (Pacific Daylight Time)
I happen after setTimeout!
Me too!
We may want to see that…but it wouldn’t be asynchronous. The whole point of asynchronous execution is that it doesn’t block anything. Because JavaScript is single-threaded, if we told it to wait for 60 seconds before doing some code, and we did that synchronously, nothing would work. Your program would freeze up: it wouldn’t accept user input, it wouldn’t update the screen, and so on. We’ve all had this experience, and it’s not desirable. Asynchronous techniques help prevent this kind of lock-up.
In this example, for clarity, we used a named function to pass to setTimeout. Unless there were some compelling reason to have a named function, we would normally just use an anonymous function:
setTimeout(function() {
console.log("After timeout: " + new Date());
}, 60*1000);
setTimeout is a little bit problematic because the numeric timeout parameter is the last argument; with anonymous functions, especially if they’re long, it can get lost or look like it’s part of the function. This is common enough, however, that you’ll have to get used to seeing setTimeout(and its companion, setInterval) used with anonymous functions. Just remember that the last line contains the delay parameter.
setInterval and clearInterval
In addition to setTimeout, which runs its function once and stops, there’s setInterval, which runs the callback at the specified interval forever, or until you call clearInterval. Here’s an example that runs every 5 seconds until the minute rolls over, or 10 times, whichever comes first:
const start = new Date();
let i=0;
const intervalId = setInterval(function() {
let now = new Date();
if(now.getMinutes() !== start.getMinutes() || ++i>10)
return clearInterval(intervalId);
console.log(`${i}: ${now}`);
}, 5*1000);
We see here that setInterval returns an ID that we can use to cancel (stop) it later. There’s a corresponding clearTimeout that works the same way and allows you to stop a timeout before it runs.
NOTE
setTimeout, setInterval, and clearInterval are all defined on the global object (window in a browser, and global in Node).
Scope and Asynchronous Execution
A common source of confusion—and errors—with asynchronous execution is how scope and closures affect asynchronous execution. Every time you invoke a function, you create a closure: all of the variables that are created inside the function (including the arguments) exist as long as something can access them.
We’ve seen this example before, but it bears repeating for the important lesson we can learn from it. Consider the example of a function called countdown. The intended purpose is to create a 5-second countdown:
function countdown() {
let i; // note we declare let outside of the for loop
console.log("Countdown:");
for(i=5; i>=0; i--) {
setTimeout(function() {
console.log(i===0 ? "GO!" : i);
}, (5-i)*1000);
}
}
countdown();
Go ahead and work through this example in your head first. You probably remember from the last time you saw this that something’s wrong. It looks like what’s expected is a countdown from 5. What you get instead is –1 six times and no "GO!". The first time we saw this, we were usingvar; this time we’re using let, but it’s declared outside of the for loop, so we have the same problem: the for loop executes completely, leaving i with the value –1, and only then do the callbacks start executing. The problem is, when they execute, i already has the value –1.
The important lesson here is understanding the way scope and asynchronous execution relate to each other. When we invoke countdown, we’re creating a closure that contains the variable i. All of the (anonymous) callbacks that we create in the for loop all have access to i—the same i.
The tidy thing about this example is that inside the for loop, we see i used in two different ways. When we use it to calculate the timeout ((5-i)*1000), it works as expected: the first timeout is 0, the second timeout is 1000, the third timeout is 2000, and so on. That’s because that calculation is synchronous. As a matter of fact, the call to setTimeout is also synchronous (which requires the calculation to happen so that setTimeout knows when to invoke the callback). The asynchronous part is the function that’s passed to setTimeout, and that’s where the problem occurs.
Recall that we can solve this problem with an immediately invoked function expression (IIFE), or more simply by just moving the declaration of i into the for loop declaration:
function countdown() {
console.log("Countdown:");
for(let i=5; i>=0; i--) { // i is now block-scoped
setTimeout(function() {
console.log(i===0 ? "GO!" : i);
}, (5-i)*1000);
}
}
countdown();
The takeaway here is that you have to be mindful of the scope your callbacks are declared in: they will have access to everything in that scope (closure). And because of that, the value may be different when the callback actually executes. This principle applies to all asynchronous techniques, not just callbacks.
Error-First Callbacks
At some point during the ascendancy of Node, a convention called error-first callbacks established itself. Because callbacks make exception handling difficult (which we’ll see soon), there needed to be a standard way to communicate a failure to the callback. The convention that emerged was to use the first argument to a callback to receive an error object. If that error is null or undefined, there was no error.
Whenever you’re dealing with an error-first callback, the first thing you should do is check the error argument and take appropriate action. Consider reading the contents of a file in Node, which adheres to the error-first callback convention:
const fs = require('fs');
const fname = 'may_or_may_not_exist.txt';
fs.readFile(fname, function(err, data) {
if(err) return console.error(`error reading file ${fname}: ${err.message}`);
console.log(`${fname} contents: ${data}`);
});
The first thing we do in the callback is see if err is truthy. If it is, there was an issue reading the file, and we report that to the console and immediately return (console.error doesn’t evaluate to a meaningful value, but we’re not using the return value anyway, so we just combine it into one statement). This is probably the most often overlooked mistake with error-first callbacks: the programmer will remember to check it, and perhaps log the error, but not return. If the function is allowed to continue, it may rely on the callback having been successful, which it wasn’t. (It is possible, of course, that the callback doesn’t completely rely on success, in which case it may be acceptable to note the error and proceed anyway.)
Error-first callbacks are the de facto standard in Node development (when promises aren’t being used), and if you’re writing an interface that takes a callback, I strongly advise you to adhere to the error-first convention.
Callback Hell
While callbacks allow you to manage asynchronous execution, they have a practical drawback: they’re difficult to manage when you need to wait on multiple things before proceeding. Imagine the scenario where you’re writing a Node app that needs to get the contents of three different files, then wait 60 seconds before combining the contents of those files and writing to a fourth file:
const fs = require('fs');
fs.readFile('a.txt', function(err, dataA) {
if(err) console.error(err);
fs.readFile('b.txt', function(err, dataB) {
if(err) console.error(err);
fs.readFile('c.txt', function(err, dataC) {
if(err) console.error(err);
setTimeout(function() {
fs.writeFile('d.txt', dataA+dataB+dataC, function(err) {
if(err) console.error(err);
});
}, 60*1000);
});
});
});
This is what programmers refer to as “callback hell,” and it’s typified by a triangle-shaped block of code with curly braces nested to the sky. Worse still is the problem of error handling. In this example, all we’re doing is logging the errors, but if we tried throwing an exception, we’d be in for another rude surprise. Consider the following simpler example:
const fs = require('fs');
function readSketchyFile() {
try {
fs.readFile('does_not_exist.txt', function(err, data) {
if(err) throw err;
});
} catch(err) {
console.log('warning: minor issue occurred, program continuing');
}
}
readSketchyFile();
Glancing over this, it seems reasonable enough, and hooray for us for being defensive programmers and using exception handling. Except it won’t work. Go ahead and try it: the program will crash, even though we took some care to make sure this semi-expected error didn’t cause problems. That’s because try...catch blocks only work within the same function. The try...catch block is in readSketchyFile, but the error is thrown inside the anonymous function that fs.readFile invokes as a callback.
Additionally, there’s nothing to prevent your callback from accidentally getting called twice—or never getting called at all. If you’re relying on it getting called once and exactly once, the language itself offers no protection against that expectation being violated.
These are not insurmountable problems, but with the prevalence of asynchronous code, it makes writing bug-free, maintainable code very difficult, which is where promises come in.
Promises
Promises attempt to address some of the shortcomings of callbacks. Using promises—while sometimes a hassle—generally results in safer, easier-to-maintain code.
Promises don’t eliminate callbacks; as a matter of fact, you still have to use callbacks with promises. What promises do is ensure that callbacks are always handled in the same predictable manner, eliminating some of the nasty surprises and hard-to-find bugs you can get with callbacks alone.
The basic idea of a promise is simple: when you call a promise-based asynchronous function, it returns a Promise instance. Only two things can happen to that promise: it can be fulfilled (success) or rejected (failure). You are guaranteed that only one of those things will happen (it won’t be fulfilled and then later rejected), and the result will happen only once (if it’s fulfilled, it’ll only be fulfilled once; if it’s rejected, it’ll only be rejected once). Once a promise has either been fulfilled or rejected, it’s considered to be settled.
Another convenient advantage of promises over callbacks is that, because they’re just objects, they can be passed around. If you want to kick off an asynchronous process, but would prefer someone else handle the results, you can just pass the promise to them (this would be analogous to giving the reservation pager to a friend of yours—ostensibly the restaurant won’t mind who takes the reservation as long as the party size is the same).
Creating Promises
Creating a promise is a straightforward affair: you create a new Promise instance with a function that has resolve (fulfill) and reject callbacks (I did warn you that promises don’t save us from callbacks). Let’s take our countdown function, parameterize it (so we’re not stuck with only a 5-second countdown), and have it return a promise when the countdown is up:
function countdown(seconds) {
return new Promise(function(resolve, reject) {
for(let i=seconds; i>=0; i--) {
setTimeout(function() {
if(i>0) console.log(i + '...');
else resolve(console.log("GO!"));
}, (seconds-i)*1000);
}
});
}
This isn’t a very flexible function right now. We might not want to use the same verbiage—we may not even want to use the console at all. This wouldn’t do us very much good on a web page where we wanted to update a DOM element with our countdown. But it’s a start…and it shows how to create a promise. Note that resolve (like reject) is a function. You might be thinking “Ah ha! I could call resolve multiple times, and break the er…promise of promises.” You could indeed call resolve or reject multiple times or mix them up…but only the first call will count. The promise will make sure that whoever is using your promise will only get a fulfillment or a rejection (currently, our function doesn’t have a rejection pathway).
Using Promises
Let’s see how we can use our countdown function. We could just call it and ignore the promise altogether: countdown(5). We’ll still get our countdown, and we didn’t have to fuss with the promise at all. But what if we want to take advantage of the promise? Here’s how we use the promise that’s returned:
countdown(5).then(
function() {
console.log("countdown completed successfully");
},
function(err) {
console.log("countdown experienced an error: " + err.message);
}
);
In this example, we didn’t bother assigning the returned promise to a variable; we just called its then handler directly. That handler takes two callbacks: the first one is the fulfilled callback, and the second is the error callback. At most, only one of these functions will get called. Promises also support a catch handler so you can split up the two handlers (we’ll also store the promise in a variable to demonstrate that):
const p = countdown(5);
p.then(function() {
console.log("countdown completed successfully");
});
p.catch(function(err) {
console.log("countdown experienced an error: " + err.message);
});
Let’s modify our countdown function to have an error condition. Imagine we’re superstitious, and we’ll have an error if we have to count the number 13.
function countdown(seconds) {
return new Promise(function(resolve, reject) {
for(let i=seconds; i>=0; i--) {
setTimeout(function() {
if(i===13) return reject(new Error("DEFINITELY NOT COUNTING THAT"));
if(i>0) console.log(i + '...');
else resolve(console.log("GO!"));
}, (seconds-i)*1000);
}
});
}
Go ahead and play around with this. You’ll notice some interesting behavior. Obviously, you can count down from any number less than 13, and it will behave normally. Count down from 13 or higher, and it will fail when it gets to 13. However…the console logs still happen. Calling reject(or resolve) doesn’t stop your function; it just manages the state of the promise.
Clearly our countdown function needs some improvements. Normally, you wouldn’t want a function to keep working after it had settled (successfully or otherwise), and ours does. We’ve also already mentioned that the console logs aren’t very flexible. They don’t really give us the control we’d like.
Promises give us an extremely well-defined and safe mechanism for asynchronous tasks that either fulfill or reject, but they do not (currently) provide any way to report progress. That is, a promise is either fulfilled or rejected, never “50% done.” Some promise libraries1 add the very useful ability to report on progress, and it is possible that functionality will arrive in JavaScript promises in the future, but for now, we must make do without it. Which segues nicely into….
Events
Events are another old idea that’s gained traction in JavaScript. The concept of events is simple: an event emitter broadcasts events, and anyone who wishes to listen (or “subscribe”) to those events may do so. How do you subscribe to an event? A callback, of course. Creating your own event system is quite easy, but Node provides built-in support for it. If you’re working in a browser, jQuery also provides an event mechanism. To improve countdown, we’ll use Node’s EventEmitter. While it’s possible to use EventEmitter with a function like countdown, it’s designed to be used with a class. So we’ll make our countdown function into a Countdown class instead:
const EventEmitter = require('events').EventEmitter;
class Countdown extends EventEmitter {
constructor(seconds, superstitious) {
super();
this.seconds = seconds;
this.superstitious = !!superstitious;
}
go() {
const countdown = this;
return new Promise(function(resolve, reject) {
for(let i=countdown.seconds; i>=0; i--) {
setTimeout(function() {
if(countdown.superstitious && i===13)
return reject(new Error("DEFINITELY NOT COUNTING THAT"));
countdown.emit('tick', i);
if(i===0) resolve();
}, (countdown.seconds-i)*1000);
}
});
}
}
The Countdown class extends EventEmitter, which enables it to emit events. The go method is what actually starts the countdown and returns a promise. Note that inside the go method, the first thing we do is assign this to countdown. That’s because we need to use the value of thisto get the length of the countdown, and whether or not the countdown is superstitious inside the callbacks. Remember that this is a special variable, and it won’t have the same value inside a callback. So we have to save the current value of this so we can use it inside the promises.
The magic happens when we call countdown.emit('tick', i). Anyone who wants to listen for the tick event (we could have called it anything we wanted; “tick” seemed as good as anything) can do so. Let’s see how we would use this new, improved countdown:
const c = new Countdown(5);
c.on('tick', function(i) {
if(i>0) console.log(i + '...');
});
c.go()
.then(function() {
console.log('GO!');
})
.catch(function(err) {
console.error(err.message);
})
The on method of EventEmitter is what allows you to listen for an event. In this example, we provide a callback for every tick event. If that tick isn’t 0, we print it out. Then we call go, which starts the countdown. When the countdown is finished, we log GO!. We could have, of course, put the GO! inside the tick event listener, but doing it this way underscores the difference between events and promises.
What we’re left with is definitely more verbose than our original countdown function, but we’ve gained a lot of functionality. We now have complete control over how we report the ticks in the countdown, and we have a promise that’s fulfilled when the countdown is finished.
We still have one task left—we haven’t addressed the problem of a superstitious Countdown instance continuing to count down past 13, even though it’s rejected the promise:
const c = new Countdown(15, true)
.on('tick', function(i) { // note we can chain the call to 'on'
if(i>0) console.log(i + '...');
});
c.go()
.then(function() {
console.log('GO!');
})
.catch(function(err) {
console.error(err.message);
})
We still get all the ticks, all the way down to 0 (even though we don’t print it). Fixing this problem is a little involved because we have already created all the timeouts (of course, we could just “cheat” and immediately fail if a superstitious timer is created for 13 seconds or longer, but that would miss the point of the exercise). To solve this problem, once we discover we can’t continue, we’ll have to clear all of the pending timeouts:
const EventEmitter = require('events').EventEmitter;
class Countdown extends EventEmitter {
constructor(seconds, superstitious) {
super();
this.seconds = seconds;
this.superstitious = !!superstitious;
}
go() {
const countdown = this;
const timeoutIds = [];
return new Promise(function(resolve, reject) {
for(let i=countdown.seconds; i>=0; i--) {
timeoutIds.push(setTimeout(function() {
if(countdown.superstitious && i===13) {
// clear all pending timeouts
timeoutIds.forEach(clearTimeout);
return reject(new Error("DEFINITELY NOT COUNTING THAT"));
}
countdown.emit('tick', i);
if(i===0) resolve();
}, (countdown.seconds-i)*1000));
}
});
}
}
Promise Chaining
One of the advantages of promises is that they can be chained; that is, when one promise is fulfilled, you can have it immediately invoke another function that returns a promise…and so on. Let’s create a function called launch that we can chain to a countdown:
function launch() {
return new Promise(function(resolve, reject) {
console.log("Lift off!");
setTimeout(function() {
resolve("In orbit!");
}, 2*1000); // a very fast rocket indeed
});
}
It’s easy to chain this function to a countdown:
const c = new Countdown(5)
.on('tick', i => console.log(i + '...'));
c.go()
.then(launch)
.then(function(msg) {
console.log(msg);
})
.catch(function(err) {
console.error("Houston, we have a problem....");
})
One of the advantages of promise chains is that you don’t have to catch errors at every step; if there’s an error anywhere in the chain, the chain will stop and fall through to the catch handler. Go ahead and change the countdown to a 15-second superstitious countdown; you’ll find thatlaunch is never called.
Preventing Unsettled Promises
Promises can simplify your asynchronous code and protect you against the problem of callbacks being called more than once, but they don’t protect you from the problem of promises that never settle (that is, you forget to call either resolve or reject). This kind of mistake can be hard to track down because there’s no error…in a complex system, an unsettled promise may simply get lost.
One way to prevent that is to specify a timeout for promises; if the promise hasn’t settled in some reasonable amount of time, automatically reject it. Obviously, it’s up to you to know what “reasonable amount of time” is. If you have a complex algorithm that you expect to take 10 minutes to execute, don’t set a 1-second timeout.
Let’s insert an artificial failure into our launch function. Let’s say our rocket is very experimental indeed, and fails approximately half the time:
function launch() {
return new Promise(function(resolve, reject) {
if(Math.random() < 0.5) return; // rocket failure
console.log("Lift off!");
setTimeout(function() {
resolve("In orbit!");
}, 2*1000); // a very fast rocket indeed
});
}
In this example, the way we’re failing is not very responsible: we’re not calling reject, and we’re not even logging anything to the console. We just silently fail half the time. If you run this a few times, you’ll see that sometimes it works, and sometimes it doesn’t…with no error message. Clearly undesirable.
We can write a function that attaches a timeout to a promise:
function addTimeout(fn, timeout) {
if(timeout === undefined) timeout = 1000; // default timeout
return function(...args) {
return new Promise(function(resolve, reject) {
const tid = setTimeout(reject, timeout,
new Error("promise timed out"));
fn(...args)
.then(function(...args) {
clearTimeout(tid);
resolve(...args);
})
.catch(function(...args) {
clearTimeout(tid);
reject(...args);
});
});
}
}
If you’re saying “Whoa…a function that returns a function that returns a promise that calls a function that returns a promise…my head is spinning!”, I can’t blame you: to add a timeout to a promise-returning function is not trivial, and requires all of the preceding contortions. Completely understanding this function is left as an advanced reader’s exercise. Using this function, however, is quite easy: we can add a timeout to any function that returns a promise. Let’s say our very slowest rocket attains orbit in 10 seconds (isn’t future rocket technology great?), so we set a timeout of 11 seconds:
c.go()
.then(addTimeout(launch, 4*1000))
.then(function(msg) {
console.log(msg);
})
.catch(function(err) {
console.error("Houston, we have a problem: " + err.message);
});
Now our promise chain will always settle, even when the launch function behaves badly.
Generators
As already discussed in Chapter 12, generators allow two-way communication between a function and its caller. Generators are synchronous in nature, but when combined with promises, they offer a powerful technique for managing async code in JavaScript.
Let’s revisit the central difficulty with async code: it’s harder to write than synchronous code. When we tackle a problem, our minds want to approach it in a synchronous fashion: step 1, step 2, step 3, and so on. However, that can have performance consequences, which is why async exists. Wouldn’t it be nice if you could have the performance benefits of async without the additional conceptual difficulty? That’s where generators can help us.
Consider the “callback hell” example we used previously: reading three files, delaying for one minute, then writing the contents of the first three files out to a fourth file. How our human minds would like to write this is something like this pseudocode:
dataA = read contents of 'a.txt'
dataB = read contents of 'b.txt'
dataC = read contents of 'c.txt'
wait 60 seconds
write dataA + dataB + dataC to 'd.txt'
Generators enable us to write code that looks very much like this…but the functionality doesn’t come out of the box: we’ll have to do a little work first.
The first thing we need is a way to turn Node’s error-first callbacks into promises. We’ll encapsulate that into a function called nfcall (Node function call):
function nfcall(f, ...args) {
return new Promise(function(resolve, reject) {
f.call(null, ...args, function(err, ...args) {
if(err) return reject(err);
resolve(args.length<2 ? args[0] : args);
});
});
}
NOTE
This function is named after (and based on) the nfcall method in the Q promise library. If you need this functionality, you should probably use Q. It includes not only this method, but many more helpful promise-related methods as well. I present an implementation of nfcall here to demonstrate that there is no “magic.”
Now we can convert any Node-style method that takes a callback to a promise. We’ll also need setTimeout, which takes a callback…but because it predates Node, it wasn’t hip to the error-first convention. So we’ll create ptimeout (promise timeout):
function ptimeout(delay) {
return new Promise(function(resolve, reject) {
setTimeout(resolve, delay);
});
}
The next thing we’ll need is a generator runner. Recall that generators are not inherently asynchronous. But because generators allow the function to communicate to the caller, we can create a function that will manage that communication—and know how to handle asynchronous calls. We’ll create a function called grun (generator run):
function grun(g) {
const it = g();
(function iterate(val) {
const x = it.next(val);
if(!x.done) {
if(x.value instanceof Promise) {
x.value.then(iterate).catch(err => it.throw(err));
} else {
setTimeout(iterate, 0, x.value);
}
}
})();
}
NOTE
grun is based heavily on runGenerator, presented in Kyle Simpson’s excellent series of articles on generators. I highly recommend that you read those articles as a supplement to this text.
This is a very modest recursive generator runner. You pass it a generator function, and it runs it. As you learned in Chapter 6, generators that call yield will pause until next is called on their iterator. This function does so recursively. If the iterator returns a promise, it waits for the promise to be fulfilled before resuming the iterator. On the other hand, if the iterator returns a simple value, it immediately resumes the iteration. You may be wondering why we call setTimeout instead of just calling iterate directly; the reason is that we gain a little efficiency by avoiding synchronous recursion (asynchronous recursion allows the JavaScript engine to free resources more quickly).
You may be thinking “This is a lot of fuss!” and “This is supposed to simplify my life?”, but the hard part is over. nfcall allows us to adopt the past (Node error-first callback functions) to the present (promises), and grun allows us access to the future today (expected in ES7 is the awaitkeyword, which will essentially function as grun, with an even more natural syntax). So now that we’ve got the hard part out of the way, let’s see how all of this makes our life easier.
Remember our “wouldn’t it be nice” pseudocode from earlier in this chapter? Now we can realize that:
function* theFutureIsNow() {
const dataA = yield nfcall(fs.readFile, 'a.txt');
const dataB = yield nfcall(fs.readFile, 'b.txt');
const dataC = yield nfcall(fs.readFile, 'c.txt');
yield ptimeout(60*1000);
yield nfcall(fs.writeFile, 'd.txt', dataA+dataB+dataC);
}
It looks a lot better than callback hell, doesn’t it? It’s also neater than promises alone. It flows the way we think. Running it is simple:
grun(theFutureIsNow);
One Step Forward and Two Steps Back?
You might (quite reasonably) be thinking that we’ve gone to so much trouble to understand asynchronous execution, then make it easier…and now we’re right back where we started, except with the extra complication of generators and converting things to promises and grun. And there is some truth in this: in our theFutureIsNow function, we have somewhat thrown the baby out with the bathwater. It’s easier to write, easier to read, and we’re reaping some of the benefits of asynchronous execution, but not all of them. The sharp question here is: “Would it be more efficient to read the three files in parallel?” The answer to that question depends a lot on the problem, the implementation of your JavaScript engine, your operating system, and your filesystem. But let’s put aside those complexities for a moment, and recognize that it doesn’t matter what order we read the three files in, and it’s conceivable that efficiency would be gained from allowing those file read operations to happen in parallel. And this is where generator runners can lull us into a false sense of complacency: we wrote the function this way because it seemed easy and straightforward.
The problem (assuming there is a problem) is easy to solve. Promise provides a method called all, which resolves when all the promises in an array resolve…and will execute the asynchronous code in parallel if possible. All we have to do is modify our function to use Promise.all:
function* theFutureIsNow() {
const data = yield Promise.all([
nfcall(fs.readFile, 'a.txt'),
nfcall(fs.readFile, 'b.txt'),
nfcall(fs.readFile, 'c.txt'),
]);
yield ptimeout(60*1000);
yield nfcall(fs.writeFile, 'd.txt', data[0]+data[1]+data[2]);
}
The promise returned by Promise.all provides an array containing the fulfillment value of each promise in the order they appear in the array. Even though it’s possible for c.txt to be read before a.txt, data[0] will still hold the contents of a.txt, and data[1] will still hold the contents ofc.txt.
Your takeaway from this section should not be Promise.all (though that’s a handy tool to know about); your takeaway should be to consider what parts of your program can be run in parallel, and what parts can’t. In this example, it’s even possible that the timeout could be run in parallel to the file reads: it all depends on the problem you’re trying to solve. If what’s important is that the three files are read and then 60 seconds passes and then the concatenated result is written to another file, we already have what we want. On the other hand, what we may want is for the three files to be read, and in no sooner than 60 seconds, the results to be written to a fourth file—in which case, we would want to move the timeout into the Promise.all.
Don’t Write Your Own Generator Runner
While it’s a good exercise to write our own generator runner, as we have done with grun, there are many nuances and improvements we could make on it. It’s better not to reinvent the wheel. The co generator runner is full-featured and robust. If you are building websites, you may want to look into Koa, which is designed to work with co, allowing you to write web handlers using yield, as we have in theFutureIsNow.
Exception Handling in Generator Runners
Another important benefit of generator runners is that they enable exception handling with try/catch. Remember that exception handling is problematic with callbacks and promises; throwing an exception inside a callback cannot be caught from outside the callback. Generator runners, because they enable synchronous semantics while still preserving asynchronous execution, have a side benefit of working with try/catch. Let’s add a couple of exception handlers to our theFutureIsNow function:
function* theFutureIsNow() {
let data;
try {
data = yield Promise.all([
nfcall(fs.readFile, 'a.txt'),
nfcall(fs.readFile, 'b.txt'),
nfcall(fs.readFile, 'c.txt'),
]);
} catch(err) {
console.error("Unable to read one or more input files: " + err.message);
throw err;
}
yield ptimeout(60*1000);
try {
yield nfcall(fs.writeFile, 'd.txt', data[0]+data[1]+data[2]);
} catch(err) {
console.error("Unable to write output file: " + err.message);
throw err;
}
}
I’m not claiming that try...catch exception handling is inherently superior to catch handlers on promises, or error-first callbacks, but it is a well-understood mechanism for exception handling, and if you prefer synchronous semantics, then you will want to be able to use it for exception handling.
Conclusion
Fully understanding the complexities involved in asynchronous programming—and the various mechanisms that have evolved for managing it—is critical to understanding modern JavaScript development. We’ve learned:
§ Asynchronous execution in JavaScript is managed with callbacks.
§ Promises do not replace callbacks; indeed, promises require then and catch callbacks.
§ Promises eliminate the problem of a callback getting called multiple times.
§ If you need a callback to be called multiple times, consider using events (which can be combined with a promise).
§ A promise cannot guarantee that it will settle; however, you can wrap it in a timeout to protect against this.
§ Promises can be chained, enabling easy composition.
§ Promises can be combined with generator runners to enable synchronous semantics without losing the advantages of asynchronous execution.
§ When writing generator functions with synchronous semantics, you should be careful to understand what parts of your algorithm can run in parallel, and use Promise.all to run those parts.
§ You shouldn’t write your own generator runner; use co or Koa.
§ You shouldn’t write your own code to convert Node-style callbacks to promises; use Q.
§ Exception handling works with synchronous semantics, as enabled by generator runners.
If your only programming experience is with languages that have synchronous semantics, learning synchronous programming the JavaScript way can be daunting; it certainly was for me. However, it’s an essential skill in modern JavaScript projects.
1Such as Q.