Working with site data in Node.js - Node.js for .NET Developers (2015)

Node.js for .NET Developers (2015)

Chapter 5. Working with site data in Node.js

Data comes in many shapes and sizes. It can come from within your application as it works, from users, or from outside data stores. As with the rest of Node.js, there is an npm package to deal individually with each of the myriad ways data might be provided to your application.

In general, this data can be broken into three major categories, with each having a couple of primary ways that data would likely come to you:

Image Data from URLs

• Route/Path pattern

QueryString

Image Data from users

• Form posts

• Field input

Image Data from external sources

• Databases (covered in Chapter 6, “Working with external data in Node.js”)

• File systems (covered in Chapter 7, “Working with file data in Node.js”)

We’ll walk through these and incorporate them into your Node.js application.

Data from URLs

The first way you’ll have to deal with data in the application relates to data that comes in the URL. Even our simple example code for rendering an array that isn’t connected to anything contains within it the idea that we will drill down into some detail about some item on the list. And that item is specified within the URL.

In this case, as shown, we opted to build a path that looks like this:

href= 'details/<%= players[i].id%>'>

This path yields the following result or similar as the URL path:

'details/5'

You could keep going and add as many path arguments as you like after the original. Each argument is given a name when you declare the path, like this:

router.get('/details/:id')

Or by adding to it, like this:

router.get('/details/:id/:name')

Then, in code, these arguments are accessed through the param collection provided by Express:

req.param("id")

Your get will look like this:

router.get('/details/:id', function (req, res) {
res.render('details', {
playerId: req.param("id")
});
});

When you create your player and then set up the details.js/.ejs view, you’ll have a playerId argument to get your hands on and display right away to make sure you have the correct data.

Always remember, as I mentioned earlier, that you must order your code to go from most specific in the path to least specific because the engine will render the first matching pattern that it finds. The warning here is this URL coming in as a web request:

http://127.0.0.1:1234/details/1/Payton

This will match the following route:

router.get('/details')

And it will match this route as well:

router.get('/details/:id')

And this route:

router.get('/details/:id/:name')

The second and third routes will never be reached if these are in the wrong order in your code file. To be properly done, these route entries need to be exactly reversed from what is shown here.

Pulling arguments from structured route paths is one way to pass and pull data from the URL. As an alternative, you could have placed the data into a QueryString. In the real world, this is equally as likely to be the pattern you choose to follow for building URLs internally to pass data.

You use the QueryString collection to access the URL’s name/value pairs either by index or by name. In general, it will be just as easy to code a solution that parses its data as with the param collection. That being said, it is your choice, and each option works at least equally well.

To implement access to the QueryString, simply reference its collection instead of the param collection:

req.query.ID

In this case, as you see, you can actually use dot notation to access the individually named members of the collection you specified. The processing engine recognizes the question mark (?) as the beginning of the collection and the ampersand (&) as the argument separator. Thus, the route itself is still the same as the base route and the previously mentioned issues with route order in the file are not relevant.

So to process this URL:

http://127.0.0.1:1234/details?ID=1&Name=Payton

the route get function to render this data in all cases would simply be

router.get('/details', function (req, res) {
res.render('details', {
playerId: req.query.ID,
name: req.query.Name
});
});

As you can see, you simply take apart the arguments by name one at a time to get to the values contained in them. Passing an entire object this way would be done by manually taking apart the object properties to provide the necessary arguments in your assembled link to the details page:

href= 'details?ID=<%= players[i].id%>&Name=<%= players[i].lastName%>'>

Continue on like this in as much depth as required. In real-world practice, this approach is rarely needed for sending information to your own .ejs files. This is because, as you have seen, you can pass entire objects or even collections of objects in this way:

res.render('survey', {
players: arrPlayers
});

For connecting to external resources and assembling a QueryString or a route, or for taking in connections to your resources from others and thus parsing an inbound QueryString or route, working with data directly inside the URL is often your only option for moving that data from place to place.

Data from users

Another technique that should be familiar to you if your background is classic ASP or MVC is form posting. With this approach, it is assumed that there is a screen into which a user is entering one or more fields and that the entire collection of those values needs to be quickly and easily transported to a URL. To implement form posting, instead of using a get function inside of your .js file, you need to use a post:

router.post('/survey', function (req, res) { });

As you can see, post is almost identical in signature to get. Before we get more deeply into it, just touching on the subject of post as an alternative to get leads us back into taking a brief look at the four basic actions universally available over an HTTP web connection:

Image Get A typical web request

Image Post Usually used for sending a collection of data to be processed

Image Put Usually used for updating a single record

Image Delete As you might have expected, usually used for deleting a single record

A file that contains routing information will typically contain at least one of each of the four methods just defined, like this:

router.get('/survey', function (req, res) { });

router.post('/survey', function (req, res) { });

router.put('/survey', function (req, res) { });

router.delete('/survey', function (req, res) { });

As you have seen, often more than one get function exists within this collection. Within the post will be the code to get the name/value pairs from the inbound form collection. Once again, the key to this is having the proper npm package installed. In this case, it is body-parser. So, in your app.js file, make sure you have this line:

var bodyParser = require('body-parser');

You need the variable because you also need to place this line in the same file after the bodyParser declaration just shown to properly format the inbound input:

server.use(bodyParser.urlencoded({ extended: true }));

With those pieces in place, from inside your post function, you’ll be able to access the body property of an inbound request by doing this to get your hands on the value you seek:

var sInput = req.body.txtInbound;

In this case, you are looking for a control called txtInbound.

To see this is action, you need to add a few things to your HTML/EJS file to activate a form post. Let’s start with a button and a textbox. Just to do this demonstration, go ahead and drop a couple of input controls in a separate row below the list control you have in the page:

<table class="table">
<tr>
<td class="center">
. . .
</td>
</tr>
<tr>
<td>
<form action="/survey" method="post">
<input id="txtInbound" name="txtInbound" type="text" />
<input type="submit" />
</form>
</td>
</tr>
</table>

Notice how you have wrapped your input controls in a form and then specified two important attributes—method=’post’ and action=’/survey’—to tell the form how you want it to behave. When you submit it, you want the form to post its information to the path indicated in the action—in this case, to your survey page.

With all of this wiring in place, let’s turn our attention back to the actual post function inside your survey.js file to have it respond to your successful form post. Again, we’ll do something more useful with this later. Just to see it work, let’s have it write any input value to the console:

router.post('/survey', function (req, res) {
var sInput = req.body.txtInbound;
console.log(sInput);
res.send('posted');
});

At this point, you can pop the page open in a browser. You should see your button control and your text box. Enter any value, and click Submit. You should see the value you typed appear in the console window. Don’t forget to respond with something; otherwise, your application will stop responding even if the code works as expected.

Typically, this is where you have your CRUD (Create, Update, Delete) interactions with an external source such as a database. When you return to this code in the next chapter, you’ll be taking that value and inserting it into a Microsoft SQL Server database.

Now you’ve seen the Node.js versions of standard get and post operations.

However, aside from these basic, good old-fashioned web techniques for moving bits of data from here to there, inside your Node.js application you do have other options. One of the best of these is the cache.

To use a cache in Node.js, all you need is the proper npm package. You already installed memory-cache when you set up our application so now you just have to do the usual to enable its use:

var cache = require('memory-cache');

This component works just as you would hope that it would, similar to the .NET cache but without some of the features. To put a value into the cache, you simply do this:

cache.put('players', arrPlayers);

And to retrieve that value, this is all it takes:

cache.get('players');

This caching component also has an expiration argument, expressed in milliseconds:

cache.put('players', arrPlayers, 5000);

After the time elapses, in this case five seconds, the item will be removed from the cache if it is not renewed.

One of the most important things to notice here is that my example for storing something in the cache uses not just an object but an array of objects. The cache will hold anything you can concoct in JavaScript, and this idea opens the door to the true power of Object-Oriented JavaScript (OOJS) and sophisticated state management that’s required in commercial applications. It’s a pattern used in .NET to take advantage of the full power of the programming model.

The Node.js cache does for JavaScript objects what the .NET cache does for business-layer objects—it makes them universally accessible. There are two kinds of objects typically stored in the cache: lookup objects and user objects.

A lookup object is typically a collection of values that rarely, if ever, changes—something like US state names or US president names. A list of states stored in the database has to be fetched only one time, and then it can be held in memory after that to allow for quicker access from the application. This works quickly and easily because there is no concern that data will get out of sync—in other words, there are no worries that new data will be entered into the database and the cached version of the data will be out of date. With data that never changes, such as US states, that problem is not a problem. A routine that re-created that data once every four or eight years is also not a big issue.

Of course, this design also works for lookup data that changes more regularly. You simply have to account for those changes in memory as well as in the database—for example, by updating the collection in memory on the same button click that allows for a database update. This is one way to greatly improve the performance of your application.

In general, interaction with the database should be avoided except in cases where it simply can’t be, such as for CRUD operations. Most other functions that people typically perform in the database, such as filtering and joining, can be done much more quickly by using server memory.

Picture a software application as the city of San Francisco, and imagine the only four bridges over the bay represent the database. No matter how many creative ways you navigate to one of those bridges, you’ll be slammed with all the rest of the city traffic doing the same. Those are the only routes. So everyone has to use them. If you keep your database interactive operations to the bare minimum required, traffic will flow better all over your “city.” That’s the whole idea behind using as much cached data as you possibly can.

A user object holds all the information specific to a site user. That can be data from the system itself, such as profile information containing permissions, and it can be state data that is tracking current user behavior. Both kinds of information will typically be required all throughout the application, and the cache is the perfect tool to use for allowing it.

Only one difference is required in the way you manage the cache. For a lookup object, this will work:

cache.put('leagues', arrLeagues);

However, for a user-specific object, you need an identifier that ties that specific object to that and only that specific user. The standard technique for doing so is to create a globally unique identifier (GUID) that you associate with that user, most often on login. Then you simply prepend the GUID to your cache entry like this:

cache.put(GUID + 'User', myUserObj);

You should have that GUID for that user included in the QueryString on every request, like this:

http://127.0.0.1:1234/details?GUID=1QW3456tr4RTYUXXYiujjii45UY89898TRFReded

That way, you can then pull it out to get access to your user object in the cache, like this:

var sGUID = req.query.GUID;
var myObj = cache.get(sGUID + 'User');

You have a rock-solid, state management strategy in place that works for every page of your application, with code consistency, in exactly the same way.

As I mentioned, this caching technique is the only truly viable solution for all web scenarios, even in the world of .NET. If you don’t believe it, try to pass a Session variable across a protocol change—that is, take one of your Session values and pass it from HTTP to HTTPS. Good luck! There’s no way that coding technique will ever work. Sessions do not cross protocol boundaries. You can, and want to, create Session equivalents using the login GUIDs, caching, and OOP, but that’s not nearly the same thing as using the Session object.

You can even take the idea one step further for web-farm scenarios by serializing the data in your objects to external data stores. Serialization turns the state of an object into text. So you serialize to store the data and deserialize to retrieve it. When a request comes in, you check the cache in that specific server for the GUID-related user object. If it isn’t there, you pull the user state from the external store according to the GUID in the QueryString and reassemble it into objects right there. And then you are back to normal code operations. One technique, all scenarios, infinitely scalable.

Now you use a tech interview question to separate the wheat from the chaff—no .NET developer worth his salt will ever go near Session. Like form posting, it’s technology from the 1990s, and .NET gave you much better ways to do the same things starting in this century, and it still does. By being well-versed in those best practices, you’re fully prepared to implement the same architecture in Node.js.

At this point, you’re effectively moving your data from page to page. Next let’s connect to some external data and see what you can do with that.