Working with file data in Node.js - Node.js for .NET Developers (2015)

Node.js for .NET Developers (2015)

Chapter 7. Working with file data in Node.js

Data can be stored in several kinds of repositories. We’ve looked at databases, and we’ve designed receivers that could take in and parse data coming from anywhere—be it our own application or others that are posting or otherwise sending information according to specific route formats that we can take apart. Now let’s look at sending, receiving, accessing, and storing files and the data contained within them.

This functionality is all supportive in nature. As a part of our website project, we have no immediate need to read and write specific files, but you can use these principles to explore options. For uploading files, I’ll just assume you might want to upload images of the players to go along with the information to be displayed in their details view, but I’ll leave it to you to fetch the actual images from the web as you like. Of course, you can use any files to prove the functionality to yourself.

Fortunately, a lot of this is simple to do using Node.js. To write info to a file and save it, you need to include this line to get access to the fs object inside of Express:

var fs = require('fs');

There’s no need for another npm package. After you do that, one line of code with the writeFile method is all you need, along with a couple of variables passed to it containing the filename and the content you want to write into the file:

var sFileName = 'helloworldcool.txt'
var sMsg = 'Yo this is cool'
fs.writeFile(sFileName, sMsg, function (err) {
if (err) return console.log(err);
// do something if successful
});

Now you have a new file saved where you specified, and it contains your entry.

To read info back out from the file and display it, you just use a different method of the same base fs object. Obviously, you typically do more than this with that data, but for the moment you’ll just write the contents of the file to the console:

var sFileName = 'helloworldcool.txt' // same filename as last example
fs.readFile(sFileName, function (err, fileData) {
if (err)
throw err;
if (fileData)
console.log(fileData.toString('utf8'));
});

These functions are simple to test—just drop the code anywhere. You might have noticed already that any code in any .js file that is not contained inside a function runs immediately when the application loads and the Node.js server starts.

I’ll let you explore both the positive and negative implications of that idea. One way or the other, it’s something that is good to be aware of if you weren’t already.

To see what’s needed to transfer a file using a typical browse button, the first thing you need to do is create the button in a view somewhere:

Please specify a file or a set of files:<br>
<form method='post' action='upload' enctype="multipart/form-data">
<input type='file' name='fileUpload'>
<input type='submit'>
</form>

To make it as easy as possible, you use basic HTML tools and then do it in a particular way. Specifically, you put your upload button inside of a form and then specify a post action and a route called upload:

<input type='file' name='fileUpload'>

Because this kind of functionality might be needed from several places, and putting it in a page-specific file might make it difficult to track down, on this occasion you’ll add a route at the app.js level to handle all uploads:

server.route('/upload')
.post(function (req, res, next) {
req.pipe(req.busboy);
req.busboy.on('file', function (fieldName, oFile, fileName) {
console.log("Uploading: " + filename);
//Path where image will be uploaded
var fStream = fs.createWriteStream(__dirname + '/images/' + fileName);
oFile.pipe(fStream);
fStream.on('close', function () {
console.log("Upload Finished of " + fileName);
res.redirect('back'); //where to go next
});
});
});

Yes, it is possible to route an entire application using this technique. If you think about it for a moment, you’ll see why it can quickly lead to .js files for files like your app.js that are absurdly long. Separating each page and its functionality is one of the keys to keeping your code manageable—both for you and for the next person.

Notice in the preceding code sample that you are using the busboy npm package as well as including the fs feature of Express.

There are a couple of lines to focus on. First take a look at this line:

var fStream = fs.createWriteStream(__dirname + '/images/' + fileName);

Here you simply call createWriteStream from the fs object to create the necessary parameters for sending your file.

You can easily perform a test to see if this works, and if you like, you can fetch from the Web any noncopyrighted image of one of the players in the list of data you have. Then name it appropriately for the player and upload it into the images folder for later retrieval. When the player is selected, you can view his or her details.

The following lines, however, isolate for you a specific and useful feature of Node.js—namely, the pipe function:

req.pipe(req.busboy);
oFile.pipe(fStream);

In this case, you are using it only to send your file info. But the function has much greater value as well.

You will find that a common scenario when dealing with streaming files is that you will stream files both to and from remote resources. And often at the same time. You will thus be engaged in doing two separate but connected processes—both reading inbound data and then sending it out to a client.

No doubt, there will be times during these processes that the end client is consuming the data more slowly than you are sending it. If this likely behavior isn’t accounted for, you will end up with some serious lagging performance issues. The solution requires that you include the ability to pause, monitor, and resume data transfers.

The stream.pipe function resolves this issue under the covers for you without you having to resort to a bunch of myStreamingFile.on functions to do all that synching work. The pipe call takes a readable stream (file) and pipes it to a writeable one (file) so that they are synchronized. It’s an elegant solution to a nagging issue, and it’s just that simple.

Streams can also be used to exercise more control over the file read process. For example, when you are reading from an existing file, you don’t have to do this as you did earlier:

fs.readFile(sFileName, function (err, fileData) {

Instead, you can do the following, which is a useful technique in cases where the file is large and you might have remote-transfer issues:

var oStream = fs.createReadStream('existingFile');
var dataLength = 0;
oStream
.on('data', function (chunk) {
dataLength += chunk.length;
})
.on('end', function () { // done
console.log('The length was:', dataLength);
});

Here you are using createReadStream from the fs object and its returned object, which you are calling oStream. The oStream you get back has some characteristics of the on functionality. This includes the data version, which is an event triggered when a chunk of data of whatever size streams in. This type of information gives you highly granular visibility into and control over the file-transfer process.

The preceding code should work in most cases as it is shown. The data type for the chunk that is returned by streams that are created by Node.js core modules will usually be a buffer or a string. Both buffer and string implement the length method.

In fact, combining the two similar stream functions, createReadStream and createWriteStream, as follows is the preferred way of doing a file copy in the land of Node.js:

fs.createReadStream("input/" + fileName )
// Write File
.pipe(fs.createWriteStream("output/" + fileName));

No other utility exists for doing this.

In the end function, you just record what the total size of the file was so that, assuming you knew the file size to start with, you can make sure you got all of it. To get the size of the file, you need to return to the fs object this way:

var stats = fs.statSync("myfile.txt")
var fileSizeInBytes = stats["size"]

Here you are using the statSync function to return an array of file metadata. One of the items in that array can be specified using the key size, which will give you the size of the array in bytes. That is the same measurement you will get from the length function of a buffer or string.

For processing internal text data, JavaScript contains a number of useful functions. For example, it contains split, which takes a character delimiter by which to divide a string and returns an array of the individual values in the string:

var sName = "Michael Jeffery Jordan!";
var arrName = sName.split(" ");
console.log(arrName); // produces { Michael, Jeffery, Jordan! }

There is also the substring function, which is used to get a piece of a string. You simply pass it the index start and end values:

var sSub = sName.substring(1, 6);
console.log(sSub); // produces "ichae"

Note that the character at the start index is included in the returned string but the character at the end index is not.

For more sophisticated searching, there is the match function, which returns a string or an array of strings that matches a Regular Expression pattern:

var sName = "Jeff Jeffery Jefferson";
var res1 = sName.match(/Jeff/); // produces { Jeff }
var resAll = sName.match(/Jeff/g); // produces { Jeff, Jeff, Jeff }

The difference in the code just shown is that the global argument /g is specified in the second match line. Without this change, the function will return only the first instance of the matching RegEx pattern provided.

Unless you are intentionally meaning to be case-sensitive, such as with a password during login, typically the best way to clean strings for processing is by using the function toUpperCase, for which there is also a matching toLowerCase. Use of one of these on both sides of your equation will assure that any minor capitalization variations in otherwise identical data will be ignored, thus producing more (typically correct) matches. There is also a trim function that removes all leading and trailing spaces.

Each of these functions is similar to what you will find in the .NET library and will be potentially useful to you as you process information using Node.js and JavaScript.