Responsive Web Design, Part 2 (2015)
Beyond Responsive: Optimizing For Offline
Beyond Service Workers: Web Storage
Service workers are very powerful, yet it might not be the right tool for every offline use case. Until recently, the only way to maintain a user’s data between visits to your site has been to store it on the server or use cookies in the browser. With Web Storage — a simple, in-browser database — we can get rid of much of the need for cookies and dramatically reduce the need for server-side functionality. Google Search, Bing, and other high-traffic sites also use it for caching on the client.
SERVER-SIDE DATA
Storing data on the server requires the creation and management of user accounts, sanitizing data sent to the server, worrying about server-side security risks and about security in the transmission of data between the client and the server. For many applications, storing data on the server is required, but in many other cases simply keeping data for the client locally during a session or between sessions, without the need to send it back and forward to the server, means a lot less development work and potentially fewer vectors for security breaches. On top of that, if our site can work offline using AppCache, then even when we need to send data to the server, if the client is offline or the server is down, we can store this locally and then synchronize once the client reconnects or the server comes back online.
WHAT ABOUT COOKIES?
Cookies, while long used to keep data on the client during and between sessions, were actually designed for communication between browser and server that persists between sessions, so that the server could keep track of the state of previous interactions with this client (technically, they’re called HTTP cookies). They’re typically used for identifying a user on return visits and storing details about that user (such as if they’re still logged in). Cookies are sent between the browser and server in unencrypted plain text each time the user opens a page. Unless an application encrypts cookie contents it’s quite trivial to read them, particularly on public Wi-Fi networks when used over standard HTTP (though much less easily over encrypted HTTPS).
Storing all client data on the server creates usability issues as well, as users need to be logged in each time they use that site. The heavy lifting of ensuring data is secure during transmission and on the server is left to you as the developer. The round-trip between browser and server will affect the performance of your site or application, and it’s rather tricky to build apps which work when the user is offline if the user’s data is all stored on the server.
For all these reasons, as web applications become increasingly sophisticated developers need ways to keep data around in the browser (particularly if we want our applications to work when the user is offline). And we want this data to be secure.
Two closely related technologies exist to help keep track of information entirely in the browser. Together known as Web Storage, they allow us to store far more structured data than cookies, are much easier to develop with than cookies, and the information stored can only be transmitted to a server explicitly by the application.
•Session storage stores data during a session and is deleted by the browser once a session is finished.
•Local storage is almost identical, but the data stored persists indefinitely, until removed by the application.
Let’s start with session storage, keeping in mind that we use local storage almost identically.
SESSION STORAGE
WHAT IS A SESSION?
The key feature of session storage is that data only persists for a session. But just what is a session? HTML5 has the concept of a top-level browsing context; this is, in essence, a browser window or tab. A session lasts for that top-level browsing context while it is open and while that top-level browsing context is pointed at the same fully qualified12 domain (or, strictly speaking, the same origin). The user can visit different URLs within the same the domain, visit a different domain and then return to the original domain, and they would still be in the same session.
During the session, a user may visit other pages of the same domain or other sites entirely, then return to the original domain. Any data saved in session storage during that session will remain available, but only to pages in the original domain, and only until the tab or window is closed.
If the user opens a link to your site in another tab or window, then the new tab or window has no access to this session storage since this new tab or window is a new session. That window will have its own, entirely separate session storage for the particular domain.
It’s worth noting that session storage is also shared with webpages inside subframes in the same domain as the top-level document in the window.
So, just to clarify, if we:
•visit http://webdirections.org in a tab and save data to session storage
•then follow a link to http://westciv.com in this same tab
•and then come back to http://webdirections.org still in the same tab
we return to the same session for http://webdirections.org, and the data in the original session storage is still available.
However, if we:
•visit http://webdirections.org in a tab and save data to session storage
•then follow a link to http://webdirections.org in a new tab or window
the data in the original session storage is not available to this new tab.
The one exception to this is when a browser crashes and is restarted. Typically, in this case, browsers will reopen all the windows that were open when the browser crashed. In this situation, the specification allows session storage to persist for reopened windows from before the crash (Safari, Blink, Chrome, Firefox and Opera browsers support this; IE8 does not, though IE9 and up do).
Which may sound like a great boon for the user, but as an application developer you might wish to consider whether you want session data to persist after a crash. A user might assume their login details were purged when their browser crashed while using a service like webmail or online banking at an internet café or another public computer; but if they were stored in session storage then the next user to launch the browser will resume the session that was current when the browser crashed. Ruh-roh.
What could we do about this?
Well, when a document loads we get a load event. Why not have an event handler that deletes the current session storage when this event fires?
window.addEventListener("load", clearSessionStorage, false);
function clearSessionStorage(){
window.sessionStorage.clear();
}
We’ll look more at the clear method of sessionStorage shortly.
WHAT GOOD IS SESSION STORAGE?
One very useful application would be to maintain sensitive information during a transaction, sign-up, sign-in and so on, which will be purged as soon as the user closes the window or tab.
It can be used to create a multi-page form or application, where the information in each page can persist and then be sent to the server all at once when the transaction is complete. It also moves some of the heavy lifting for protecting sensitive data away from application developers to the browser developer.
Applications, like an email reader, could use it to keep local copies of emails, which will be automatically purged as soon as the user closes the window or tab.
USING SESSIONSTORAGE
sessionStorage is a property of the window object in the DOM. Because it is as yet not universally supported, we need to check that this property exists before we use it:
if('sessionStorage' in window) {
//we use sessionStorage
}
else {
//we do something else, perhaps use cookies, or another fallback
}
Right, now we have our sessionStorage object, how do we use it?
KEY-VALUE PAIRS
sessionStorage stores key–value pairs. Each pair is a piece of information (the value) identified by a unique identifier (the key). Both the key and the value are strings (more on the implications of this in a moment).
We use the setItem method of the sessionStorage object to store data like so:
//Get the value of the input with id="name"
var name = document.querySelector('#name').value;
//Store this value with the key "name"
window.sessionStorage.setItem('name', name);
Now we’ve stored the value of the input “name” in an item of the sessionStorage object also called ‘name’. It will remain there until this window or tab is closed, and it will then automatically be purged by the browser when the user closes that window or tab13.
Notice that we haven’t had to create a sessionStorage object, initialize it, or even create an item. Where supported, sessionStorage is waiting there, ready for us to use. And simply setting an item using setItem creates that item if it doesn’t exist.
READING FROM SESSIONSTORAGE
There’s not much point in storing these details if we can’t get them back at some point. We do this by using the function getItem of the sessionStorage object, using a single parameter, the key we used to set the item.
So, to get the value of the item with the key “name”, we’d use:
var name = window.sessionStorage.getItem('name');
NONEXISTENT ITEMS
Now, what happens if for some reason there’s no item in sessionStorage with the key we are trying to access? In place of a string value, it returns null, not the empty string. So, it’s worthwhile testing whether the result returned is not null before using it:
var savedEmail = window.sessionStorage.getItem('email');
if(savedEmail !== null){
document.querySelector('#email').value = savedEmail;
}
SAVING DATA BETWEEN SESSIONS
When information is less sensitive, it often makes sense to store it between sessions. Particularly as websites become more application-like and can increasingly work offline, saving preferences or the state of a document can make for much better usability. For these situations, we have local storage. In almost every way identical to session storage, the key differences are that:
•where the contents of a particular session storage are only available to the window or tab in which they were saved — and only for the fully qualified domain in which they were saved — with local storage, any window or tab at the same fully qualified domain can access the local storage for that domain.
•the data stored in local storage persists between sessions
Best of all, using local storage for persistence between sessions is almost identical to using session storage.
USING LOCALSTORAGE
As we’ve mentioned, all the methods of localStorage are the same as the methods of sessionStorage:
•we set items with setItem
•we get items with getItem
But let’s look at some further features of both sessionStorage and localStorage.
LOCALSTORAGE.REMOVEITEM()
Because items in localStorage will otherwise persist forever, there are times we may want to delete them. We can do this with localStorage.removeItem(key), using the key for the item we want to remove. We can also use this with sessionStorage, but since that is purged completely when the user closes the window or tab, we’re less likely to want to do that.
LOCALSTORAGE.CLEAR()
If we want to delete the entire localStorage, we can use localStorage.clear(). But be warned: anything your app has saved to localStorage for this user is gone for good. We saw a little earlier that sessionStorage too has a clear method, which we used on page load to ensure that if the browser has crashed, the sessionStorage isn’t restored. We won’t necessarily want to do that, but if there’s sensitive information the user might assume is deleted when the browser crashes, you may want to do this.
LOCALSTORAGE.KEY()
As we saw, we access localStorage and sessionStorage with keys, which are strings. Web Storage provides a way of getting the keys, using the key() method. This takes an integer argument and returns the associated key value. For example, let’s suppose we did this:
window.localStorage.setItem(“title”, “Mr”);
window.localStorage.setItem(“name”, “John”);
window.localStorage.setItem(“familyName”, “Allsopp”);
Then we ask for the window.localStorage.key(2), we’ll get “familyName” (remember, indexes to arrays in JavaScript are zero-based). What good is this? Well, combined with the length property, which we’ll see just below, we can now iterate over all the items in localStorage or sessionStorage.
LOCALSTORAGE.LENGTH()
We can determine how many items sessionStorage or localStorage is currently storing using the length property. We could then use this, along with the key method, to iterate over the items in the storage. Here, we’ll get every item in the localStorage and add it to an array. I’m not saying you’re going to need to do this very often, though localStorage and sessionStorage are synchronous, and we can’t be sure some or all of the items aren’t stored on disk, so working with them may be slow. This is one way of moving them into memory before working on them.
var currentKey;
var currentItem;
var allItems = [];
for (var i=0; i < window.localStorage.length; i++) {
currentKey = window.localStorage.key(i);
currentItem = window.localStorage.getItem(currentKey);
allItems.push({key: currentKey, item: currentItem});
};
GOTCHAS, TIPS, AND TRICKS
While Web Storage is not as burdened with gotchas as AppCache. There are a number of quirks and issues you’ll need to be aware of to work most effectively with it. Let’s take a look at some of these.
SESSIONSTORAGE AND LOCALSTORAGE STORE ALL DATA AS STRINGS
As mentioned earlier, the values stored in localStorage and sessionStorage are strings, which has some implications for developers.
Among other things, when we store Boolean values, integers, floating point numbers, dates, objects and other non-string values, we need to convert to and from a string when writing to and reading from storage. Perhaps the most effective way of doing this is to use the JSON format.
JSON AND LOCALSTORAGE
As we’ve just seen, when working with non-string values, we’ll need to convert them to strings if we want to store them in local storage or session storage; we’ll need to convert them back from strings to their original format when we get them out of storage. The most straightforward way to do this is to use JavaScript’s JSON object to convert to and from a string value.
If you’re not familiar with it, JSON (JavaScript Object Notation) is a format for representing JavaScript values (numbers, booleans, arrays and objects) as strings. The standard JavaScript JSON object can convert to and from JSON strings and JavaScript values.
The JSON object has two methods:
•JSON.stringify(), which converts a JavaScript value to a JSON formatted string. You may be wondering why it’s not JSON.toString. In JavaScript, all objects have a toString method which returns the string representation of the object itself. In this case, we don’t want the string representation of the JSON object, which is what JSON.toString would give us.
•JSON.parse(), which takes a string and recreates the object, array or other value that this string represents (provided the string is valid JSON)
When saving any non-string value to localStorage, we’ll need to convert it to JSON; and when reading from localStorage, we’ll need to parse it back from the JSON-formatted string. Something like this:
var person = JSON.parse(window.localStorage.getItem(“john”));
window.localStorage.setItem(“john”, JSON.stringify(person));
There’s also a more subtle side effect of storing values as strings. JavaScript strings are 16-bit, so each character, even an ASCII character, is two bytes (in UTF-8, characters are one byte). This effectively halves the available storage space.
LOCAL STORAGE AND PRIVACY SETTINGS
While we know local storage is a different technology from cookies, browsers largely treat them as the same from a user’s privacy perspective. Where a user chooses to prevent a site from storing cookies, attempts to access local storage for that site (both writing and reading previously saved data) will report a security error.
Even if the user has a privacy setting that blocks the use of local storage, the window.localStorage will still exist, and there’s no method or property of the localStorage object that allows us to determine whether this is the case. But we can test for whether local storage is available by attempting to set an item, and catching any exceptions.
function storageEnabled(){
//Are cookies enabled? try setting an item to see if we get an error
try {
window.localStorage.setItem("test", "t");
return true
}
catch (exception) {
//It's possible we're out of space, but it's only 1 byte,
//So much more likely it's a security error
//Most browsers report an error of 18 if you want to check
return false
}
}
If the European Cookies Law14 (the EU e-Privacy Directive) applies to sites you build, be mindful that the law also applies to HTML5 Web Storage.
PRIVATE BROWSING
Many browsers now have private (or incognito) browsing modes, where no history or other details are stored between sessions. In this situation, what happens with session storage and local storage varies widely by browser.
•Safari returns null for any item set using localStorage.setItem either before or during the private browsing session. In essence, neither session storage nor local storage is available in private browsing mode. Safari throws the same error it gives when it exceeds the limit for a domain — QUOTA_EXCEEDED_ERR (see below for more) — rather than a security error, as it does when cookies are disabled.
•Chrome and Opera return items set prior to the start of private browsing, but once private browsing begins they treat local storage like session storage: only items set on the local storage by that session will be returned; and like local storage for other private windows and tabs.
•Firefox, like Chrome, will not retrieve items set on localStorage prior to a private session starting; but in private browsing treats local storage like session storage for non-private windows and tabs, and like local storage for other private windows and tabs.
GETTERS AND SETTERS
In addition to using getItem and setItem, we can use a key directly to get and set an item in sessionStorage and localStorage, like so (where the key here is “familyName”):
var itemValue = window.localStorage.familyName;
window.localStorage.familyName = itemValue;
If we want to set or get an item using a key value calculated within the program itself, we can do so using array notation and the key name. The equivalent to the example above would be:
var keyname = "familyName"
var itemValue = window.localStorage[keyname];
window.localStorage[keyname] = itemValue;
LOCAL STORAGE AND SESSION STORAGE LIMITS
The Web Storage specification recommends browsers implement a limit on the amount of data local storage or session storage can save for a given domain. If you try to exceed the limit that various browsers have in place (for some browsers, users can change this allowance), setItem throws an error. There’s no way of asking localStorage for the amount of space remaining, so it’s best to set item values within a try and catch for any error:
try {
window.localStorage.setItem(key, value);
}
catch (exception) {
//Test if this is a QUOTA_EXCEEDED_ERR
}
If the available space for this localStorage is exceeded, the exception object will have the name QUOTA_EXCEEDED_ERR and an error code of 22.
As mentioned, strings are 16-bit in JavaScript, which means that each and every one-byte character is two bytes. On the web we typically use UTF-8 encoding, a one-byte encoding; when saving the string “John” (four bytes in UTF-8), we are actually storing eight bytes. This effectively halves the available storage space.
Currently, major browsers have the following limits per domain on Web Storage15. Note that these are the sizes in bytes, and so the numbers of characters you can store uncompressed are half these:
•Chrome: 5MB
•Firefox: local storage 5MB; session storage unlimited
•Opera: 5MB
•Safari iOS: 5MB
•Internet Explorer: 10MB
•Android: local storage 5MB; session storage unlimited
•Safari: local storage 5MB; session storage unlimited
If the storage needs of your application are likely to exceed 5MB, then web databases are likely a better solution. However, the situation with web databases is complicated, with two different standards. One, Web SQL, is widely supported but deprecated; the other, IndexedDB, is currently supported in Firefox, Chrome, Opera, Android 4.4+ and IE10. When iOS8 and Safari 8 for Mac OS X are released, IndexedDB will be supported in all major browsers and on all major platforms. We’ll look at these in a little more detail shortly.
STORAGE EVENTS
One of the features of local storage is that the same database can be shared between multiple open tabs or windows; which also raises the issue of how these different top-level browsing contexts (the technical term for a window or tab in HTML5) can keep data synchronized. Here’s where storage events come into play. When localStorage changes, a storageChanged event is sent to the other windows and tabs open for that domain (there’s a reason for the emphasis).
We can create an event handler so that when a storage object has been changed we can be notified and respond to those changes.
window.addEventListener('storage', storageChanged, false);
Now, when localStorage is changed (by setting a new item, deleting an item or changing an existing item) our function storageChanged(event) will be called. The event passed as a parameter to this function has a property, storageArea, which is the window’s localStorage object. (Note this doesn’t work for sessionStorage because sessionStorage is restricted to a single window or tab.) What other information do we get in our event handler? The event has these storage-specific properties:
•key: the key of the item changed
•oldValue: the value changed from
•newValue: the value changed to
•url: the URL of the page whose localStorage was changed
There are two things to be aware of with storage events.
•The event only fires if the storage is changed, not if it is simply accessed and not if we set an item to the same value that it currently has.
•In the specification, the event is not received in the window or tab where the change occurred, only in other open windows and tabs that have access to this localStorage. Some browsers have implemented storage events in such a way that the event is also received by the window or tab that causes the change, but you shouldn’t rely on this.
While it may be useful to know a stored value has been changed if the user has two or more tabs or windows open for your site or app, storage events can be more useful than that. We can use them to very simply pass messages between different open windows that are pointed to the same domain. Now, you might be thinking that we already have postMessage for this very purpose, but here we can kill two birds with one stone — persist the state of an application in local storage, as well as pass a message to other windows open at the domain about the state change. Another reason this is superior to postMessage is that unlike with postMessage we don’t have to know about the existence of other windows to send them messages.
How might we use storage events? Well, suppose the user logs out of our app in one window but has other windows open for the app. We could listen for changes to localStorage and then log the user out in all open windows of the app.
Here’s how we might listen to whether the user is signed in or out of our service in our storage event handler. We’ll use an item with the key “status” to save the current status. To make things simpler (so we don’t need to convert to and from a Boolean value), we’ll use a string value (“signed in”) when the user is signed in.
function storageChanged(storageEvent) {
if(storageEvent.key === "status"&& storageEvent.newValue === "signed in")
{
//The user just signed in
}
else if (storageEvent.key === "status"){
//The user just signed out
}
}
WEB STORAGE PERFORMANCE CONCERNS
Quite often developers tend to avoid local storage because of its performance shortcomings. The key criticism relates to the fact that Web Storage is synchronous. This means a script using sessionStorage or localStorage waits while getItem, setItem and other storage methods are invoked. In theory, this can affect both the browser’s response to user input and execution of JavaScript in a page. In practice, I’d argue that this is not likely to be a significant problem in most cases.
To consider these concerns, I conducted tests across a number of devices and browsers which demonstrate that even for poorly implemented code that performs a very significant number of getItem and setItem operations, the performance of Web Storage is unlikely to have a significant impact. Yes, if you are writing hundreds of large (10s or 100s of KB of data per access) to local storage frequently, it may not be the ideal solution. But in most situations for which Web Storage was designed, I’d suggest it’s going to be adequate.
ORIGIN RESTRICTIONS
We said earlier that that session storage and local storage are restricted to windows or tabs in the same domain, but, in fact, the restriction is tighter than simply the top-level domain (such as webdirections.org).
To have access to one another’s Web Storage, tabs or windows must have the same fully qualified domain; that is, top-level domains (for example webdirections.org), subdomains (for example test.webdirections.org); and protocol (https://webdirections.org has a different local storage from http://webdirections.org).
At first glance, this might seem overly restrictive but imagine john.wordpress.org having access to the local storage of james.wordpress.org?
BROWSER SUPPORT
Web Storage is supported in all versions of Internet Explorer since IE8, as well as Firefox, Chrome and Safari for many versions, and on Safari for iOS and the stock Android browser for many versions as well. The challenge for backwards compatibility is essentially limited to IE7 and earlier.
For browsers that don’t support Web Storage there are several polyfills16, which provide support for the local storage API in these browsers.
JSON is supported natively in all browsers that support local storage.
Web Storage solves a long-standing challenge for web developers: reliably and more securely storing data between sessions entirely on the client-side. While there are assertions that performance limitations make local storage harmful, in the real world, services like Google and Bing are using local storage, and performance experts like Steve Souders and Nicholas Zakas defend and advocate their use. That’s not to say Web Storage is perfect or ideal in all situations. The synchronous nature of the API and potential limits per origin do mean that in certain circumstances an alternative may be required. Web Storage is, however, eminently usable for a great many client-side data storage needs.
While Web Storage is a good solution for when we are storing a relatively small amount of simple data, for more complex situations it’s not ideal. When a more sophisticated database solution is required, we can use IndexedDB and Web SQL, but an introduction to them is beyond the scope of this chapter.