Data Push Apps with HTML5 SSE (2014)
Chapter 8. More SSE: The Rest of the Standard
The SSE standard contains a few other features that I have glossed over in this book, and this chapter will take a look at them. These other features of SSE have been ignored for a couple of reasons. First, I didn’t need them! By making the decision to always pass around one JSON object per line, and having the JSON object be self-descriptive, the event and multiline features were never needed. The second reason is that it would have made the fallbacks slower and more complicated. By being pragmatic and not trying to create a perfect polyfill of SSE, we could allow our fallbacks to use the protocol that best suited them. But it is good to know about them, and this chapter will introduce each feature, show when you might want to use it, and even give some hints as to how to implement it in our fallbacks.
Headers
Here is a simple script (found as log_headers.html in the book’s source code):
<html>
<head>
<title>Logging test</title>
</head>
<body>
<script>
var es = new EventSource("log_headers.php");
</script>
</body>
</html>
This goes to show just how small an SSE script can be. Of course, it does absolutely nothing on the frontend. Here is the corresponding backend script:
<?php
$SSE = (@$_SERVER["HTTP_ACCEPT"] == "text/event-stream");
if($SSE)
header("Content-Type: text/event-stream");
else
header("Content-Type: text/plain");
file_put_contents("tmp.log", print_r($_SERVER, true) );
?>
This is also embarrassingly short. It simply writes everything it finds in the superglobal $_SERVER to a file called tmp.log. This includes the HTTP headers that the browser sent to the server, which is usually what we are interested in. tmp.log only shows the most recent request; it is overwritten each time. Try it out with each of your target browsers.
NOTE
If tmp.log does not get created when accessed through a web server, it is probably write permissions. On a Unix system, run touch tmp.log then chmod 666 tmp.log. Then try again.
I wanted to show this one first, because you can take that file_put_contents("tmp.log",print_r($_SERVER,true)); line and put it at the top of any script that you want to troubleshoot, or just understand.
If you want to also see the contents of COOKIES, POST and all the other superglobals, it is trivial to add them, too. However, even better is to show the output of phpinfo(), an excerpt of which is shown in Figure 8-1. I will not show the script here because it is quite PHP-specific, but take a look at show_phpinfo.php if you are curious.
The show_phpinfo.php script grabs phpinfo() output (which is in HTML), does a little formatting, then outputs it as an SSE block. It wraps it in a JSON string to make sure the line breaks don’t cause problems. (The code also works with the XHR and long-poll fallbacks, and also includes some of the headers we look at in this chapter, to make it more generally useful.) Here is what the frontend looks like:
<html>
<head>
<title>PHPInfo Test</title>
</head>
<body>
<div id="x">(loading...)</div>
<script>
var es = new EventSource("show_phpinfo.php");
es.addEventListener("message", function(e){
var s = JSON.parse(e.data);
document.getElementById("x").innerHTML = s;
},false);
</script>
</body>
</html>
Figure 8-1. Sample output of show_phpinfo.html
If you see “(loading…)” and nothing else, you are likely getting a 403 Forbidden for the access to show_phpinfo.php. See the following warning block for why.
WARNING
Do not put this particular script on a production server. phpinfo() goes into great detail about your system, and some of it could be useful to a hacker.
Because people might upload all the book’s source code to their web server before they read this chapter, an .htaccess file is included that specifically blocks access to show_phpinfo.php, using this block:
<Files "show_phpinfo.php">
deny from all
</Files>
On a system where you are confident that the outside world will not have access, you can go ahead and delete that block from the .htaccess file.
Note that .htaccess files will only work if your Apache configuration is set to allow their use. Sometimes they are disabled for reasons of either performance or central control. Changing AllowOverride to All or at least AllowOverride AuthConfig Limit in your Apache configuration files will do the job. Seehttps://httpd.apache.org/docs/2.0/mod/core.html#allowoverride for more information.
See Authorization (with Apache) in Chapter 9 for more on using the .htaccess file to control access to SSE resources.
Most other languages supply the same access to the headers. Here is how to do it with a standalone Node.js server:
var http = require("http");
http.createServer(function(request,response){
console.log(request.method+" "+request.url);
console.log(request.headers);
if(request.url!="/sse"){
response.end("<html>"+
"<head><title>Logging test</title></head>"+
"<body><script>"+
"var es = new EventSource('/sse');"+
"</script></body></html>\n");
return;
}
response.writeHead(200,
{ "Content-Type": "text/plain" });
response.end();
}).listen(1234);
Start this with node log_headers.node.js, and it will listen on port 1234 on all IP addresses of your server. The key line is console.log(request.headers);. It outputs to the console, but you could easily change this to log to a file, as the PHP example did. The rest of the script is scaffolding to send back an HTML file that can call the server again using SSE. Also of interest might be the console.log(request.method+" "+request.url); line, to show which file was requested.
Event
As we have seen throughout the book, the server prefixes the data to send with data:. Then on the client side this is received by creating a handler for the message event:
es.addEventListener("message", function(e){
var d = JSON.parse(e.data);
document.getElementById(d.symbol).innerHTML = d.bid;
},false);
It turns out that the message is the default, and you can have the server label each line of data in such a way that a different function can be used to take care of it on the frontend.
The labeling is done with an event: line preceding the data. Then on the client side it is handled by specifying a handler for just that kind of event. An example will make this clear. Going back to the FX application, the data could have been sent like this:
event:AUD/GBP
data:{"timestamp":"2014-02-28 06:49:55.081","bid":"1.47219","ask":"1.47239"}
event:USD/JPY
data:{"timestamp":"2014-02-28 06:49:56.222","bid":"94.956","ask":"94.966"}
event:EUR/USD
data:{"timestamp":"2014-02-28 06:49:56.790","bid":"1.30931","ask":"1.30941"}
event:EUR/USD
data:{"timestamp":"2014-02-28 06:49:57.002","bid":"1.30983","ask":"1.30993"}
event:EUR/USD
data:{"timestamp":"2014-02-28 06:49:57.450","bid":"1.30972","ask":"1.30982"}
event:AUD/GBP
data:{"timestamp":"2014-02-28 06:49:57.987","bid":"1.47235","ask":"1.47255"}
event:AUD/GBP
data:{"timestamp":"2014-02-28 06:49:58.345","bid":"1.47129","ask":"1.47149"}
Compare this with the first code block in Fine-Grained Timestamps. If you are very bandwidth-sensitive, using event: like this appears to save 6 bytes per message. However, by changing “symbol” to “s” in our original JSON, it would be a mere 1-byte difference, and if we were using CSV instead of JSON, then using event: would be 7 bytes more expensive.
NOTE
The event name can be made up of any Unicode characters, except carriage return and line feed. If you need multiline event names, first find a mirror and ask yourself: “Really?” If the answer is still yes, then work out some escaping mechanism—for instance, JSON-encoding event names and then using the encoded version in your call to addEventListener.
On the client side, instead of the “message” handler shown earlier, I instead create a handler for each possible “event.” In this case that means one event handler for each FX symbol:
es.addEventListener("EUR/USD", function(e){
var d = JSON.parse(e.data);
document.getElementById("EUR/USD").innerHTML = d.bid;
},false);
es.addEventListener("USD/JPY", function(e){
var d = JSON.parse(e.data);
document.getElementById("USD/JPY").innerHTML = d.bid;
},false);
es.addEventListener("AUD/GBP", function(e){
var d = JSON.parse(e.data);
document.getElementById("AUD/GBP").innerHTML = d.bid;
},false);
I am sure that made you cringe, throw up your hands, and scream, “Yuck!” Yes. When the data being multiplexed (FX symbols in this case) is the same format, and processed in the same way, using event: for each data stream is going to cost you more than it gains. It was a bad example…I’m sorry.
A better example? Well, one where the data for each “event” is going to be processed in a different way. How about a chat application? It is reasonable to imagine this kind of data stream being sent:
event:enter
data:{id:17653,name:"Sweet Suzy"}
event:message
data:{msg:"Hello everyone!",from:17563}
event:exit
data:1465
The chat messages are sent as JSON. The JSON has a msg field for the actual chat message and a from field with the ID of the user who sent it. When members enter the chat room they are announced with an enter event, which gives the user ID, and information about them (here just their name). When members leave the chat room, an exit message is sent and the data is just their numeric ID, not a JSON object:
es.addEventListener("enter",
function(e){ addMember(JSON.parse(e.data)); },false);
es.addEventListener("exit",
function(e){ removeMember(e.data); },false);
es.addEventListener("message",
function(e){ addMessage(JSON.parse(e.data)); },false);
We use the following functions to do the actual work:
function addMember(d){
members[d.id] = d;
var img = document.createElement("img");
img.id = "member_img_" + d.id;
img.alt = d.name;
img.src = "/img/members/" + d.id + ".png";
document.getElementById("memberimg").appendChild(img);
}
function removeMember(id){
var img = document.getElementById("member_img_" + id);
img.parentNode.removeChild(img);
delete members[id];
}
function addMessage(d){
var msg = document.createElement("div");
msg.innerHTML = d.msg;
document.getElementById("messages").appendChild(msg);
}
NOTE
addMessage() could use d.from. I’ve also skipped over error checking: be careful with this code in production because it allows a JavaScript injection attack (though the server should be taking care of stripping out bad tags from the chat messages).
How can the fallbacks be made to work with event: lines? One way is to add some code to our earlier processNonSSE(msg) (see Dealing with Data). We will also need a global to remember which event is being processed currently:
var currentEvent = null;
...
function processNonSSE(msg){
var lines = msg.split(/\n/);
for(var ix in lines){
var s = lines[ix];
if(s.length == 0)continue;
if(s.indexOf("event:") == 0){
currentEvent = s.substring(6);
}
else{
if(currentEvent == "exit"){
removeMember(s);
}
else{
if(s[0] != "{"){
s = s.substring(s.indexOf("{"));
if(s.length == 0)continue;
}
var d = JSON.parse(s);
if(currentEvent == "enter")
addMember(d);
else if(currentEvent == "message")
addMessage(d);
//else unknown event
}
}
}
}
Notice that some of the complexity here is because of not using a JSON object for all events. This is one reason I suggest just settling on using a JSON object for all data; it makes dealing with the fallbacks easier.
That is one way. The other way is to add an event field to the JSON object (again, this requires changing the “exit” event to use a JSON object). This resembles the way we used the id: row for SSE clients, but also repeated the “id” information in the JSON object (see Sending Last-Event-ID).
But if we are going to do that, why bother with using the event row at all? We end up with all messages coming through processOneLine(s) and the code looks a bit like this:
switch(d){
case "enter":addMember(d);break;
case "exit":removeMember(d.id);break;
case "message":addMessage(d);break;
}
So, to sum up, the event: feature of SSE is one way to organize different actions, but it offers no advantages over doing it yourself with an extra JSON or CSV field, and doing it that way makes dealing with other browsers easier and more efficient. So I suggest you only use event: when both these conditions are true:
§ All your clients have native SSE support.
§ You want to use a mix of different data types for your event types, including some simple data types such as integer, float, or string (and therefore including your own event field is not possible).
Multiline Data
Throughout the book I have advocated using JSON objects for message passing. One of the reasons for that it is gave us one exactly line per message. Why is that a benefit? Because it makes it very easy to do the parsing in the fallbacks used by the older browsers.
Did you notice how in the FX application the backend only sent an extra carriage return after the data when in SSE mode? When using long-poll, XHR, or iframe techniques, it skipped this because we didn’t need it: one line of JSON is always a complete message. (Incidentally, we saved one byte, or actually six bytes, because the fallbacks did not prefix the data lines with data: either. Saving 6 bytes was not the reason this was done. Saving some client-side processing was.)
So why does SSE require that extra blank line between messages? It is there because the SSE standard allows for a message to be split across multiple lines. For example, the server can send this data:
data:Roses are red
data:Violets are blue
data:No need to escape
data:When you do as I do
<-- Extra LF
For the sake of understanding how the client will deal with it, let’s pretend that the server flushes the data after each line, then goes to sleep for a second or two. The client will receive “Roses are red.” No blank line has been received, so it buffers it up and waits. Two seconds later it gets a second line, “Violets are blue,” so it buffers this: “Roses are red\nViolets are blue.” Notice that it is just buffering—it is not telling the client that any data has arrived yet. After the fourth line it has buffered up “Roses are red\nViolets are blue\nNo need to escape\nWhen you do as I do.” Finally, the client gets a blank line. It calls the JavaScript event handler passing the single long string built up in its buffer.
NOTE
The string passed to the event handler does not have the final LF.
(The standard says clients should go to the trouble of adding an LF after each line when it buffers it up, only to then remove the final one at the end. Standards do things like that, and often find themselves alone at parties, with no one to talk to but the potted plant in the corner.)
What if you really wanted a blank line at the end of your message? Send a blank data: line. For instance, the following sequence will pass the string “111\n\n333\n\n” to the event handler:
data:111
data:
data:333
data:
data:
Why does SSE let us do this? It does it so that there is no need to escape carriage returns. In contrast when we use JSON, the above poem looks like this:
data:"Roses are red\nViolets are blue\nNo need to escape\nWhen you do as I do"
Unlike in the buffer example earlier in this section, \n refers to two bytes, first a \, then an n. Including the data: and the following blank line, the JSON string is 80 bytes, whereas the non-JSON version is 91 bytes. All those data: strings added up to more than the extra byte for the \, and the extra two bytes for the quotes.
NOTE
How do we implement handling multiple lines for a single message in the fallbacks? Basically you would have to implement the SSE buffering algorithm described earlier in this section, in JavaScript. And the server has to send that extra blank line for all clients, not just the SSE clients. This is not that hard, and you wouldn’t need to use the data: prefix, so the byte difference would not be against you. But when this approach is compared to the ease of always using one line of JSON, I feel you would need a jolly good reason to want to go down this path.
In summary, use the multiline feature of SSE when all these conditions are true:
§ All your clients have native SSE support.
§ You have naturally multiline data to send.
§ You have a good reason not to use JSON.
Whitespace in Messages
This is a quick, short section. Throughout the book I have used data:XXX, event:XXX, etc. The standard also allows you to write data: XXX, event: XXX, etc. In other words, you can have a space after the colon. I am an easygoing person, happy to let people choose their own way of doing things, but I’m going to take a stand here: never do this. It just wastes a byte, and has no advantage whatsoever.
But this feature creates a potential problem: if you are sending raw strings as your data, if you ever need to have a leading space in your data, it will get sucked away. What to do? The simple solution is to use JSON. Gosh, I do keep harping on about that, don’t I! The downside is minor: two extra bytes per string (for the quotes), as well as an extra escape slash character if your string has any special characters. But that is still a downside; is there another solution? Yes. If you want to send raw strings, and there is the chance of an important leading space, then prefix all strings with a space. It wastes one byte per line. If that waste still bothers you, only do this when your data has a leading space…but that is a lot of fuss for the sake of a byte.
Headers Again
In the FX application, I passed in xhr=1 or longpoll=1 in the URL so that the server could identify the fallback. We then identified SSE as the absence of either of those. There is another way. But before we look at it, here is a reminder of how those are used:
longpoll
Send a text/plain content-type header; exit after sending a message.
xhr
Send a text/plain content-type header.
sse
Send a text/event-stream content-type header.
Send a data: prefix, an extra carriage return, and id: lines.
The alternative way is that SSE clients will send an Accept: text/event-stream header, which should uniquely identify them as supporting SSE natively. So the FX application had these lines:
$GLOBALS["is_longpoll"] = array_key_exists("longpoll", $_POST)
|| array_key_exists("longpoll", $_GET);
$GLOBALS["is_xhr"] = array_key_exists("xhr", $_POST)
|| array_key_exists("xhr", $_GET);
$GLOBALS["is_sse"]=!($GLOBALS["is_longpoll"] || $GLOBALS["is_xhr"]);
...
if($GLOBALS["is_sse"])header("Content-Type: text/event-stream");
else header("Content-Type: text/plain");
By instead using that header, there is no longer a need to send xhr=1; there is still a need to send longpoll=1 though, so the difference between that and XHR/iframe can be detected. The code ends up looking something like this:
$GLOBALS["is_sse"] = @$_SERVER["HTTP_ACCEPT"] == "text/event-stream";
$GLOBALS["is_longpoll"] = array_key_exists("longpoll", $_POST)
|| array_key_exists("longpoll", $_GET);
$GLOBALS["is_xhr"]=!($GLOBALS["is_longpoll"] || $GLOBALS["is_sse"]);
...
if($GLOBALS["is_sse"])header("Content-Type: text/event-stream");
else header("Content-Type: text/plain");
You have perhaps spotted why I didn’t do it this way: it is the same amount of complexity, with no advantages. Using the explicit xhr or longpoll has a couple of small advantages. First it appears in the server logs, whereas HTTP headers usually do not. That might help troubleshooting. Second, there is the risk of a buggy browser forgetting to send the header, or missing out the hyphen, etc. Sending a URL parameter is fairly riskless.
So Is That Everything?
In this chapter we have looked at the event: feature of SSE as well as how it supports sending messages with multiple lines, plus how leading spaces can cause problems. We did not use any of these features in the FX application, because by using JSON they become unnecessary.
To answer the “So Is That Everything?” question: no, it is still not everything the SSE standard mentions. We still have CORS to talk about. This, along with authentication, will be covered in the next chapter.