Pushy Node.js

Last week I hoped to blog-shame Guillermo Rauch into releasing Socket.io v1.0 for my own convenience. Alas, it didn’t work (gotta work on my SEO), but I see a lot of traffic from @rauchg on the corresponding GitHub project, so my spirits are high. Meanwhile, I realized that for my own dabbling, v0.91 is pretty darn good on its own. Good enough, in fact, to socketize our example Node.js app in time for this post.

Why would I need Socket.io in the first place? Because of mobile phones. In a straightforward web server implementation, requests are always originating from the client. Client pulls, server obliges and sends markup and other resources back, and then returns to listening on the port, awaiting further requests (Node.js server does the exact same thing – this is not just ‘old tech’). With Ajax, the nature of requests is different but not the direction. To add liveliness, XHR calls are made to fetch data and update portions of the page without the full refresh, but those XHR calls again originate in the client. Even when it looks as if the server is pushing, it is all a sham – a technique called ‘long polling’ where a connection is kept alive and open and the server never closes it, choosing instead to push data nuggets to the client via the connection originally initiated by the client. Finally, with Web Sockets it is possible to have a true server push, where server initiates the connection when data is good and ready. Enough of the client acting as bored kids on a family trip (“are we there yet? are we there yet?”).

OK, but why mobile phones? Because they conditioned us to expect push notifications. We are so used for our phones telling us when there is something new to observe, that we are almost offended when we need to explicitly refresh an app to get new content. It is no surprise that developers now want that kind of lively experience on the desktops as well. It is also a prerequisite for a mobile Web app hoping to convince customers that it is just as good as a native counterpart.

I decided to add a new page to the example app I keep evolving since the first post on Node.js – too lazy to create a new app. It is also a good way to go beyond ‘Hello, World’ because examples on Socket.io home page are all in app.js. Since I am using express.js and have a few pages with their corresponding controllers and views, I decided to move most of the socket action to a dedicated page. This is a realistic real-world scenario – not all of your pages will have a need for server push. This is all assuming you are not writing a Single Page App (SPA between friends), at which point all bets are off.

Socket.io home page definitely fits this approach, which means that you get the endorphin kick from getting the code to work, but you immediately need to make changes for the real world app. In my case, I was using Require.js, and the page where the client code needs to go is namespaced for jQuery. Here is what I needed to do in the shared Dust.js partial:

   requirejs.config({
      shim: {
         'socketio': {
             exports: 'io'
         }
      },
      paths: {
         socketio: '../socket.io/socket.io'
      },
      baseUrl: "/js"
   });

Now we are ready to write some server push code. I have decided to create a mockup of something we are currently working on in the context of JazzHub – a build happening somewhere on the server that our page is watching. The page is simple – we want a button to start the build, a progress bar to watch it working, and a failure in the build at some point along the way just to mix it up.

We will start by using NPM to fetch socket.io module and hooking it up in app.js. Socket.io is designed to coexist peacefully with express.js, and to piggy-back on the server that it is starting:

var express = require('express')
, routes = require('./routes')
, dust = require('dustjs-linkedin')
, helpers = require('dustjs-helpers')
, cons = require('consolidate')
, user = require('./routes/user')
, simple = require('./routes/simple')
, widgets = require('./routes/widgets')
, http = require('http')
, sockets = require('./routes/sockets')
, io = require('socket.io')
, path = require('path');

We have required another controller for the new page (‘./routes/sockets’) as well as the library itself (‘socket.io’). We can how hook it up to the server:

var server = http.createServer(app);
sockets.io = io.listen(server);

In the last line we have passed the Socket.io root object to the new page’s controller so that we can access it there.

The new page needs a view, and we will again use Dust.js template and Bootstrap for our button and progress bar:

{>layout/}
{<head}
   <script src="/socket.io/socket.io.js"></script>
{/head}
{<content}
	<h2>Web Sockets</h2>
	<p>
		This page demonstrate the use of Sockets.io to push data from the Node.js server.
	</p>
	<p><button type="button" class="btn btn-primary" id="playButton" data-state="start">
		<span class="glyphicon glyphicon-play" id="playButtonIcon"></span></button>
	</p>
	<div class="progress" style="width: 50%">
       <div id="progress" class="progress-bar" role="progressbar" aria-valuenow="100" aria-valuemin="0" aria-valuemax="100" style="width: 100%;">
          <span class="sr-only">100% Complete</span>
       </div>
    </div>

	<p>This page is served by server {pid}</p>
{/content}
{<script}
	<script src="/js/sockets/sockets-page.js"></script>
{/script}

Here is something that took me a while to figure out, and judging by the Stack Overflow questions, it is puzzling to many a developer. If you take a look how we are referencing the client side portion of Socket.io library, it makes no sense:

<script src="/socket.io/socket.io.js"></script>

All we did was install Socket.io using NPM, and the library contains client side portion as well, but we didn’t put it in ‘/public’ where our styles and other static client-side files are. Nevertheless, our server was finding and serving this file to the client somehow. It wasn’t until I looked at the server side console that I noticed this line in the sea of Socket.io debug chatter:

debug: served static content /socket.io.js

Apparently, Socket.io is not only handling requests from its client side code, it is assisting Express in finding and serving the client side code to begin with. A bit magical for my taste but OK.

You may have noticed that I don’t have JavaScript inlined in the Dust template for the page. I did this for cleanliness, but also because curly braces in JavaScript code need to be escaped in Dust.js (because curly braces are special characters), making JavaScript exceedingly ugly. Mental note to talk to Dust.js guys about finding a better way to handle inlined JavaScript.

The content of the referenced file ‘sockets-page.js’ is here:

require(["jquery", "socketio"], function($, io) {
    // connect to socket
    var socket = io.connect('http://localhost'); // this needs to change in the real code
    socket.on('build', function (build) {
    	if (build.progress==0)
           _resetProgress();
    	else {
           $("#progress").attr("aria-valuenow", ""+build.progress)
           .css("width", build.progress+"%");
   	   if (build.errors) {
              $("#progress").removeClass("progress-bar-success")
              .addClass("progress-bar-danger");
           }
        }
    	var state = (build.running)?"stop":"start";
    	if ($("#playButton").data("state")!=state) {
           if (state=="stop") {
              $("#playButtonIcon").removeClass("glyphicon-play")
             .addClass("glyphicon-stop");
           } else {
       	      $("#playButtonIcon").removeClass("glyphicon-stop")
             .addClass("glyphicon-play");
           }
           $("#playButton").data("state", state);
       }
    });

    // bind event listeners
    $("#playButton").on("click", _handleButtonClick);

    // private function
    function _handleButtonClick(evt) {
       var state = $("#playButton").data("state");
       $.post("sockets", { action: state });
    }

    function _resetProgress() {
       $("#progress").removeClass("progress-bar-danger")
       .attr("aria-valuenow", "0")
       .css("width", "0%")
       .removeClass("progress-bar-danger")
       .addClass("progress-bar-success");
    }
});

The code above does the following: it registers a listener for the dual-purpose button we placed on the page. It’s initial function is to start the build. Once the build is in progress, a subsequent click will stop it (we change the icon glyph to reflect this). We handle the button click by POST-ing to the same controller that handles the GET request that renders the page, and passing the action in the request body (it is exceedingly easy to do this in jQuery, and equally easy to access it on the other end in Express).

In order to handle both GET and POST, we will register our controller in app.js thusly:

	app.get('/sockets', sockets.get);
	app.post("/sockets", sockets.post);

If you recall, we shimmed Socket.io so that we can use it with Require.js. In the controller code above we are requiring both jQuery and socket.js. The moment this code runs, it will establish a handshake with socket.io code on the server, and once it does, messages can start flowing in both directions. We will define one custom message ‘build’ and pass the JavaScript object containing build status (running/not running), percentage done (0-100) and whether there are errors. This information will in turn affect how we render Bootstrap button and progress bar.

Meanwhile on the server, the router for the page contains the other half of the code. We will create fake build activity. In the real world, this information will have arrived from another app where the build is actually running. In fact, it is common that some kind of message broker is used for app to app messaging on the server (a topic for a future post). For now, we will fake the build by making it last 10 seconds, with progress sent to the client every second:

exports.get = function(req, res) {
  res.render('sockets', { title: 'Web Sockets', active: 'sockets', pid: process.pid });
};

var build = {
   running: false,
   progress: 0,
   errors: false
};

var _lastTimeout;

exports.post = function(req, res) {
   var action = req.body.action;

   if (action==="stop") {
	   // stop the build.
	   build.running = false;
	   if (_lastTimeout)
		   clearTimeout(_lastTimeout);
	   _pushEvent("build");
   }
   else if (action==="start") {
       // reset the build, start from 0
       build.running = true;
       build.errors=false;
       build.progress = 0;
       _pushEvent("build");
       _lastTimeout = setTimeout(_buildWork, 1000);
   }
};

function _buildWork() {
	build.progress += 10;
	if (build.progress==70)
		build.errors=true;
	if (build.progress < 100) {
		_pushEvent("build");
		_lastTimeout = setTimeout(_buildWork, 1000);
	}
	else {
		build.running = false;
		_pushEvent("build");
	}
}

function _pushEvent(event) {
	exports.io.sockets.emit(event, build);
}

The code above should be fairly easy to read – we are faking the build by setting timeout of 1000ms (our 10% ‘ticks’). We move the ‘build.progress’ property and ’emit’ a message to all the active sockets (if you recall, we are using the ‘io’ object we attached in app.js). Any number of clients looking at this page will see the build in progress and will be able to start it and stop it.

When we start the server and navigate to the newly added ‘Sockets’ page, we can see progress bar and the button, as expected. Pressing the button starts the build and page is updated as the build progresses, turning from green to red at the 70% mark, as expected:

You can observe the the whole dance in action in this animated GIF.

Time for the post-demo discussion. Readers following this blog may remember my concerns about Node.js that kept me on the fence for a while. Node.js and a JavaScript templating library such as Dust.js offer a very fast cycle of experimentation and exploration that Bill Scott from PayPal among others has found instrumental for the process of Lean UX. However, it is hard to make such a tectonic shift for that sole reason. For me, adding server push to the mix is what tipped the scales in Node.js’ favor. It is hard to match the efficiency and scale possible this way, and alternative technologies that consume a process or a thread per request will find a very hard time trying to match the number of simultaneous connections possible with Node. Not to mention how easy and enjoyable the whole coding experience is, if you care about the state of mind of your developers.

Of course, this is not a ground-breaking revelation – the fact that Node.js is particularly suitable for DIRT-y apps was the major driving force for its explosive growth. Nevertheless, I will repeat it here in case you missed all the other mentions. If you are a JEE developer considering moving from servlets and JSPs to Node, a combination of Node.js, express.js and one of the JavaScript-based templating libraries will make for a fairly painless transition. Still, you will find yourself with a nagging feeling that the new stack is not so much better as different, particularly since you will not immediately feel an improvement in scalability as you are testing your new code in isolation. Only when you start adding server push code will you find yourself in a truly new territory and will be able to justify the expense and the effort.

Now I feel bad for halfheartedly ranting against Guillermo Rauch for not shipping Socket.io v1.0 fast enough for my liking. This experiment convinced me that if you don’t do push, you will not get Node.

© Dejan Glozic, 2014