Socket.io for Tardy React Status Indicators

The White Rabbit, Wikimedia Commons
The White Rabbit, Wikimedia Commons

In one of my favorite movies “The Blues Brothers”, the wife of the trucker pit stop owner proudly exclaims that they play ‘both kinds of music – Country and Western’. Readers of my blog know I too tend to serve two kinds of articles. In think-pieces, I tend to pontificate on a topic that buzzes between my ears that week. In ‘hey, look at this code we wrote’, I share something we learned by doing. I am sure there is an audience for both kinds but if you are in the latter camp, this is your day.

I’m late, I’m really really late

The problem we had to solve recently was as follows: a React store contains a list of items. These items are rendered in a hierarchy of React components, with a ‘dummy’ React component rendering a card for each item. The item has a number of visual elements on it that can be easily rendered from the item’s properties, but it also contains a status indicator.

server-cards

This status indicator was killing us: when we fetch the array of items via the XHR API call, we can get them fairly quickly. However, each item represents an entity that can be working or not working (and therefore have a green or red status indicator). Computing this status takes a while, needs to be done separately for each item, and cannot hold the main XHR call.

We solved this problem with the help of the Socket.io module (at this point I need to remind you we use Node.js for our server side). You can apply the solution even if you are not using it, nor using Node at all – just use whatever Web Sockets library comes with your stack of choice.

The devil is in the pudding

OK, on to the details. What we do is as follows:

  • Whenever we want to update the status of the store, we issue an XHR.
  • The XHR fetches the items on the server via a downstream API call.
  • Before sending the response, the server endpoint initiates an array of async parallel calls to obtain status for each of the items. It sends the response to the client without waiting.
  • The client renders the cards, with the status rendered as a gray circle.
  •  When the status API calls return, the result is sent back via Web Socket channel as an event.
  • On the client, this event is converted into a Flux action, which in turn updates the Store, which updates the views.

react-status-sequence

Produce the code

OK, enough architecture, produce the code. In the interest of space conservation, I am going to assume you have rudimentary knowledge of both React and Flux. If not, Google away until React components, dispatcher, actions and stores are all concepts that make sense to you.

Our action creator is capable of dispatching two actions – fetching items and initiation service status update:

var AppDispatcher = require("./dispatcher");

var ServiceConstants = require("./action-constants");

var ServiceActions = {
	fetchItems: function(data) {
		AppDispatcher.handleViewAction({
			actionType: ServiceConstants.FETCH_ITEMS,
			data: data
		});
	},
	serviceStatusUpdate: function(data) {
		AppDispatcher.handleViewAction({
			actionType: ServiceConstants.SERVICE_STATUS_UPDATE,
			data: data
		});
	}
};

module.exports = ServiceActions;

OK, on to the ItemStore. It is a fairly standard affair. When asked to fetch items, it makes an XHR request to the endpoint in the Node.js server that returns server items. When status update arrives, it merges the provided status with the items and fires a change event again. This will drive the view to re-render:

// Modules

var Constants = require("./action-constants");
var EventEmitter = require("events").EventEmitter;

// Globals

var AppDispatcher = require("./dispatcher");
var XhrUtils = require("./xhr-util");

var EVENT_CHANGE = "items";
var items;
var emitter = new EventEmitter();
var listeningForSocket = false;

// Public Methods ------------------------------------------------------------->

var ItemStore = {

	getItems: function(type) {
		return items;
	},

	emitChange: function(data) {
		emitter.emit(EVENT_CHANGE, data);
	},

	addChangeListener: function(callback) {
		emitter.on(EVENT_CHANGE, callback);
	},

	removeChangeListener: function(callback) {
		emitter.removeListener(EVENT_CHANGE, callback);
	}

};

module.exports = ItemStore;

// Private Methods ------------------------------------------------------------>

function fetchItems(url) {
	if (!url) {
		console.warn("Cannot fetch items - no URL provided");
		return;
	}

	// Fetch content
	XhrUtils.doXhr({url: url, json: true}, [200], function(err, result) {
		if (err) {
			console.warn("Error fetching assets from url: " + url);
			ItemStore.emitChange(Constants.STATE_ERROR);
			return;
		}

		items = result;
		ItemStore.emitChange(Constants.ITEMS);
	});
}

function serviceStatusUpdate(data) {
	for (var i in data) {
		var s = data[i];
		_updateStatus(s);
	}
	ItemStore.emitChange(Constants.ITEMS);
}

function _updateStatus(s) {
	for (var i in items) {
		var item = items[i];
		if (item.id===s.id) {
			item.status = s.status;
			break;
		}
	}
}

// Dispatcher ------------------------------------------------------------>

// Register dispatcher callback
AppDispatcher.register(function(payload) {
	var action = payload.action;

	// Define what to do for certain actions
	switch (action.actionType) {
		case Constants.SERVICE_STATUS_UPDATE:
			serviceStatusUpdate(action.data);
			break;
		case Constants.FETCH_ITEMS:
			fetchItems(action.data);
			break;
		default:
			return true;
	}

	return true;

});

Finally, the view. It fires the action that fetches items upon mounting, and proceeds to render the initial state with the loading indicator. Once the items are ready, it re-renders itself with the server items in place. Finally, when the status update arrives, it just re-renders and lets React diff the DOM for changed properties. Which is why we love React in the first place:

var React = require('react');
var Router = require('react-router');
var ItemStore = require('../flex/item-store');
var ActionCreator = require('../flex/action-creator');
var ActionConstants = require('../flex/action-constants');

module.exports = React.createClass({
  getInitialState: function() {
    return { loading: true, items: [] };
  },

  componentDidMount: function() {
    ItemStore.addChangeListener(this._handleAssetsChanged);
    ActionCreator.fetchItems("/api/items");
  },

  componentWillUnmount: function() {
    ItemStore.removeChangeListener(this._handleAssetsChanged);
  },

  _handleAssetsChanged: function(type) {
    if(type === ActionConstants.STATE_ERROR) {
      this.setState({
        error: "Error while loading servers"
      });
    } else {
      this.setState({
        loading: false,
        error: null,
        items: ItemStore.getItems() || []
      });
    }
  },

  render: function render() {
    var loading;
    var items;
    var error;

    if (this.state.loading) {
      loading = (
         <div className="items-loading">
            <div className="LoadingSpinner-dark"></div>
         </div>
      );
    }
    else {
      items = (
         <ul>
          {this.state.items.map(function(item) {
            var statusClass = "status-indicator";
            if (item.status==="active")
              statusClass+= " status-active";
            else if (item.status==="error")
              statusClass+= " status-error";
            else
              statusClass+= " status-loading";
            return (
               <li className="service-card" key={item.id}>
                  <div className="service-card-status">
                  <div className={statusClass}/></div>
                  <div className="service-card-name">{item.title}</div>
                  <div className="service-card-type">{item.dist}</div>
              </li>
            );
          })}
        </ul>
      );
    }
    if (this.state.error) {
      error = (
         <div className="error-box">{this.state.error}</div>
      );
    }
    return (
       <div id='list'>
          <h1>Servers</h1>
             This list shows currently available servers and their status (
             <span className="text-loading">gray</span> - loading,&nbsp; 
             <span className="text-active">green</span> - active,&nbsp; 
             <span className="text-error">red</span> - error)
          {loading}
          {items}
          {error}
      </div>
    );
  }
});

The status update portion involves Socket.io module. When the status is being computed, it is being emitted via the Web Socket. Of course, for this example we are faking the delay, but you can imagine actual status API request being involved:

var io = require('socket.io')(server);

function emitStatus(items) {
  var status = [];
  for (var i in items) {
    var item = items[i];
    var s = {};
    s.status = (Math.random()<.5)?"active":"error";
    s.id = item.id;
    setTimeout(function(status) {
      io.emit("status", [status]);
    }, (Math.floor((Math.random() * 900) + 100)), s);
  }
}

This status even is being picked up on the client, producing the status update action that is dispatched, picked up by the store and eventually causing the view to update:

  var socket = io.connect(location.origin);
  socket.on('status', function (data) {
  	ActionCreator.serviceStatusUpdate(data);
  });

Conclusion and commentary

One of the many realization we have come to using React in production is that one can get very far with views alone. A top-level view can make XHR requests, then pass the data down to child components as props. Child components can capture low-level events, and translate them into events of higher level of abstraction via callbacks passed in as props.

We realized many simple pages can be entirely done this way. What we have noticed though is that when Web Sockets and server-side events are involved, Flux architecture really comes to its own and shows its potential. Mixing server side events with events produced by users interacting with the UI is much better done using Flux.

The problem we tried to solve is rendering the initial page. If the user stays on the page for a prolonged period of time, status changes of the items can continue, but the code above can handle it without any changes. Of course, if others can create or remove server items, new events and actions are needed to model that.

As usual, the entire example is available as a Node.js app on Github, and I have deployed it in Bluemix so that you can see the app running here: https://react-status-demo.mybluemix.net. Just keep refreshing the browser to see the three-step rendering (page, then items, and finally status).

© Dejan Glozic, 2016

Isomorphic Apps Part 2: Node, React.js, and Socket.io

Two Heads, 1930, Wikimedia Commons
Two Heads, 1930, Wikimedia Commons

When I was a kid, I went to the movies to watch Mel Brooks’ “History of The World, Part I”. I had a great time and could not wait for the sequel (that featured, among other things, Hitler on ice, a Viking funeral and laser-shooting rabbis in ‘Jews in Space’ teaser). Alas, ‘Part II’ never came. Determined to not subject my faithful readers to such a disappointment, here comes the promised part II of my ‘Isomorphic Apps’ trilogy.

In the first part of this story, we created an isomorphic app by taking advantage of the fact that we can use Dust.js as an Express view engine, and then compile partials into JavaScript and re-use them on the client as needed. In order to compare approaches with only one variable changed, we will switch to React.js for the view.

What’s the deal with React.js

React.js is attracting a lot of attention these days due to the novel approach it has taken to building dynamic Web apps. At the heart of the approach is the notion of a virtual DOM. React.js components manipulate an abstraction of a DOM that is then transformed into the physical DOM in a highly optimized fashion. Even more ingeniously, browser’s DOM is only one of the possible transformations: virtual DOM can be also serialized into plain HTML, which makes it possible to use it on the server. Even more recently, it can be serialized into native code to address mobile (and even desktop) UI components.

I am old enough to remember Java’s “Write once, run anywhere” slogan, and this looks like new generation’s attempt to make a run for this chimera. But even putting React native on a side for a moment, the fact that you can render on the server makes React supremely suitable for isomorphic apps, something Angular.js is lacking.

React.js is also refreshingly simple to figure out. Angular.js has this famous adoption roller coaster, and sometimes when you don’t get an Angular peculiarity, you feel the fault is with you, not Angular. React.js took an approach that life is short, and we can do better things with our time than figure out the maddening quirks of a complex framework. There is no two-way binding (because it has shown to be a double-edged sword – see what I did here). When the model changes, you just naively rebuild the view (sometimes referred to as ‘write pages like it’s the 90s’). Seems massively suboptimal, but remember that you are only rebuilding the virtual DOM – React.js figures out the actual delta and only applies the delta against the real DOM. And since most of the performance (or lack thereof) lies in the physical DOM, React.js promises fast apps without writing a lot of code for smart and surgical updating on model changes.

Configuring React.js as an Express view engine

Alright, I hope this wet your appetite for some coding. We will start by cloning the page from part I and adding another view engine in app.js (because I am cheap/lazy and don’t want to run another app for this). For this we need to install react on the server, as well as the express view adapter.

We will start by installing ‘react’ and ‘express-react-views’ and configuring the jsx view engine:


var react = require('express-react-views');

...

app.engine('jsx', react.createEngine());
app.set('view engine', 'jsx');

The last line above should only be set if you will use JSX as the only view engine for Express. In my case, I had to omit that line because I am already serving some Dust pages, and you can only set one default engine. The only thing I lost this way was the ability to find JSX templates without the extension – they can still be rendered when extension is included.

The controller for our React.js page is almost identical to the one we wrote for Dust.js:


var model = require('../models/todos');

module.exports.get = function(req, res) {
   model.list(req.user, function(err, todos) {
      res.render('isomorphic_react.jsx',
         { title: 'React - Isomorphic', user: req.user, todos: todos });
   });
};

Most of the fun happens in the view, as expected. React.js requires some getting used to. For starters, JSX syntax is actually XML (and not even XHTML), so all elements require termination. Many attribute names require camel case, which is very annoying (I always hated Jade for this mental transformation, and now JSX is doing the same for me). At least the JSX transformer is yelling at you in the console about possible errors you made, so fixing up your JSX is not too hard:

var React = require('react');
var DefaultLayout = require('./rlayout');
var RTodo = require('./rtodo');

var Todos = React.createClass({
  render: function() {
    return (
      <DefaultLayout { ...this.props} selection="react">
        <h1>Using React.js for View</h1>
        <h2>Todos</h2>
        <div className="new">
           <textarea id="new-todo-text" placeholder="New todo"/>
        </div>
        <div className="delete">
           <button type="button" id="delete-all"
              className="btn btn-primary">Delete All</button>
        </div>
        <div id="todos" className="todos">
           {this.props.todos.map(function(todo) {
          	return <RTodo key={todo.id} {...todo} />;
           })}
        </div>
        <script src="/js/prettyDate.js"></script>
        <script src="/js/rtodo.js"></script>
        <script src="/js/rtodos.js"></script>
      </DefaultLayout>
    );
  }
});

module.exports = Todos;

The code above requires some explanation. Unlike with Dust.js, both inclusion into a common layout template and instantiation of partials is done through React.js component model. Notice that we imported DefaultLayout component that is our standard page boilerplate. The payload of the page is simply specified as content of the instantiated component in the ‘render’ method above.

Another important point is that unlike Dust.js, properties are not automatically passed down the component hierarchy – we need to explicitly do it (notice the strange “{ …this.props }” expression in the DefaultLayout declaration – what I am saying is ‘pass all the properties down to the child component’). We can also define new properties, which I am doing by passing ‘selection’ that will be used by the header component (to highlight the ‘React’ link).

Another important section of the template is where I am instantiating RTodo component (a single Todo card). Flow control can be tricky in JSX because the entire template is one giant return statement, so everything needs to evaluate to an expression. Notice the trick with using the array map to iterate over the list of todos and render each child todo component.

This code will produce a page very similar to the one with Dust.js, with identical results. In fact, it is possible to go back and forth because both pages are using the same REST service for the model.

JSX compiler

So far we took care of the server side. As with Dust.js, we can compile components we need on the client side, this time using jsx compiler that comes by installing ‘react-tools’:


#!/bin/bash
node_modules/react-tools/bin/jsx --extension jsx views/ public/js/ rtodo

We can compile any number of components and place them into the JS directory under /public folder so that Express can serve them to the browser.

The client side script is very similar to the one used by the Dust.js page. The only difference is in the ‘Add’ action handler:

var socket = io.connect('/');
socket.on('todos', function (message) {
  if (message.type=='add') {
    var newTodo = document.createElement('div');
    React.render(React.createElement(RTodo, message.state),
              newTodo);
    $(".todos").prepend(newTodo);
  }
  ...

The code is remarkably similar – instead of calling ‘dust.render’ to render the partial using the element we received via the Socket.io message, we ask React to render the compiled element into a new DOM element we created on the fly. We then prepend this element into the parent DIV.

Commentary and comparisons

First off, I would say that this second attempt at writing an isomorphic app was a success because I was able to replicate Dust.js example from part I with identical behaviour. However, it is not as good a fit for React.js. A better example would see us modifying a model and asking React.js to re-render an existing DOM branch. Now that I feel reasonably comfortable around React.js, I think I will create something more dynamic for it in the near future. A true React-y way of doing the list of todos would be to simply re-render the entire list on each Socket.io message. We would let React.js figure out that all it needs to do is insert a new Todo DIV into the parent node. This way we would not need to create DOM elements ourselves, as in the code above.

After spending a year working with Dust.js, JSX took some getting used to. As I said, I hated Jade because it created an additional layer of abstraction between me and HTML, and I never quite knew what final HTML it will produce. JSX evokes the same feelings in me, but the error/correction loop has shortened as I learned it more. In addition, the value I get in return is much higher with JSX than with Jade.

Nevertheless, certain things will not get better with time. JSX is awkward to work with when it comes to logic. Remember, the entire template is an expression, so statements are really hard to fit in. Tertiary conditonals work, and as you saw, it is possible to use tricks with maps to iterate over children in a collection. I still prefer Dust.js for straightforward pages, but I can see how React.js can work with components in a very dynamic app.

I like React.js component model, as well as the fact that code and markup are close to each other – for a developer this is very useful. I also like the fact that, JSX quirks aside, there is much less magic compared to Angular.js. Of course, React.js is really just a View of MVC, so it is not a fair comparison. On the other hand, I am now dying to hook it up into Backbone as a view – it feels like a great combination (and of course, there are already articles on exploiting this exact combination). The more I think and read about it, Backbone models/collections/router and React.js views may just end up being my favorite stack for writing highly dynamic apps with server side bonus for SEO and initial experience.

A word of caution

If your system has elements of both a site and an app, use micro-services and implement site portions with a more straightforward templating solution (as I already covered in the previous blog post). This is going to make content authoring easier and increase the number of content providers with cursory knowledge of HTML that will feel confident authoring and/or modifying something like Dust.js templates. Leave React.js for 10x developers working on the highly dynamic ‘app’ portions of the system. By the way, this is an area where micro-services shine – this kind of partitioning is one of their key selling points. You can easily have micro-services using Dust.js and micro-services using React.js (and as I have already shown, even mixed in the same Node app).

One of the downsides of a mixed system (one using both Dust.js and React.js) is that sometimes content pages have dynamic component sprinkled in them. The challenge is invoking those component without requiring your casual developers to be afraid to touch such pages. Invoking React.js component in a Dust.js page would require inserting JavaScript tags, which is less then ideal. This is where Web Components are much easier to reason about, and there are already attempts to bridge the two worlds – invoking React.js components as custom Web Components.

And that’s a wrap

As before, you can browse the source code as IBM DevOps Services project, and the latest version of the app is running in Bluemix. In the final instalment of this trilogy, I will make our example a bit more dynamic (to let React.js show its true potential), and add some structure using Backbone.js. Until then, React away!

© Dejan Glozic, 2015

Full Stack Toronto Conference 2014

IMG_0871

We at IBM are not strangers to large, well capitalized conferences. As things go in the conference-industrial complex, it is a big deal when one of your keynote speakers is Kevin Spacey, or Imagine Dragons entertain you after hours. So to say that the first Full Stack Toronto Conference was on the opposite side of the spectrum would be an understatement.

How about ‘no food’, ‘no drinks except coffee in the morning’, and ‘no entertainment’ of any kind except finding parking around Ryerson University building? OK, not fair, there was a get together in the nearby Irish pub the first night that I didn’t go to because I was tired.

This conference was really just a meetup making the next step. And on a weekend. Starting on 8:45am. Who does that? Our keynote speaker Anila Arthanari, Director Of Software Development at Infusionsoft, wore a t-shirt ‘I am not a morning person’ expressing the mood of most of the audience. By all accounts, this was supposed to be a flop.

And yet. When you peel the layers of conference pageantry, what remains is the kernel of it all – good talks. When talks are good, people will not mind walking out to a nearby panini store to buy lunch, or walk a block up the Church Street to get a Starbucks hit. Nothing matters if talks are good.

And they were. We had multiple tracks, and I could not go to all the talks, but those I attended were very informative, thought-provoking and immanently applicable. After every talk I had tons of things I wrote down to try after, or catch up on. So what were my key takeaways from the conference?

  1. Lots of people use Angular.js. If you need a client side MVC, you can do worse, with a caveat that version 2.0 is on the slowly approaching horizon, and to say that migration will be interesting would be an understatement.
  2. More and more people use Browserify over RequireJS. Put off by weird configuration syntax, and feeling the need easier code reuse in the case of Node.js, people seem to prefer to just ‘require’ their modules. It makes it easier to go back and forth. I am definitely going to try using it soon. This blog might help as well.
  3. Micro-services are everywhere. In the tongue-in-cheek ‘Show Us Your Stack’ track, multiple presenters described their journey from monoliths to micro-service systems. I like how people are now past the hype and deep in the gory details on standing up such systems in practice. Many were openly asking the audience for their feedback and red flags if they see any.
  4. Not everybody uses Angular.js. I know this is a contradiction, but people who value control and being able to grow into a client side MVC still value Backbone.js and its modular approach. If anything, Angular.js 2.0 promises to be more modular and less arrogant, for the lack of a better word. Here is hoping that in the future, considering Angular.JS will not be such an ‘all or nothing’ dilemma.
  5. Isomorphic and ‘federation of single-page apps’ is a thing. I thought I will be the only one pushing for rendering stuff on both the server and the client using the same templates, but Matthew Conlen from New York Data Company talked about exactly such an approach. Personally, I find it funny that people are happy to partition the API space into micro-services, but don’t feel the need to do the same with the Web apps. As the system grows, a single one page app providing all the UI is going to be the bottleneck of the system. Which is ironic, because user interfaces are the most transient and need to be evolved at a rapid pace. In essence, we are creating a system where API services can move at a rapid speed, but the UI is one big ball of MVC mud.
  6. React can help with isomorphic. Once you decide rendering on both sides of the fence is important to you, React is an attractive proposition because it can do exactly that, and plays nice with Node.js.
  7. Internet Of Things is still in its infancy. It seems like we all feel this weird excitement over turning the lights with Node.js apps and sending messages to robots and receiving MQTT messages that it is now 24C in the room. It is apparent that all this stuff will matter one day and great things will come, but I don’t see what to do with it today other than marvel in the possibilities. I guess once somebody does something really awesome with IoT, we will all slap our collective foreheads and say ‘but of course, so elegant’.

By the way, yours truly presented as well. You can see my slides up on Slideshare, and the source code of my demo on GitHub. You may find it interesting – I got Angular.js to fit into the micro-service driven Web UI, use normal URLs (no horrible hashes or hash bangs), and share a common header with other pages. I also demonstrated SSO using Facebook as the identity provider, lively UI using Web Sockets and isomorphic approach using Dust.js rendered on both client and server. The best part was when an audience member posted a todo into the demo running live on Bluemix, and his entry popped up on screen as I was demoing it. Audience participation, live demo, unexpected proof that the code actually works as designed – priceless!

So there you have it – you can make the attendees feed themselves, only give them coffee in the morning (what is life even), and dispense with most of the usual conference perks, and they will still come if the talks are good. I would say that the first FullStack TO conference focused on the most important thing, and succeeded. Good talks first, creature comforts to follow – good priorities in my book. Looking forward to the next year!

© Dejan Glozic, 2014

The Year of Blogging Dangerously

391px-Extremely_yummy_raspberry_cheesecake

Wow, has it been a year already? I am faking surprise, of course, because WordPress has notified me well ahead of time that I need to renew my dejanglozic.com domain. So in actuality I said ‘wow, will it soon be a year of me blogging’. Nevertheless, the sentiment is genuine.

It may be worthwhile to look back at the year, if only to reaffirm how quickly things change in this industry of ours, and also to notice some about-faces, changes of direction and mind.

I started blogging in the intent to stay true to the etymological sense of the word ‘blog’ (Web log). As a weekly diary of sorts, it was supposed to chronicle trials and tribulations of our team as it boldly goes into the tumultuous waters of writing Web apps in the cloud. I settled on a weekly delivery, which is at times doable, at other times a nightmare. I could definitely do without an onset of panic when I realize that it is Monday and I forgot to write a new entry.

Luckily we have enough issues we deal with daily in our work to produce enough material for the blog. In that regard, we are like a person who just moved into a new condo after his old apartment went up in flames and went to Ikea. If an eager clerk asks him ‘what do you need in particular’, his genuine answer must be ‘everything – curtains, rugs, new mattress, a table, chairs, a sofa, a coffee table …’.

At least that’s how we felt – we were re-doing everything in our distributed system and we were able to re-use very little from our past lives, having boldly decided to jump ahead as far as possible and start clean.

Getting things out of the system

That does not mean that the blog actually started with a theme or a direction. In the inaugural post The Turtleneck and The Hoodie, I proudly declared that I care both about development AND the design and refuse to choose. But that is not necessarily a direction to sustain a blog. It was not an issue for a while due to all these ideas that were bouncing in my head waiting to be written down. Looking back, I think it sort of worked in a general-purpose, ‘good advice’ kind of way. Posts such as Pulling Back from Extreme AJAX or A Guide to Storage for ADD Types were at least very technical and based on actual research and hands-on experience.

Some of the posts were just accumulated professional experience that I felt the need to share. Don’t Get Attached to Your Code or Dumb Code Good, Smart Code Bad were crowd pleasers, at least in the ‘yeah, it happened to me too’ way. Kind of like reading that in order to lose weight you need to eat smart and go outside. Makes a lot of sense except for the execution, which is the hard part.

344px-'Be_smart..Act_dumb^_-_NARA_-_513932

Old man yells at the cloud

Funnily enough, some of my posts, after using up all the accumulated wisdom to pass on, sound somewhat cranky in hindsight. I guess I disagreed with some ideas and directions I noticed, and the world ignored my disagreement and continued, unimpressed. How dare people do things I don’t approve of!

Two cranky posts that are worth highlighting are Swimming Against the Tide, in which I am cranky regarding client side MVC frameworks, and Sitting on the Node.js Fence, in which I argue with myself on pros and cons of Node.js. While my subsequent posts clearly demonstrate that I resolved the latter dilemma and went down the Node.js route hook, line and sinker, I am still not convinced that all that JavaScript required to write non-trivial Single Page Apps (SPAs) is a very good idea, particularly if you have any ambition to run them on mobile devices. But it definitely sounds funny to me now – as if I was expressing an irritated disbelief that, after publishing all the bad consequences of practicing extreme Ajax, people still keep doing it!

I heart Node.js

Of course, once our team went down Node.js route (egged on and cajoled by me), you could not get me to shut up about it. In fact, the gateway drug to it was my focus on templating solutions, and our choice of Dust.js (LinkedIn fork). By the way, it is becoming annoying to keep adding ‘LinkedIn fork’ all the time – that’s the only version that is actively worked on anyway.

Articles from this period are more-less setting the standard for my subsequent posts: they are about 1500 words long, have a mix of outgoing links, a focused technical topic, illustrative embedded tweets (thanks to @cra who taught me how not to embed tweets as images like a loser). As no story about Node.js apps is complete without Web Sockets and clustering, and both were dully covered.

Schnorr_von_Carolsfeld_Bibel_in_Bildern_1860_006

I know micro-services!

Of course, it was not until I went to attend NodeDay in February that a torrent of posts on micro-services was unleashed. The first half of 2014 was all ablaze with the posts and tweets about micro-services around the world anyway, which my new Internet buddy Adrian Rossouw dully documented in his Wayfinder field guide. It was at times comical to follow food fights about who will provide the bestest definition of them all:

If you follow a micro-services tag for my blog, the list of posts is long and getting longer every week. At some point I will stop tagging posts with it, because if everything is about them, nothing is – I need to be more specific. Nevertheless, I am grateful for the whole topic – it did after all allow me to write the most popular post so far: Node.js and Enterprise – Why Not?

monty-1920-1200-wallpaper

What does the future hold?

Obviously Node.js, messaging and micro-services will continue to dominate our short-term horizon as we are wrestling with them daily. I spoke about them at the recent DevCon5 in NYC and intend to do the same at the upcoming nodeconf.eu in September.

Beyond that, I can see some possible future topics (although I can’t promise anything – it is enough to keep up as it is).

  • Reactive programming – I have recently presented at the first Toronto Reactive meetup, and noticed this whole area of Scala and Akka that is a completely viable alternative to implement micro-services and scalable distributed systems that confirm to the tenets of Reactive Manifesto. I would like to probe further.
  • Go language – not only because TJ decided to go that route, having an alternative to Node.js while implementing individual micro-services is a great thing, particularly for API and back-end services (I still prefer Node.js for Web serving apps).
  • Libchan – Docker’s new project (like Go channels over the network) currently requires Go (duh) but I am sure Node.js version will follow.
  • Famo.us – I know, I know, I have expressed my concerns about their approach, but I did the same with Node.js and look at me now.
  • Swift – I am a registered XCode developer and have the Swift-enabled update to it. If only I could find some time to actually create some native iOS apps. Maybe I will like Swift more than I do Objective-C.

I would like to read this post in a year and see if any of these bullets panned out (or were instead replaced with a completely different list of even newer and cooler things). In this industry, I would not be surprised.

Whatever I am writing about, I would like to thank you for your support and attention so far and hope to keep holding it just a little bit longer. Now if you excuse me, I need to post this – I am already late this week!

© Dejan Glozic, 2014

Socket.io: Mind the Gap

Wikimedia Commons, 'Mind the Gap', 2008, Clicsouris
Wikimedia Commons, ‘Mind the Gap’, 2008, Clicsouris

Welcome to our regular edition of ‘Socket.io version 1.0 watch’ or ‘Making sure Guillermo Rauch is busy working on Socket.io 1.0 instead of whatever he does to pay the rent that does nothing for me’. I am happy to inform you that Socket.io 1.0 is now available, with the new logo and everything. Nice job!

With that piece of good news, back to our regular programming. First, a flashback. When I was working on my doctoral studies in London, England, one of the most memorable trivia was a dramatic voice on the London Underground PA system warning me to ‘Mind the Gap’. Since then I seldomly purchase my clothes in The Gap, choosing its more upmarket sibling Banana Republic. JCrew is fine too. A few years ago a friend went to London to study and she emailed me that passengers are still reminded about the dangers of The Gap.

We have recently experienced a curious problem in our usage of WebSockets – our own gap to mind, as it were. It involves a topology I have already written about. You will most likely hit it too, so here it goes:

  1. A back-end system uses message queue to pass messages about state changes that affect the UI
  2. A micro-service serves a Web page containing Socket.io client that turns around and establishes a connection with the server once page has been loaded
  3. In the time gap between the page has been served and the client calls back to establish a WebSockets connection, new messages arrive that are related to the content on the page.
  4. By the time the WebSockets connection has been established, any number of messages will have been missed – a message gap of sorts.
mind-the-gap
WebSockets gap: in the period of time from HTTP GET response to the establishment of the WebSockets connection, msg1 and msg2 were missed. The client will receive messages starting from the msg3.

The gap may or may not be a problem for you depending on how you are using the message broker to pass messages around micro-services. As I have already written in the post about REST/MQTT mirroring, we are using MQTT to augment the REST API. This augmentation mirrors the CRUD verbs that result in state change (CUD). The devil is in the details here, and the approach taken will decide whether the ‘message gap’ is going to affect you or not.

When deciding what to publish to the subscribers using MQ, we can take two approaches:

  1. Assume subscribers have made a REST call to establish the baseline state, and only send deltas. The subscribers will work well as long as the took the baseline and didn’t miss any of the deltas for whatever reason. This is similar to showing a movie on a cable channel in a particular time slot – if you miss it, you miss it.
  2. Don’t assume subscribers have the baseline state. Instead, assume they may have been down or not connected. Send a complete state of the resource alongside the message envelope. This approach is similar to breaking news being repeated many times during the day on a news channel. If you are just joining, you will be up to date soon.

The advantages of the first approach are the message payloads. There is no telling how big JSON resources can be (a problem recently addressed by Tim Bray in his fat JSON blog post). Imagine we are tracking a build resource and it is sending us updates on the progress (20%, 50%, 70%). Do we really want to receive the entire Build resource JSON alongside this message?

On the other hand, the second approach is not inconsistent with the recommendation for PUT and PATCH REST responses. We know that the newly created resource is returned in the response body for POST requests (alongside Location header). However, it is considered a good practice to do the same in the requests for PUT and PATCH. If somebody moves the progress bar of a build by using PATCH to update the ‘progress’ property, the entire build resource will thus be returned in the response body. The service fielding this request can just take that JSON string and also attach it to the message under the ‘state’ property, as we are already doing for POST requests.

Right now we didn’t make up our minds. Sending around entire resources in each message strikes us as wasteful. This message will be copied into each queue of the subscribers listening to it, and if it is durable, will also be persisted. That’a a lot of bites to move around while using a protocol whose main selling point is that it is light on the resources. Imagine pushing these messages to a native mobile client over the air. Casually attaching entire JSON resources to messages is not something you want to do in these situations.

In the end, we solved the problem without changing our ‘baseline + deltas’ approach. We tapped into the fact that messages have unique identifiers attached to them as part of the envelope. Each service that is handling clients via WebSockets has a little buffer of messages that are published by the message broker. When we send the page the client, we also send the ID of the last known message embedded in HTML as data. When WebSockets connection is established, the client will communicate (emit) this message ID to the server, and the server will check the buffer if new messages have arrived since then. If so, it will send those messages immediately, allowing the client to catch up – to ‘bridge the gap’. After it has been caught up, the message traffic resumes as usual.

As a bonus, this approach works for cases where the client drops the WebSockets connection. When connection is re-established, it can use the same approach to catch up on the messages it has missed.

The fix: the service sends the ‘message marker’ (last message id). Client echoes the marker when connecting with WebSockets. Detecting the hole in message sequence, the service immediately sends the missing messages allowing the client to catch up.

As you can see, we are still learning and evolving our REST/MQTT mirroring technique, and we will most likely encounter more face-palm moments like this. The solution is not perfect – in an extreme edge case, the WebSockets connection can take so long that the service message buffer fills up and old messages start dropping off. A solution in those cases is to refresh the browser.

We are also still intrigued with sending the state in all messages – there is something reassuring about it, and the fact that the similarity to PATCH/PUT behavior only reinforces the mirroring aspect is great. Perhaps our resources are not that large, and we are needlessly fretting over the message sizes. On the other hand, when making a REST call, callers can use ‘fields’ and ’embed’ to control the size of the response. Since we don’t know what any potential subscriber will need, we have no choice but to send the entire resource. We need to study that approach more.

That’s it from me this week. Live long, prosper and mind the gap.

© Dejan Glozic, 2014

Release the Kraken.js – Part I

Kraken
Image credit: Yarnington 2011

Of course when you’re a kid, you can be friends with anybody. […] You like Cherry Soda? I like Cherry Soda! We’ll be best friends!

Jerry Seinfeld

We learned from Rene Zellweger that sometimes you can have people at “Hello”. In case of myself and the project Kraken.js, it had me at “Node.js/express.js/Dust.js/jQuery/Require.js/Bootstrap”. OK, not fit for a memorable movie quote but fairly accurate – that list of libraries and platforms was what we at JazzHub eventually arrived at as our new stack to write cloud development tools for BlueMix. Imagine my delight when all these checkboxes were ticked in a presentation describing PayPal’s experience in moving from Java to Node for their Lean UX needs, and announcing project Kraken. You can say I had my ‘Cherry Soda’ moment.

Of course, blessed with ADD as I am, I made a Flash-like scan of the website, never spending time to actually, you know, study it. Part of it is our ingrained fear and loathing of frameworks in general, and also our desire to start clean and very slowly add building blocks as we need them. This approach is also advocated by Christian Heilmann in his article ‘The Vanilla Web Diet’ in the Smashing Magazine Book 4.

Two things changed my mind. First, as part of NodeDay I had a pleasure to meet and talk to Bill Scott, Senior Director of UX Engineering, Jeff Harrell, Director of Engineering Architecture, and Richard Ragan, Principal Engineer and ‘the Dust.js guy’, all from PayPal, and now feel retroactively bad for not giving the fruit of their labor more attention. Second, I had a chat with a friend from another IBM group also working on a Node.js project, and he said “If we were to start again today, I would definitely give Kraken.js a go – I wish we had it back then, because we had to essentially re-implement most of it”. Strike two – a blog post is in order.

So now with a combination of guilt, intra-company encouragement and natural affinity of our technologies, I started making inventory of what Kraken has to offer. At this point, hilarity ensued: the Getting Started page requires installation of Yeoman, and then the Kraken generator. Problem: there is no straightforward way to install Yeoman on Windows. I have noticed during a recent NodeDay that when it comes to Node, Macs outnumber Windows machines 99 to 1, but considering that Kraken is targeting enterprise developers switching to Node, chances are many of them are running Windows on their work machines. A Yeoman for Windows would definitely help spread the good word.

Update: it turns out I was working on the outdated information – after posting the article, I tried again and managed to install Yeoman on a Windows 7 machine.

Using Yeoman is of course convenient to bootstrap your apps – you create a directory, type ‘yo kraken’, answer couple of questions and the scaffolding of your app is set up for you. At this point our team lead Curtis said ‘I would be fine with a zip file and no need to install Yeoman’. He would have been right if using Yeoman was a one-time affair, but it turns out that Kraken Yeoman generator is a gift that keeps on giving. You can return to it to incrementally build up your app – you can add new pages, models, controllers, locales etc. It is a major boost to productivity – I can see why they went through the trouble.

Anyway, switched to my MacBook Pro, I installed Yeoman, then Kraken generator, then typed ‘yo Kraken’ and my app was all there. Once you start analyzing the app structure, you see the Kraken overhead – there are more directories, more files, and Kraken has decided on how things should be structured and tries to stick to it. On the surface of it, it is less clear for the beginner, but I am already past the pure express ‘Hello, World’ apps. I am interested in how to write real world production apps for the enterprise, and Kraken delivers. It addresses the following enterprise-y needs:

  1. Structure – app is clearly structured – models, controllers, templates, client and server side – it is all well labelled and easy to figure out
  2. i18n – Kraken is serious about enterprise needs, so internationalization is baked in right from the start
  3. Errors – it provides translated files for common HTTP errors (404, 500, 503) as a pattern
  4. Templates – Kraken uses Dust, which fits our needs like a glove, and the templates are located in the public folder so that they can be served to both Node.js and the browser, allowing template sharing. Of course, templates are pre-compiled into JS files for performance reasons.
  5. Builds – Kraken uses Grunt to automate all the boring tasks of preparing your app for deployment, and also incorporates Mocha tests as a standard part. Having a consistent approach to testing (and the ability to incrementally add tests for new pages) will improve quality of your apps in the long run.
Kraken-hello
Kraken.js HelloWorld app directory structure

One thing that betrays that PayPal deals with financial transactions is how serious they are about security. Thanks to its Lusca module, Kraken bakes protection against the most prevalent types of attacks right into your app. All enterprise developers know this must be done before going into production, so why tacking it on in a panicky scramble at the last minute when you can bake it into the code from the start.

Of course, the same goes for i18n – Makara module handles translations. At this point I paused looking at the familiar file format from the Java world – *.properties files. My gut reaction would be to make all the supporting files in JSON format (and to Kraken authors’ credit, they did do so for all the configuration files). However, after thinking about the normal translation process, I realized that NLS files are passed to the translation team that is not necessarily very technical. Properties files are brain-dead – key/value pairs with comments. This makes them easy to author – you don’t want to receive JSON NLS files with missing quotes, commas or curly braces.

Makara is designed to work with Dust.js templates, and Kraken provides a Dust helper to inline locale-based substitutions into the template directly (for static cases), as well as an alternative for more dynamic situations.

As for the main file itself, it did give me pause because the familiar express app was nowhere to be found (I absent-mindedly typed ‘node app.js’, not realizing the app is really in index.js):

'use strict';

var kraken = require('kraken-js'),
    app = {};

app.configure = function configure(nconf, next) {
    // Async method run on startup.
    next(null);
};

app.requestStart = function requestStart(server) {
    // Run before most express middleware has been registered.
};

app.requestBeforeRoute = function requestBeforeRoute(server) {
    // Run before any routes have been added.
};

app.requestAfterRoute = function requestAfterRoute(server) {
    // Run after all routes have been added.
};

if (require.main === module) {
    kraken.create(app).listen(function (err) {
        if (err) {
            console.error(err.stack);
        }
    });
}

module.exports = app;

This is where the ‘framework vs library’ debate can be legitimate – in a true library, I would require express, dust etc. in addition to kraken, start the express server and somehow hook up kraken to it. I am sure express server is somewhere under the hood, but I didn’t spend enough time to find it (Lenny Markus is adamant this is possible) . This makes me wonder how I would approach using cluster with kraken – forking workers to use all the machine cores. Another use case – how would I use Socket.io here – Socket.io piggy-backs on the express server and reuses the port. I will report my findings in the second instalment.

I would be remiss not to include Kappa here – it proxies NPM repository allowing control over what is pulled in via NPM and provides for intra-enterprise sharing of internal modules. Every enterprise will want to reuse code but not necessarily immediately make it Open Source on GitHub. Kappa fits the bill, if only for providing an option – I have not tested it and don’t know if it would work in our particular case. If anything, it does not suffer from NIH – it essentially wraps a well regarded npm-delegate module.

Let’s not forget builds: as part of scaffolding, Yeoman creates a Gruntfile.js configured to build your app and do an amazing amount of work you will get for free:

  1. Run JSHint to verify your JS files
  2. Build Require.js modules
  3. Run less compiler on the css files
  4. Build locale-specific versions of your dust templates based on i18n properties
  5. Compile dust templates into *.js files for efficient use on the client
  6. Run Mocha tests

I cannot stress this enough – after the initial enthusiasm with Node.js, you are back to doing all these boring but necessary tasks, and to have them all hooked up in the new app allows the teams to scale while retaining quality standards and consistency. Remember, as a resident Node.js influencer you will deal with many a well intentioned but inexperienced Java, PHP or .NET developer – having some structural reinforcement of good practices is a good insurance against everything unravelling into chaos. There is a tension here – you want to build a modern system based on Node.js micro-services, but don’t want each micro service to be written in a different way.

Kraken is well documented, and there are plenty examples to understand how to do most of the usual real world tasks. For an enterprise developer ready to jump into Node.js, the decision tree would be like this:

  1. If you prefer frameworks and like opinionated software for the sake of consistency and repeatability, use Kraken in its entirety
  2. If you are starting with Node.js, learn to write express.js apps first – you want to understand how things work before adding abstractions on top of them. Then, if you are OK with losing some control to Kraken, go to (1). Otherwise, consider continuing with your express app but cherry-picking Kraken modules as you go. To be fair, if Kraken is a framework at all (a very lightweight one as frameworks go), it falls into the class of harvested frameworks – something useful extracted out of the process of building applications. This is the good kind.
  3. If you ‘almost’ like Kraken but some details bother you or don’t quite work in your particular situation, consider participating in the Open Source project – helping Kraken improve is much cheaper than writing your own framework from scratch.

A friend of mine once told me (seeing me dragging my guitar for a band rehearsal) that she “must sit down one afternoon and learn how to play the guitar already”. I wisely opted out of such a mix of arrogance and cluelessness here and called this ‘Part I’ of the Kraken.js article. I am sure I will be able to get a better idea of its relative strengths and weaknesses after a prolonged use. For now, I wanted to share my first impressions before I lose the innocence and lack of bias.

Meanwhile, in the words of “the most interesting man in the world”: I don’t always use frameworks for Node.js development, but when I do, I prefer Kraken.js.

© Dejan Glozic, 2014

Pushy Node.js II: The Mullet Architecture

marvel-mullets

The post title unintentionally sounds like a movie franchise (e.g. Star Trek II: The Wrath of Khan), but in my defense I DID promise I will return to the topic of pushing events to the browser using Socket.io and Node.js in the post about message queues. The idea was to take the example I wrote about in Pushy Node.js and modify it so that the server messages are received from another service using a message queue.

I got the idea for the sequel title from an awesome tweet I saw the other day:

Alas, nothing can be hidden under the Google administration – a quick search yields an article from 2010 on mullet architecture (an actual one, with houses, not software). There is even such a thing as ‘reverse mullet’ – party in the front and business in the back. Frank Lloyd Wright aside, I did mean ‘software architecture’ in this particular post. I wanted to mimic the situation we face these days, where Node.js front end typically communicates with the legacy services in the back that nobody dares to touch. As a result, this post will mix Node.js, Socket.io, MQTT protocol and Java app (posing as the mullet’s backside). I am all for accuracy in advertising.

To refresh your memory, the example app in the previous article was creating a hypothetical system in which builds are running on the server, and as they are going through their execution, we use Web Sockets to drive the Bootstrap progress bar in the browser. To make the original sample palatable, we faked the build by simply using the timeouts. We will continue with the fakery, but remove one layer by moving the fake build into a dedicated Java app. This Java app (if you squint, you can imagine it is in fact a Jenkins server running builds) will react to the POST request to start or stop the build, but the build service will publish messages using the message broker. Our Node.js app will subscribe to the ‘builds’ topic and pass the information about the build progress to the client using Socket.io.

This all provides the ‘how’, but not the ‘why’. Recall that the power of message queues is in avoiding polling (our Node.js app does not need to constantly nag the Java app “are we there yet?”), and also in providing for flexible architecture. Imagine, for example, that in the future we want to capture trend data on build average times and success/failure rates. We could easily write another service that will subscribe to the ‘builds’ topic and store the data in a DB, and also provide a nice data visualization. I can easily see a nice Node.js app storing data into a NoSQL database like as MongoDB or CouchDB, and a nice page rendered using Dust.js template and D3.js for data visualization. Why, the app is practically writing itself!

The key point here is that you can easily add this feature tomorrow by simply adding another subscriber to the ‘builds’ message topic – you don’t need to modify any of the existing apps. This low friction way of growing the system by adding micro-services is very powerful, and has also been reinforced during the recent NodeDay in the presentation by Richard Roger, founder of NearForm.

All right then, we know why and we know how, so let’s get busy. First, we will install RabbitMQ as our message broker. I have chosen RabbitMQ because it is open source, supports multiple protocols, scales well and is available as a service in PaaSes such as CloudFoundry or Heroku. As for the protocol, I will use MQTT, a lightweight pub/sub protocol particularly popular for anything M2M. If you want to play with MQTT but don’t want to install RabbitMQ, a lightweight open source MQTT broker called Mosquitto is also available, although I don’t know how well it can hold up in production.

For our Java app, both AMQP and MQTT clients are readily available. RabbitMQ provides the AMQP client, while Eclipse project Paho has the MQTT client. For this example, I will use the latter. When you have time, read the article REST is for sleeping, MQTT is for mobile by J Spee.

Let’s look at the diagram of what the control flow in the original example app looked like:

pushy-node

A Node.js app handles the initial GET request from the browser by rendering the page with a button and a progress bar. Button click issues a jQuery Ajax POST request to the app, initiating a fake build. As the build is going through the paces, it uses Socket.io and Web Sockets to push the progress to the brower, moving the progress bar.

Now let’s take a look of what the new topology will look like:

pushy-node-mq

The flow starts as in the original example, but the Node.js app just proxies the build start/stop command to the Java app listening on a different port. The Java app will kick a fake build on a worker thread. As the build is running, it will periodically publish messages of its progress to the message broker on a predetermined topic using MQTT client. Node.js app registered as a subscriber to the same topic will receive the messages and pass them on the browser using Web Sockets.

In order to save space, I will not post the code snippets that didn’t change – check them out in the original article. The first change here is how we react to POST request from the browser in the Node.js app. Instead of kicking off or stopping the fake build, we will just proxy the call to the Java app:

exports.post = function(req, res) {
	var options = {
       hostname: 'localhost',
       port: 9080,
       path: '/builds/build',
       method: 'POST'
	};
	var rreq = http.request(options, function (rres) {
		console.log('Redirect status: '+rres.statusCode);
	});
	rreq.write(JSON.stringify(req.body));
	rreq.end();
}

On the Java end, we will write a JEE servlet to handle the POST request:

package fake.java.build.service;

import javax.servlet.ServletInputStream;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.wink.json4j.JSON;
import org.apache.wink.json4j.JSONException;
import org.apache.wink.json4j.JSONObject;
import org.eclipse.paho.client.mqttv3.MqttClient;
import org.eclipse.paho.client.mqttv3.MqttException;
import org.eclipse.paho.client.mqttv3.MqttMessage;
import org.eclipse.paho.client.mqttv3.MqttPersistenceException;

public class BuildService extends HttpServlet {
	private static final String TOPIC = "builds";
	private static FakeBuild build;

	class FakeBuild implements Runnable {
		boolean running;
		boolean errors;

		public void run() {
			MqttClient client;
			try {
				client = new MqttClient("tcp://localhost:1883", "pushynodeII");
				client.connect();
			} catch (MqttException e) {
				e.printStackTrace();
				return;
			}
			running = true;
			errors = false;

			for (int i = 0; i <= 100; i += 10) {
				if (i == 70)
					errors = true;
				if (i == 100)
					running = false;
				// notify state
				publishState(client, running, i, errors);
				if (running == false)
					break;
				try {
					Thread.sleep(1000);
				} catch (InterruptedException e) {
					e.printStackTrace();
				}
			}
			try {
				client.disconnect();
			} catch (MqttException e) {
				e.printStackTrace();
			}
		}

		private void publishState(MqttClient client, boolean running,
				int progress, boolean errors) {
			try {
				JSONObject body = new JSONObject();
				body.put("running", running);
				body.put("progress";, progress);
				body.put("errors", errors);
				MqttMessage message = new MqttMessage(body.toString()
						.getBytes());
				try {
					client.publish(TOPIC, message);
				} catch (MqttPersistenceException e) {
					e.printStackTrace();
				} catch (MqttException e) {
					e.printStackTrace();
				}

			} catch (JSONException e) {
				e.printStackTrace();
			}
		}
	}

	protected void doPost(HttpServletRequest req, HttpServletResponse res) {
		try {
			ServletInputStream bodyStream = req.getInputStream();

			JSONObject body = (JSONObject) JSON.parse(bodyStream);
			bodyStream.close();
			doToggleBuild("start";.equals(body.get("action")));
		} catch (Exception e) {
			e.printStackTrace();
			res.setStatus(500);
		}
	}

	private void doToggleBuild(boolean start) {
		if (start) {
			build = new FakeBuild();
			Thread worker = new Thread(build);
			worker.start();
		} else {
			if (build!=null)
				build.running = false;
		}
	}
}

In the code above, a ‘start’ command will kick the build on the worker thread, which will sleep for 1 second (as in the JavaScript case), and then send a MQTT message to update the subscribers on its status. In case of a ‘stop’ command, we will just turn the ‘running’ flag to false and let the build to naturally stop and exit, and also update the status. We will pass the build state in message body as JSON (duh).

Once the Java app is up and running and messages are flowing, it is time to add a MQTT message subscriber in Node.js app. To that end, I installed the mqtt module using NPM. All we need to do to start listening is create a client in app.js, subscribe to the topic and register a callback function to handle incoming messages:

var mqtt = require('mqtt')
   ,io = require('socket.io');

....

    var server = http.createServer(app);
    io = io.listen(server);

    var mqttClient = mqtt.createClient(1883, 'localhost');
    mqttClient.subscribe('builds');

    mqttClient.on('message', function (topic, message) {
       io.sockets.emit('build', JSON.parse(message));
    });

The code above will create a client for localhost and the default tcp port 1883. In a real world situation, you will most likely supply these as environment or config variables. The client listens on the topic and simply passes on the body of the message as a JavaScript object by passing the message to all the currently opened sockets (clients). One important meta-comment highlighting difference in approaches between Node and Java: because Node is asynhconous, I was able to create a client and bind it to a TCP port, then proceed to bind the app to another port for HTTP requests. In a JEE app, we would need to handle incoming messages in a separate thread, and possibly manage a thread pool if one thread cannot handle all incoming message traffic. As if you needed more reasons to switch to Node.js.

In terms of behavior, this code in action looks the same in the browser as the previous example, but we made an important step towards an architecture that will work well in todays’ mixed language architectures and have room for future growth. What I particularly like about it is that Java and Node.js apps can pass the messages in an implementation-agnostic way, using JSON. Note that the Java app in this example is not as far-fetched as it may seem: you can make the example even more real by using a Jenkins service. Jenkins has REST API that our Node app can call to start and stop build jobs, and once a job is underway, we can install an MQTT notifier plug-in to let us know about the status. The plug-in is simple and uses the same Paho client I used here, so if you don’t like what it does, it is trivial to roll your own.

The intent of this post was to demonstrate an architecture you will most likely work with these days, even if you are a total Node-head. Apart from legacy back-end services, even some newly written ones may need to be in some other language (Java, C++, Go). Nobody in his right mind would write a video transcoding app in JavaScript (if that is even possible). Using message brokers will help you arrive at an architecture that looks like you actually meant it that way, rather than fell backwards into it (what I call ‘accidental architecture’). Even if it is a mullet, wear it like you mean(t) it!

© Dejan Glozic, 2014