Dust.js as a React Delivery Vehicle

648px-Uniformed_Letter_Carrier_with_Child_in_Mailbag

Even people in love with Single Page Apps and pooh-poohing server side rendering know that SPAs cannot just materialize in the browser. Something has to deliver them there. Until recently, AngularJS was the most popular client side JS framework, and I have seen all kinds of ADVs (Angular Delivery Vehicles) – from Node.js (N is for Node.js in the MEAN stack), to wonderfully ironic JSP files serving Angular. Particularly exuberant developers go all the way to using static HTML to launch their apps into orbit, extolling the virtues of CDNs.

Of course, I would like to see their faces the first time they realize the need to compute something dynamically in those static files. But that is beside the point. The important takeaway is that AngularJS has a very nice property of being embeddable. You don’t need to surrender your entire page to Angular – you can sequester it to a DIV (in fact, you can have multiple Angular apps running in the same page). This is a very nice feature, but also a necessary one – again, AngularJS is a pure client side framework.

With great power…

React changes this equation because it can be rendered on the server. Similarly to AngularJS, React root component can be mounted to any node inside your page, but unlike Angular, this page can also be rendered on the server by React.

This opens the door for a whole new class of isomorphic or universal apps, and I have written about it already. What you need to decide is whether you want to render the entire page, or a subset of it. If you choose the latter, then the subsequent question becomes – how do you start the rendering process until React takes over?

My first inclination was to go full React. Using the awesome react-engine module, it is easy to configure React as the view engine in the Express framework, and render the entire page. However, we soon hit some snags:

  1. As we render our UIs using micro services that use UI composition to glue the pages together, we hit a problem of React being grumpy about having to render HTML snippet arriving from outside of its control – something much easier with a templating library such as Dust.js.
  2. JSX turned out to be fairly quirky and finicky when rendering HTML HEAD element, as well as trying to inline JavaScript in tags. In fact, the latter became so error prone that we completely gave up on it, electing to put all our script in files. I didn’t like this particular restriction, and this remains an area where I am sour on JSX and the fact it most decidedly isn’t HTML.

Of course the problems I listed above did not show up in the small example I published in the initial article. They reared their ugly head once we started getting serious. In my original example, the header was a React component. This works great if you control the entire app, but the whole deal of UI composition is to integrate with apps not necessarily written using React, and you can only do that using the least common denominator (HTML, CSS and vanilla JS).

Remind me again about UI composition

In a nutshell, UI composition is the approach where common areas on the page are served as HTML snippets from a REST API. The API service can also serve CSS and JavaScript files that accompany the snippet (if you don’t serve them from a CDN). The idea is that a dedicated microservice can be providing these common areas this way, allowing other microservices to pull the content and inline it in their pages regardless of their stack.

UI composition inverts the flow you would normally use with iframes. Instead of rendering the common areas, then placing an iframe for the content and loading the content from an external URL, we load the common area as HTML and inline it into our page. In the olden days this approach was often accomplished using ‘edge side includes’ or ESIs. This approach has many benefits, including no need for the dreaded iframes, full control over the entire page, and the ability to integrate microservices implemented using different stacks. For example, the teams in my current project use:

  1. Node.js microservices using React for isomorphic rendering
  2. Node.js microservices using Dust.js and jQuery
  3. Java microservices using AngularJS

All of these microservices render the exact same header even though they use different stacks and libraries. Powerful stuff.

OK, so React?

The problem here with React is that when we use it to render the entire page server-side, it starts to squirm in discomfort. Inlining HTML is a problem for React because, while it can be used on the server, it was designed to shine on the client. Inlining raw HTML presumably obtained as a user input into DOM is a major security exposure without sanitization, and React will tell you so in no uncertain terms. Here is the line that inlines the shared header taken straight out of our production code:

<div dangerouslySetInnerHTML={{__html: decodeURIComponent(this.props.header)}} />

This is React telling us very clearly: “I really, really, really don’t like what you are asking me to do”. And this is even before we bring in react-engine. React engine is a PayPal module that enables isomorphism when combined with Node.js/express and react-router. We can render on the server, then pack the properties as part of the HTML response, hydrate the React components on the client and continue where we left off.

Only we have a little problem: notice that in the snippet above we have inlined the header by passing it down to the React component as a prop. What this means is that this prop (‘header’) will be sent to the client alongside other properties. The only problem is that this property contains HTML snippet for the entire header, and because it can really mess up react-engine, it is also URLencoded, making it even bulkier.

This is an unhappy situation: first we inline the header HTML (which is fine), then we send that same inlined HTML, this time encoded, as a React prop to the client. This unnecessarily bloats the generated HTML.

RDV

This is what we are going to do to solve this problem: we are going to call back our trusty Dust.js sitting sad on the sidelines ever since we got smitten with React. We can press it into service to do what it does best – outline the page, prep it up to the point where React can take over.

It’s a good thing that Express framework will happily handle more than one view engine. Of course, only one of them can be registered as default, but this just means that we will need to spell out the extension of dust files (because React is the default). Big deal.


var path = require('path')
, express = require('express')
, renderer = require('react-engine')
, dust = require('dustjs-linkedin')
, helpers = require('dustjs-helpers')
, cons = require('consolidate');

...

// create the view engine with `react-engine`
var engine = renderer.server.create({
  routes: require(path.normalize(__dirname + '/public/routes.jsx')), 
  docType: ""
});

// set the engines
app.engine('.jsx', engine);
app.engine('.dust', cons.dust);

// set the view directory
app.set('views', __dirname + '/public/views');

// set jsx as the view engine
app.set('view engine', 'jsx');

// finally, set the custom view
app.set('view', renderer.expressView);

Once we have both engines set up, we can arrange our delivery vehicle like this. We will first set up a reusable layout template using Dust.js that will deliver the outline of each page:

<!DOCTYPE html>
<html>
 <head>
   <meta charset='utf-8'>
   <meta http-equiv="X-UA-Compatible" content="IE=edge">
   <meta name="viewport" content="width=device-width, initial-scale=1" >
   {+headContent/}
   <title>
     {title}
   </title>
   <link rel="stylesheet" href="/css/styles.css">
 </head>
 <body>
   {header|s}
   <div class="main-content">
     {+mainContent/}
   </div>
   {+javascript/}
 </body>
</html>

Notice how inlining of the header now happens in Dust.js, relieving React from doing it. We can now use this template to render our page like this:

{>"layout"/}
{<mainContent}
<div id="react-mount-node">{spa|s}</div>
{/mainContent}
{<javascript}
<script src="/bundle.js"></script>
{/javascript}

We have created a DIV in the template above where we are mounting React, and we will inline the React server-side content using the ‘spa’ variable.

Our Express controller that will handle both Dust.js and React can look something like this:

var request = require("request");

module.exports.get = function(req, res) {
  var settings = {
    title: 'SPA - Demo',
    name: 'React SPA',
    selection: 'header-spa'
  };

  var headerUrl = "http://"+req.headers.host+"/header?selection=header-spa";

  request.get(headerUrl, function (err, response, body) {
    if (err)
      console.log(err);
    else
      settings.header = body;
    res.render(req.url, { name: settings.name }, function(err, html) {
      if (err) {
        settings.spa = err.message;
        console.log(JSON.stringify(err.stack));
      }
      else
        settings.spa = html;
      res.render("spa.dust", settings); 
    });
  });
}

The code above is what I call ‘React Delivery Vehicle’. It has three steps:

  1. First we fetch the header from the URL that is providing us with the header HTML snippet. We capture it into a variable ‘header’. In production, we will heavily cache this step.
  2. Then we render the React root component as usual. This will employ react-engine and react-router to render all the views necessary for the provided request URL. However, we don’t send the rendering to the response. Instead, we capture it in the variable ‘spa’.
  3. Finally, we invoke Dust.js view engine to render the full page, passing in the ‘settings’ object. This object will contain both the header and the content rendered by React. Dust will simply inline both while rendering the page outline. The result will be sent directly to the Express response.

Commentary

The solution described above plays to the strengths of both Dust.js as a server side template engine and React as a component library. Dust.js is great at rendering HTML outline the way you would expect, with no need to worry about JSX quirks around HEAD meta tags, or JavaScript inlining. React takes over to render isomorphic components into the mount node. This means we don’t need to send any data that is not needed on the client. This fixes the HTML bloat problem I mentioned before.

As for the negatives, setting up two rendering engines on the server has a slight overhead, and switching mentally from Dust.js to JSX adds context switching tax. Luckily, you can set up reusable Dust.js templates and not really worry about them too much – most of the action will be in JSX anyway.

We like this approach and are currently switching to it across the board. If you have alternative ideas or comments, drop me a line. Meanwhile, the source code for the entire example is available on GitHub as usual, and the sample app employing this mechanism runs on Bluemix.

© Dejan Glozic, 2015

Pessimism as a Service

As far as I can remember, I was forgetful (ironic, I know). I could be driving for ten minutes, wandering why I don’t see the phone Bluetooth symbol on my car’s dash, and realizing I forgot my cellphone at home. Lately, when I reach the door, I ask myself: “OK, what have you forgotten”? Not “have you forgotten anything” but “what”, assuming that an affirmative answer is a forgone conclusion. Such a negative, “guilty until proven innocent” approach saved me many times, but taxed my soul. Am I really that predictable? Is cynicism the only way?

As our super cool, micro-service packed, React supercharged project is picking up steam, I am looking at everything we have done and counting the ways we have deployed ‘Pessimism as a Service’ to production. These examples may seem disconnected to you, but I assure you, there is a cold, calculated thread binding them. Hey, it’s a totally accepted artistic form – my own omnibus, as it were.

Micro services and human nature

I said it before, and I will say it again – micro services are more about people and process than about technology. In his strained attempt to disguise his distaste for micro services, Martin Fowler has still mustered a fainted praise in the way micro services tend to enforce code modularity.

The trouble is that, with a monolithic system, it’s usually pretty easy to sneak around the barrier. Doing this can be a useful tactical shortcut to getting features built quickly, but done widely they undermine the modular structure and trash the team’s productivity. Putting the modules into separate services makes the boundaries firmer, making it much harder to find these cancerous workarounds.

Martin Fowler on Strong Module Boundaries

It this will not inject you with a healthy dose of Weltschmerz, nothing will. What he is saying is that reaching directly into modules instead of using proper interfaces is a tech version of a cookie jar, and instead of counting on your maturity and discipline, micro services simply hide the cookie jar or put it on a top shelf, were you can’t reach it because you skipped the gym too many times.

Large systems are built by real-world organizations, and people are messy, petty, complicated, full of hidden agendas and desires. Engineers who try to look at micro services as a rational system fail to grasp the potent property that requires high emotional intelligence to understand. And it is nothing new – in fact I posit that the first micro service architecture has been practiced by the Nipmuk Indians, living near the lake in today’s Massachusets of the impossible name Chargoggagoggmanchauggagoggchaubunagungamaugg. Translated, it is really a module boundary protocol:

You fish on your side [of the lake], I fish on mine, nobody fishes in the middle.


– Full Indian name for the lake Manchaug, shortened by locals not familiar with micro-service architecture

So, yeah. Ideally, a monolithic system could be highly modular and clean if implemented by highly disciplined, rational people impervious to human foibles. When you manage to hire a teamful of such people, do let me know. In the mean time, the jaded micro service system we are using is humming in production.

AKKA is not a true micro service system

True story – I went to present in the first Toronto Reactive meetup because: (a) I mixed Reactive with React and (b) I wanted to learn what the whole Reactive Manifesto was by presenting on it. Hey, learning by doing!

As such, I was exposed to the AKKA framework. You can read all about Reactive in one of my previous blogs, but suffice to say that AKKA is a framework based on the ‘actor’ pattern and designed specifically to foster an asynchronous, dynamic and flexible architecture that can be deployed to a single server, and then spread out across any number of clusters as the needs grow.

There is a lot to like in AKKA, but I must sadly posit here that it is not a true representative of a micro service system. It is a system inspired by micro services, implementing many of their tenets and with some really nice properties. And yet it betrays one of the key aspects of micro services in that it is not pessimistic. In order to get the benefits of it, you need to lock yourself into a Scala/AKKA stack, paraphrasing the famous Ford Model T joke (you could order it in any color as long as it was black). You lose the ability to choose your stack per micro service.

This property is often misunderstood as a licence for anarchy – a recipe for disaster, cobbling together a concoction of languages, platforms, stacks and runtimes that nobody will be able to keep running and maintain. Of course that unchecked freedom has its price: a real world microservice system will most likely be using only 2-3 stacks (in our case, they ended up being Node.js and Java) and a small number of client side frameworks (for our extended team, React and AngularJS). But there is an ocean of separation between one and two platforms – the former representing lock-in, the latter being freedom.

As I always assume I forgot something, we should always assume that something better is just around the corner, and we don’t want to be hopelessly locked in when it arrives. But we also don’t want to bet our farm on it just yet. This is where the ability to start small is vital: we can try out new approaches in a single micro service without the obligation for a wholesale switch. AKKA requires that we profess our undying love to it and its Scala/JVM stack. Your milage may vary, but I cannot put all my money in that or any other single basket.

React is smart so you can be dumb

On to the client side of the the full stack. My readers know I have expressed my reservation about AngularJS before. I always found its syntax weird, its barrier of entry too high for a practical working system, and that’s before we even mention the version 2.0 schism. However, I always feared I will be viewed as ‘old man that yells at cloud‘ for not recognizing Angular’s genius, until React arrived.

You see, I got React instantly. I didn’t have to scratch my head and re-read its examples. When you read React code, you know exactly what is happening. Of course, that’s because it does less – just the View part. You need to implement Flux for coordinating actions, data stores and views, but Flux is even simpler, and consists of a single dispatcher module you fetch from NPM. You also need something like react-router in order to handle client side page switching. Then you need something like react-engine if you want isomorphic apps (I was told the new term is ‘universal’; I will use both for fun).

You may not fathom the difference in approaches between AngularJS and React until you watch the video explaining React’s design philosophy. You can tell that Facebook deploys React to production. In my opinion, Angular suffers from being designed by rock stars for other rock stars. Once you start getting real and deploying non-trivial apps to production, you need to scale, and that means increasing the number of people that can be productive with your framework of choice. React was designed with the assumption that if the framework is predictable and relatively simple, the velocity can be increased without the proportional increase in the bug rate. Otherwise, what’s the point?

React designers took human nature into account, assumed that we are all dumb at various times of day or week, and ensured that even in those unhappy moments, we can still read our React code and understand what it is doing with relative ease. It feels like a rotten compromise, but it is pure genius.

Web Components just around the corner

Ah, Web Components. The ultimate native component model that will solve Everything. Three years ago there was a lot of excitement, and people jumping on the polyfills to ‘temporarily’ shim the browsers until everybody implements them natively. Fast-forward to November 2015, and today you still cannot bet your project on them in production. Yes, they are natively implemented in Chrome, but if you didn’t want to use IE-only browser extensions 15 years ago, why would you do it when Google, and not Microsoft, is the vendor trying to sell its agenda as a standard.

Yes, there has been some movement on cross-browser support for Web Components, at least when Shadow DOM is concerned. Nevertheless, nothing stands still, and now some aspects of the ES6 module loading are at odds with HTML Imports (an important part of Web Components spec).

And of course, what has also happened in the last three years is that we got React. It has a very strong component model (albeit one that you can only peruse if you lock yourself into React), and more importantly, it extends to the server and even native rendering. This makes React attractive in ways that Web Components will never be able to match.

A year ago, we seriously toyed with the idea of just using shims until Web Components, clearly the future of the component models, arrive. I am glad I allowed my jaded self to prevail and instead used React – it helped us ship to production, with no performance compromises coming from shims, and looking back, we would be nowhere close to the promised glorious future if we allowed exuberance to sway our better judgement.

I am not saying ‘No’ to Web Components forever – they are actually not incompatible with React, and in fact a low-level Web Component can be used just like a native component in a React application, reaping the benefits of the DOM diffing. However, we don’t intend to write Web Components ourselves – we are fully isomorphic and server-side rendering gives us benefits that a comparable Web Component would not.

I predict that Web Components will be the way for incompatible frameworks to co-exist, the way to ‘fish in the middle’ of the Nipmuk lake mentioned above.

Optimism dreams, pessimism ships

These four examples show why enthusiasm and optimism rule the prototypes, meetups and articles, but pessimism takes over in production. Taking human nature into account, rolling with the imperfections of reality, expecting and preparing for the worst pays off tenfold once the projects get serious.

Now, if I can only remember if I turned the stove off before leaving home.

© Dejan Glozic, 2015