Dust.js as a React Delivery Vehicle

648px-Uniformed_Letter_Carrier_with_Child_in_Mailbag

Even people in love with Single Page Apps and pooh-poohing server side rendering know that SPAs cannot just materialize in the browser. Something has to deliver them there. Until recently, AngularJS was the most popular client side JS framework, and I have seen all kinds of ADVs (Angular Delivery Vehicles) – from Node.js (N is for Node.js in the MEAN stack), to wonderfully ironic JSP files serving Angular. Particularly exuberant developers go all the way to using static HTML to launch their apps into orbit, extolling the virtues of CDNs.

Of course, I would like to see their faces the first time they realize the need to compute something dynamically in those static files. But that is beside the point. The important takeaway is that AngularJS has a very nice property of being embeddable. You don’t need to surrender your entire page to Angular – you can sequester it to a DIV (in fact, you can have multiple Angular apps running in the same page). This is a very nice feature, but also a necessary one – again, AngularJS is a pure client side framework.

With great power…

React changes this equation because it can be rendered on the server. Similarly to AngularJS, React root component can be mounted to any node inside your page, but unlike Angular, this page can also be rendered on the server by React.

This opens the door for a whole new class of isomorphic or universal apps, and I have written about it already. What you need to decide is whether you want to render the entire page, or a subset of it. If you choose the latter, then the subsequent question becomes – how do you start the rendering process until React takes over?

My first inclination was to go full React. Using the awesome react-engine module, it is easy to configure React as the view engine in the Express framework, and render the entire page. However, we soon hit some snags:

  1. As we render our UIs using micro services that use UI composition to glue the pages together, we hit a problem of React being grumpy about having to render HTML snippet arriving from outside of its control – something much easier with a templating library such as Dust.js.
  2. JSX turned out to be fairly quirky and finicky when rendering HTML HEAD element, as well as trying to inline JavaScript in tags. In fact, the latter became so error prone that we completely gave up on it, electing to put all our script in files. I didn’t like this particular restriction, and this remains an area where I am sour on JSX and the fact it most decidedly isn’t HTML.

Of course the problems I listed above did not show up in the small example I published in the initial article. They reared their ugly head once we started getting serious. In my original example, the header was a React component. This works great if you control the entire app, but the whole deal of UI composition is to integrate with apps not necessarily written using React, and you can only do that using the least common denominator (HTML, CSS and vanilla JS).

Remind me again about UI composition

In a nutshell, UI composition is the approach where common areas on the page are served as HTML snippets from a REST API. The API service can also serve CSS and JavaScript files that accompany the snippet (if you don’t serve them from a CDN). The idea is that a dedicated microservice can be providing these common areas this way, allowing other microservices to pull the content and inline it in their pages regardless of their stack.

UI composition inverts the flow you would normally use with iframes. Instead of rendering the common areas, then placing an iframe for the content and loading the content from an external URL, we load the common area as HTML and inline it into our page. In the olden days this approach was often accomplished using ‘edge side includes’ or ESIs. This approach has many benefits, including no need for the dreaded iframes, full control over the entire page, and the ability to integrate microservices implemented using different stacks. For example, the teams in my current project use:

  1. Node.js microservices using React for isomorphic rendering
  2. Node.js microservices using Dust.js and jQuery
  3. Java microservices using AngularJS

All of these microservices render the exact same header even though they use different stacks and libraries. Powerful stuff.

OK, so React?

The problem here with React is that when we use it to render the entire page server-side, it starts to squirm in discomfort. Inlining HTML is a problem for React because, while it can be used on the server, it was designed to shine on the client. Inlining raw HTML presumably obtained as a user input into DOM is a major security exposure without sanitization, and React will tell you so in no uncertain terms. Here is the line that inlines the shared header taken straight out of our production code:

<div dangerouslySetInnerHTML={{__html: decodeURIComponent(this.props.header)}} />

This is React telling us very clearly: “I really, really, really don’t like what you are asking me to do”. And this is even before we bring in react-engine. React engine is a PayPal module that enables isomorphism when combined with Node.js/express and react-router. We can render on the server, then pack the properties as part of the HTML response, hydrate the React components on the client and continue where we left off.

Only we have a little problem: notice that in the snippet above we have inlined the header by passing it down to the React component as a prop. What this means is that this prop (‘header’) will be sent to the client alongside other properties. The only problem is that this property contains HTML snippet for the entire header, and because it can really mess up react-engine, it is also URLencoded, making it even bulkier.

This is an unhappy situation: first we inline the header HTML (which is fine), then we send that same inlined HTML, this time encoded, as a React prop to the client. This unnecessarily bloats the generated HTML.

RDV

This is what we are going to do to solve this problem: we are going to call back our trusty Dust.js sitting sad on the sidelines ever since we got smitten with React. We can press it into service to do what it does best – outline the page, prep it up to the point where React can take over.

It’s a good thing that Express framework will happily handle more than one view engine. Of course, only one of them can be registered as default, but this just means that we will need to spell out the extension of dust files (because React is the default). Big deal.


var path = require('path')
, express = require('express')
, renderer = require('react-engine')
, dust = require('dustjs-linkedin')
, helpers = require('dustjs-helpers')
, cons = require('consolidate');

...

// create the view engine with `react-engine`
var engine = renderer.server.create({
  routes: require(path.normalize(__dirname + '/public/routes.jsx')), 
  docType: ""
});

// set the engines
app.engine('.jsx', engine);
app.engine('.dust', cons.dust);

// set the view directory
app.set('views', __dirname + '/public/views');

// set jsx as the view engine
app.set('view engine', 'jsx');

// finally, set the custom view
app.set('view', renderer.expressView);

Once we have both engines set up, we can arrange our delivery vehicle like this. We will first set up a reusable layout template using Dust.js that will deliver the outline of each page:

<!DOCTYPE html>
<html>
 <head>
   <meta charset='utf-8'>
   <meta http-equiv="X-UA-Compatible" content="IE=edge">
   <meta name="viewport" content="width=device-width, initial-scale=1" >
   {+headContent/}
   <title>
     {title}
   </title>
   <link rel="stylesheet" href="/css/styles.css">
 </head>
 <body>
   {header|s}
   <div class="main-content">
     {+mainContent/}
   </div>
   {+javascript/}
 </body>
</html>

Notice how inlining of the header now happens in Dust.js, relieving React from doing it. We can now use this template to render our page like this:

{>"layout"/}
{<mainContent}
<div id="react-mount-node">{spa|s}</div>
{/mainContent}
{<javascript}
<script src="/bundle.js"></script>
{/javascript}

We have created a DIV in the template above where we are mounting React, and we will inline the React server-side content using the ‘spa’ variable.

Our Express controller that will handle both Dust.js and React can look something like this:

var request = require("request");

module.exports.get = function(req, res) {
  var settings = {
    title: 'SPA - Demo',
    name: 'React SPA',
    selection: 'header-spa'
  };

  var headerUrl = "http://"+req.headers.host+"/header?selection=header-spa";

  request.get(headerUrl, function (err, response, body) {
    if (err)
      console.log(err);
    else
      settings.header = body;
    res.render(req.url, { name: settings.name }, function(err, html) {
      if (err) {
        settings.spa = err.message;
        console.log(JSON.stringify(err.stack));
      }
      else
        settings.spa = html;
      res.render("spa.dust", settings); 
    });
  });
}

The code above is what I call ‘React Delivery Vehicle’. It has three steps:

  1. First we fetch the header from the URL that is providing us with the header HTML snippet. We capture it into a variable ‘header’. In production, we will heavily cache this step.
  2. Then we render the React root component as usual. This will employ react-engine and react-router to render all the views necessary for the provided request URL. However, we don’t send the rendering to the response. Instead, we capture it in the variable ‘spa’.
  3. Finally, we invoke Dust.js view engine to render the full page, passing in the ‘settings’ object. This object will contain both the header and the content rendered by React. Dust will simply inline both while rendering the page outline. The result will be sent directly to the Express response.

Commentary

The solution described above plays to the strengths of both Dust.js as a server side template engine and React as a component library. Dust.js is great at rendering HTML outline the way you would expect, with no need to worry about JSX quirks around HEAD meta tags, or JavaScript inlining. React takes over to render isomorphic components into the mount node. This means we don’t need to send any data that is not needed on the client. This fixes the HTML bloat problem I mentioned before.

As for the negatives, setting up two rendering engines on the server has a slight overhead, and switching mentally from Dust.js to JSX adds context switching tax. Luckily, you can set up reusable Dust.js templates and not really worry about them too much – most of the action will be in JSX anyway.

We like this approach and are currently switching to it across the board. If you have alternative ideas or comments, drop me a line. Meanwhile, the source code for the entire example is available on GitHub as usual, and the sample app employing this mechanism runs on Bluemix.

© Dejan Glozic, 2015

Advertisements

ReactJS: The Day After

A man with an excruciating headache, Wikimedia Commons
A man with an excruciating headache, Wikimedia Commons

The other day I stumbled upon a funny Onion fake news report of the local man whose one-beer plan went terribly awry. Knowing how I professed undying love to ReactJS in the previous article, and extrapolating from life that after every night on the town comes the morning of reckoning, it is time to revisit my latest infatuation.

Alas, those expecting me to declare my foolishness and heartbreak with ReactJS are hoping in vain. Instead, what you will get here is a sober (ha) account of the problems, gotchas and head scratchers we encountered running ReactJS in production. We continue to use it and plan to build our next set of micro services using it, but we have a more realistic view of it now. So let’s dive in.

  1. Code Splitting – First off, my example didn’t just use ReactJS, but also react-router and react-engine. This amazing trio together allowed us to realize the dream of isomorphic apps, where you start rendering on the server, let the browser quickly render the initial content, load JavaScript, mount React components and continue with the same code on the client.
    Nevertheless, when we got past the small example, we realized that we need to split the code we initially bundled together using browserify. At the time of this writing, code splitting is not entirely painless. React-router in its version 0.13 has examples that all presume the use of Webpack to build your JavaScript. We are using browserify and must suffer until React-router 1.0 arrives. In the mean time, we can use react-router-proxy-loader, which allows us to asynchronously load code from a bundle that does not expect Webpack.

  2. React-engine growing pain – As any new library, react-engine has some rough edges. We are happy to report that one of the issues we had with it (the inability to control how react-router is being instantiated) has already been resolved. We are hoping to be able to make react-engine omit some of the data it sends to the client because it is only ever used for server-side rendering.
  3. ReactJS id properties – React attaches ‘reactid’ data property to almost all DOM elements, using ids that are sometimes very long, resulting in situations like:
    <span data-reactid=".ejv9lnvzeo.1.2.3.0.0.$7c87c148-e1a4-4cb8-81f8-c5e74be7684b.0.1.0.0">Hello</span>
    

    If you are using gzip for the markup (as you should), these strings compress very well, but you still end up with a very messy and hard to read HTML when you view source. React team is debating back and forth on the need of these properties and they may disappear at some point in the future. I for one will not miss them.

  4. Fussy with the whitespace – While you may think when working with JSX that you are coding in HTML, you are not, and nowhere is it more apparent than when you try to add some free text in the body of HTML elements, or to mix free text and elements. JSX converts snippets of text into spans at will, resulting in HTML that bears little resemblance to the initial JSX.
    I wish there was a better way to do this. I know all the virtues of React and how JSX is most decidedly not HTML, but some things like free-form text with some embedded tags should not result in a flurry of spans (and the hated data-reactid properties).
  5. Fussy with JavaScript tags – Inserting JavaScript tags in JSX is easy if you are referencing external JS files, but if you try to inline some JavaScript right there, JSX can through you curveball after curveball until you give up and extract that code into a file. This is not a show stopper but it is annoying when you want to inline couple of lines. From the maintainability point of view, it is probably better to keep JavaScript in its own file, so I am not going to protest too loudly.

ReactJS and Web Components

As with any JS framework, making a choice is normally followed by a little nagging voice in your head concerned that you chose wrong. When it comes to religious choices (AngularJS vs ReactJS vs EmberJS etc.), there is little you can do – you just need to make a leap of faith, make sure the framework works for your particular use case and jump.

However, Web Components are something else – they promise to be ‘the native Web’ at some point, so choosing between Web Components and ReactJS is not a religious debate. Even today, with the shims it is possible to run Web Components in browsers not supporting them natively, and natively in Chrome. A growing body of reusable Web components is something you don’t want to be left out of if you are Reactified to the max.

Luckily, Andrew Rota helped out with his presentation on complementarity of ReactJS and Web Components at the recent ReactJS Conf 2015. It is worth the watch, and the skinny is that since about October 2014, custom components are a fair game in JSX. This means that you can place HTML imports in the head element, and then freely use custom components in JSX the same way you would native HTML elements.

In fact, you are not loosing out on the promise of ReactJS virtual DOM. React treats custom components the same way as native HTML components – it will compare your new render to the current DOM state and only change what needs changing (adding, removing, or changing elements and properties that are not the same). This means that you can extend the power of ReactJS to Web Components.

Of course, there are some caveats, but it turns out that things you need to care about when writing Web Components for ReactJS consumption are generally applicable. Writing small components, extremely well encapsulated, that do not leak or make assumptions about the page they are running in, or try to insert stuff outside their own boundary.

No turning back

So this turned to be a click bait of sorts, for we are not turning back from ReactJS, just learning how to use it efficiently and how to be better at it. Stay tuned for the new cool stuff we were able to do with it.

© Dejan Glozic, 2015

PayPal, You Got Me At ‘Isomorphic ReactJS’

Juvet_mirror_01

I love PayPal’s engineering department. There, I’ve said it. I have followed what Jeff Harrell and the team have been doing ever since I started reading about their wholesale jump into Node.js/Dust.js waters. In fact, their blogs and presentations finally convinced me to push for Node.js in my IBM team as well. I had a pleasure of talking to them at conferences multiple times and I continue to like their overall approach.

PayPal is at its core a no-nonsense enterprise that moved from Java to Node.js. Everything I have seen coming from them had the same pragmatic approach, with concerns you can expect from running Node.js in production – security, i18n, converting a large contingent of Java engineers to Node.js.

More recently, I kept tab on PayPal’s apparent move to from Dust.js to ReactJS. Of course, this time around we learned faster and were already playing with React ourselves (using Dust.js for simpler, content-heavy pages and reserving ReactJS for more dynamic use cases). However, we haven’t really started pushing on ReactJS because I was still looking at how to take advantage of React’s ability to render on the server.

Well, the wait is over. PayPal has won my heart again by releasing a React engine that connects the dots in a way so compatible with what we needed that made me jump with joy. Unlike the version I used for my previous blog post, this one allows server side components to be client-mountable, offering true isomorphic goodness. Finally, a fat-free yogurt that does not taste like paper glue.

Curate and connect

The key importance of PayPal’s engine is in what it brings together. One of the reasons React has attracted so much attention lately is its ability to render into a string on the server, then render to the real DOM on the client using the same code. This is made possible by using NodeJS, which is by now our standard stack (I haven’t written a line of Java code for more than a year, on my honour).

But that is not enough – in order to carry over into the client with the same code, you need ‘soft’ page switching – showing boxes inside other boxes and updating the browser history as these boxes are swapped. This has been brought to us now by another great library – react-router. This module inspired by Ember’s amazing router is quickly becoming ‘the’ router for React applications.

What PayPal did in their engine was connect all these great libraries, then write important glue code to put it all together. It is now possible to mix normal server side templates with pages that start their life on the server, then continue on the client with the state preserved and ready to go as soon as JavaScript is loaded.

Needless to say, this was the solution we were looking for. As far as we are concerned, this will put an end to needless ‘server vs client’ wars, and allow us to have our cake and eat it too. Mmm, cake.

Show us some sample code

OK, let’s get our hands dirty. What many people need when writing applications is a hybrid between a site and an app – the ability to put together a web site that has one or more single-page apps embedded in it. We will build an example site that has two plain ReactJS pages rendered on the server, while the third page is really an SPA taking advantage of react-engine and the ability to go full isomorphic.

We will start by creating a shared layout JSX component to be consumed by all other pages:

var React = require('react');
var Header = require('./header.jsx');

module.exports = React.createClass({

  render: function render() {
    var bundle;

    if (this.props.addBundle)
      bundle = <script src='/bundle.js'/>;

    return (
      <html>
        <head>
          <meta charSet='utf-8' />
          <title>
            {this.props.title}
          </title>
          <link rel="stylesheet" href="/css/styles.css"/>
        </head>
        <body>
          <Header {...this.props}></Header>
          <div className="main-content">
             {this.props.children}
          </div>
        </body>
        {bundle}
      </html>
    );
  }
});

We extracted common header as a separate component that we required and inlined:

var React = require('react');

module.exports = React.createClass({

  displayName: 'header',

  render: function render() {
    var linkClass = 'header-link';
    var linkClassSelected = 'header-link header-selected';

    return (
      <section className='header' id='header'>
        <div className='header-title'>{this.props.title}</div>
        <nav className='header-links'>
          <ul>
            <li className={this.props.selection=='header-home'?linkClassSelected:linkClass} id='header-home'>
              <a href='/'>Home</a>
            </li>
            <li className={this.props.selection=='header-page2'?linkClassSelected:linkClass} id='header-page2'>
              <a href='/page2'>Page 2</a>
            </li>
            <li className={this.props.selection=='header-spa'?linkClassSelected:linkClass} id='header-spa'>
              <a href='/spa/section1'>React SPA</a>
            </li>
          </ul>
        </nav>
      </section>
    );
  }
});

The header shows three main pages – ‘Home’, ‘Page2’ and ‘React SPA’. The first two are plain server side pages that are rendered by express and sent to the client as HTML:

var Layout = require('./layout.jsx');
var React = require('react');

module.exports = React.createClass({

  render: function render() {

    return (
      <Layout {...this.props}>
        <h2>Home</h2>
        <p>An example of a plain server-side ReactJS page.</p>
      </Layout>
    );
  }
});

On to the main course

The third page (‘React SPA’) is where all the fun is. Here, we want to create a single-page app so that when we navigate to it by clicking on its link in the header, all subsequent navigations inside it are client-side. However, true to our isomorphic requirement, we want the initial content of ‘React SPA’ page to be rendered on the server, after which react-router and React component will take over.

To show the potential of this approach, we will build a very useful layout – a page with a left nav containing three links (Section 1, 2 and 3), each showing different content in the content area of the page. If you have seen such a page once, you saw it a million times – this layout is internet’s bread and butter.

We start building our SPA top-down. Our top level ReactJS component will reuse Layout component:

var Layout = require('./layout.jsx');
var React = require('react');
var Nav = require('./nav.jsx');
var Router = require('react-router');

module.exports = React.createClass({

  render: function render() {

    return (
      <Layout {...this.props} addBundle='true'>
        <Nav {...this.props}/>
        <Router.RouteHandler {...this.props}/>
      </Layout>
    );
  }
});

We have loaded left nav as a Nav component:

var React = require('react');
var Link = require('react-router').Link;

module.exports = React.createClass({

  displayName: 'nav',

  render: function render() {
    var activeClass = 'left-nav-selected';

    return (
      <section className='left-nav' id='left-nav'>
        <div className='left-nav-title'>{this.props.name}</div>;
        <nav className='left-nav-links'>
          <ul>
            <li className='left-nav-link' id='nav-section1'>
              <Link to='section1' activeClassName={activeClass}>Section 1</Link>
            </li>
            <li className='left-nav-link' id='nav-section2'>
              <Link to='section2' activeClassName={activeClass}>Section 2</Link>
            </li>
            <li className='left-nav-link' id='nav-section3'>
              <Link to='section3' activeClassName={activeClass}>Section 3</Link>
            </li>       
          </ul>
        </nav>
      </section>
    );
  }
});

This looks fairly simple, except for one crucial difference: instead of adding plain ‘a’ tags for links, we used Link components coming from react-router module. They are the key for the magic here – on the server, they will render normal links, but with ‘breadcrumbs’ allowing React router to mount click listeners on them, and cancel normal navigation behaviour. Instead, they will cause React components registered as handlers for these links to be shown. In addition, browser history will be maintained so that back button and address bar works as expected for these ‘soft’ navigations.

Component RouteHandler is responsible for executing the information specified in our route definition:

var Router = require('react-router');
var Route = Router.Route;

var SPA = require('./views/spa.jsx');
var Section1 = require('./views/section1.jsx');
var Section2 = require('./views/section2.jsx');
var Section3 = require('./views/section3.jsx');

var routes = module.exports = (
  <Route path='/spa' handler={SPA}>;
    <Route name='section1' handler={Section1} />;
    <Route name='section2' handler={Section2} />;
    <Route name='section3' handler={Section3} />;
	<Router.DefaultRoute handler={Section1} />;
  </Route>
);

As you can infer, we are not declaring all the routes for our site, just the section for the single-page app (under the ‘/spa’ path). There we have built three subpaths and designated React components as handlers for these routes. When a Link component whose ‘to’ property is equal to the route name is activated, the component designated as handler will be shown.

Server needs to cooperate

In order to get our HTML5 push state enabled router to work, we need server side cooperation. In the olden days when SPAs were using hashes to ensure client side navigation is not causing page reloading, we didn’t need to care about the server because hashes stayed on the client. Those days are over and we want true deep URLs on the client, and we can have them using HTML5 push state support.

However, once we start using true links for everything, we need to tell the server to not try to render pages that belong to the client. We can do this in express like this:

app.get('/', function(req, res) {
  res.render('home', {
    title: 'React Engine Demo',
    name: 'Home',
    selection: 'header-home'
  });
});

app.get('/page2', function(req, res) {
  res.render('page2', {
    title: 'React Engine Demo',
    name: 'Page 2',
    selection: 'header-page2'
  });
});

app.get('/spa*', function(req, res) {
  res.render(req.url, {
    title: 'SPA - React Engine Demo',
    name: 'React SPA',
    selection: 'header-spa'
  });
});

Notice that we have defined controllers for two routes using normal ‘res.render’ approach, but the third one is special. First off, we have instructed express to not try to render any pages under /spa by sending them all to the React router. Notice also that instead of sending normal view names in res.render, we are passing entire URL coming from the request. This particular detail is what makes ‘react-engine’ ingenious – the ability to mix react-router and normal views by looking for the presence of the leading ‘/’ sign.

A bit more boilerplate

Now that we have all these pieces, what else do we need to get this to work? First off, we need the JS file to configure the react router on the client, and start the client side mounting:

var Routes = require('./routes.jsx');
var Client = require('react-engine/lib/client');

// Include all view files. Browserify doesn't do
// this automatically as it can only operate on
// static require statements.
require('./views/**/*.jsx', {glob: true});

// boot options
var options = {
  routes: Routes,

  // supply a function that can be called
  // to resolve the file that was rendered.
  viewResolver: function(viewName) {
    return require('./views/' + viewName);
  }
};

document.addEventListener('DOMContentLoaded', function onLoad() {
  Client.boot(options);
});

And to round it all, we need to deliver all this JavaScript, and JSX templates, to the client somehow. There are several ways to approach JavaScript modularization on the client, but since we are using Node.js and sing the isomorphic song, what can be more apt than using Browserify to carry over CommonJS into the client? The following command line will gather the entire dependency tree for index.js into one tidy bundle:


browserify -t reactify -t require-globify public/index.js -o public/bundle.js

If you circle back all the way to Layout.jsx, you will notice that we are including a sole script tag for /bundle.js.

The complete source code for this app is available on GitHub. When we run ‘npm install’ to bring all the dependencies, then ‘npm start’ to run browserify and start express, we get the following:

react-engine-demo

When we click on header links, they cause full page reload, rendered by express server. However, clicks on the left nav links cause the content of the SPA page to change without a page reload. Meanwhile, the address bar and browser history is dutifully updated, and deep links are available for sharing.

Discussion

You can probably tell that I am very excited about this approach because it finally brings together fast initial rendering and SEO-friendly server-side pages, with full dynamic ability of client side apps. All excitement aside, we need to remember that this is just views – we would need to write more code to add action dispatcher and data stores in order to implement full Flux architecture.

Performance wise, the combined app renders very quickly, but one element sticks out. Bundle.js in its full form is about 800KB of JavaScript, which is a lot. When running the command to minify it, it is trimmed down to 279KB, and when compression is enabled in express, it goes further down to 62.8KB to send down the wire. We should bear in mind that this is ALL JavaScript we need – ReactJS, as well as our own components. It should also be noted that this JavaScript is loaded asynchronously and that we are sending content from the server already – we will not see a white page while script is being downloaded and parsed.

In a more complex application, we would probably want to segregate JavaScript into more than one bundle so that we can load code in chunks as needed. Luckily, react-router already addressed this.

The app is deployed on Bluemix – you can access it at http://react-engine-demo.mybluemix.net. Try it out, play with the source code and let me know what you think.

Great job, PayPal! As for me, I am completely sold on ReactJS for most real world applications. We will be using this approach in our current project from now on.

© Dejan Glozic, 2015