Beam My Model Up, Scotty!

star-trek-transporter

I know, I know. Scotty is from the original series, and the picture above is from TNG. Unlike Leonard from The Big Bang Theory, I prefer TNG over the original, and also Picard over Kirk. Please refrain from hate mail.

Much of both real and virtual ink has been spilled over Star Trek transporter technology and quirks. Who can forget the episode when Scotty has preserved himself as a transporter pattern in an endless loop only to be found and freed 70 years later by the TNG crew (see, there’s your Scotty-TNG link). But this article is about honouring all the hours I spent on various Star Trek instalments by applying the transporter principles to Web development.

As the Web development collective mind matures, a consensus is forming that spaghetti are great on your dining table but bad for your code. MVC design pattern on the server is just accepted as gravity, and several JavaScript frameworks are claiming it is necessary on the client as well. But is it always?

Here is case in point. Our team recently rebuilt a dashboard – a relatively complex peace of interactive technology. This is our third attempt at it (third time’s the charm, right)? Armed with all the experience of the previous two versions, plus our general desire to pull back from the extreme Ajax, we were confident this would be it. Why build a dashboard? Yes, I know, everybody and his uncle has one, and there are perfectly good all singing, all dancing commercial ones to be had. Here is a counter-question: why do you have a kitchen in your house when you can eat at perfectly good restaurants? Everybody has a dashboard because it is much more convenient to have your own (Google Analytics has one, WordPress I am writing this on has one). They can be simple because they don’t need to cater to all the possible scenarios, and much more tightly integrated. And not to mention cheaper (like eating at home vs. eating out). But I digress.

In the previous version, we sent a lot of JavaScript to the client, and after transporting and parsing it, JavaScript turned around and used XHR to fetch the model from the server as JSON. Then we wired up the model and the view and every time users made a change, we would update the model to keep it in sync. Backbone.js or Angular.js would have come in handy for this, only they didn’t exist when we wrote that code. Plus we hate frameworks.

In this version we wanted the dashboard page to arrive mostly assembled from the server. That part was clear. But we wanted to avoid the MV* complexity and performance consequences if possible. Plus it is really hard to even apply the usual data binding with dashboards because you cannot just wire the template with the model and let the framework do its magic. In a dashboard, the template itself is editable – users can drag and drop widgets around, delete them and add new ones. Essentially both the template and the data are built up from the model. This is not your grandmother’s MVC, that’s for sure.

Then we remembered Star Trek transporters and figured – why don’t we disperse the model and embed the model shards into the DOM as we are building the initial HTML on the server? We were waiting to hit a brick wall at some point, but it didn’t happen – this is an actually viable option. Here is why this approach (I will call it M/V) works for us like a charm:

  1. Obviously it is easy to do on the server – when we are building the initial response, embedding model shards into the HTML is trivial using HTML5 custom data properties (those attributes that start with ‘data-‘).
  2. The model arrives ready to use on the client – no need for an extra request to fetch the model after the fact. OK, Backbone.js has a way of bootstrapping models so that they arrive as payload with the initial response. However, that approach qualifies as CrazyHacks™ at best, and you save nothing payload-wise – the sum of the shards equals one whole model.
  3. When the part of the DOM needs to change due to the user interaction, it is easy to keep the associated ‘data-*’ properties in sync.
  4. When moving entire DOM branches (say, when dragging widgets between columns or changing layouts), model shards stay with their DOM elements – this is a real winner in our use case.
  5. We use DOM custom event bubbling to notify about the change – one listener at the top of the DOM branch is all that is needed to keep track of the sharded model’s dirty state. This helps us to stay lean because there is no need for JavaScript pub/sub implementations that increase size. It is laughably trivial to fire and listen to these events using jQuery, and they are fast, being implemented natively by the browser. Bootstrap uses the same approach.
  6. When the time comes to save, we traverse the DOM, collect the shards and assemble them again into a transient JavaScript object, then send it as JSON to the server via XHR. We use PUT/POST when we have the entire model, but more often the not, the view only renders a fraction of it (lazy loading), so we use PATCH instead.

This approach bought us some unreal savings in performance and size. Right now, we brought down total JavaScript from ~245kB to ~70KB minified gzipped (this includes jQuery, Require.js and Bootstrap). The best part is that this JavaScript loads asynchronously, after the initial content has already been displayed. The page is not only objectively fast, it is subjectively even faster because the initial content (what Twitter team calls ‘time to first tweet’) is sent in the first response.

As it is often the case in life, this approach may not work for everybody. A pattern where true MVC really shines is when you have multiple views listening to the same model. Changing the model updates all the views simultaneously, without the need for the pub/sub hell. But if you have exactly one view and one model, using M/V becomes a real option.

I can hear the collective gasp of architects in the audience. There are many things you can say about M/V, but architecturally pure it is not. But here is the problem. Speed and page size should be a valid architectural concern too. In fact, if you want your Web app to be usable on mobile devices, it is practically the only concern that matters. Nobody is going to wait your architecturally pure pig of a page to load while waiting for coffee – life is too short for 1MB+ web pages, pure or not.

Say No to layer cake, embrace small, live long and prosper!

© Dejan Glozic, 2013

Lean Startup Inc.

Y2K was a banner year for behavioral scientists. In the years leading up to the Internet bubble bust, most investors reported on their profiles that their risk tolerance was ‘high’. At that time, they assumed that the risk meant that they can get an even higher return on their speculative investments than they have hoped for in their greedy minds. Then came Y2K and the realization that ‘risk’ is a double-edged sword, and both sides are very, very sharp. Many investors got so burnt that for more than a decade they reacted to the stock market the way Bela Lugosi reacted to garlic (now Bauhaus told me in my youth that Bela Lugosi’s dead, but there will never be another Dracula for me but Bela).

The Lean Startup movement I already mentioned in one of my previous posts is meant first and foremost for actual startups. It applies perfectly to a couple of founders, a few first employees rapidly burning through Angel Investors’ money, walking around with starry eyes, dreams of becoming ‘the next Facebook’ and drawing a short list of islands they will buy once that happens. It is a mad race for getting to the business end of the ‘hockey stick’ chart – the part where revenue takes off to actually finance the business (or, in the case of Snapchat, becomes greater than zero). The challenge is to reach this point before venture capital runs out and the repo men cometh.

An expansive aspect of the methodology is that it can be applied to any new effort amid extreme uncertainty, even in huge multinationals or governments. Any corporate team attempting to start a version 1.0 of a product or a service is effectively a startup. Instead of VCs, there are executive sponsors, and instead of actual money, there is headcount – number of people working on the project until business declares that time’s up.

I am a big fan of Michael Lopp and his alter ego Rands. I followed his blog for years, an even bought his book Being Geek. In the chapter called ‘A Deliberate Career’, he cited a ‘third option’, apart from a startup and an established company:

There’s a constant threat in a start-up, and that’s the threat of failure. You can ignore it when you’re busily working three weekends straight, but it’s always there: “We could fail.” The larger company’s success has hidden this threat under a guise of predictability, domesticity, and sheer momentum.
…..
Still, you can find the same [start-up] attributes in a large company in a specific group that has been tasked with the new and sexy.

It appears that working on an exciting new project in an established company is a win-win situation – all the fun of a startup but no risk. But there is a bug here. Fear is a powerful motivator. Remove it, and everything slows down, because if the cheque clears like clockwork, what’s the hurry?

In my recent post on client side frameworks, I kept mentioning Angular.js. On the surface, it seems to fit the description of the ‘best of both worlds’ project – Open Source exposure backed by security of Google. Then you have to remember that Google uses spaghetti approach to their services – throw a plateful towards the wall, and see which one will stick. So Angular.js is great, unless your team works on Closure or GWT or Dart, and you know that at any point in time the axe can fall and people will be leaving flowers for it in the graveyard of Google Services. That’s enough fear for me, thank you.

A friend from my youth went to spend some time in Greece. He didn’t have a lot of money and the beautiful island of Santorini was not exactly cheap, so he had to do all kinds of jobs to support himself. He told me and my wife something that stuck:

Not knowing where your next meal will come from does wonders to focus your mind.

Since lack of fear means lack of alertness, sense of urgency and get-go, teams working on new projects in large companies often lose sight of the possibility that the project may not result in an actual product. Your mind is playing tricks with you, similar to the DotCom investors who had ‘high risk tolerance’. Large companies don’t fail, you say. And you are right – not in the way startups do. With no buffer and no sheer mass to amortize the blow, any kind of snag can knock you down if you are a startup. Still, in a big company you can fail to see your vision through to completion, and see the joy of customers actually finding your product useful. You may still have your job, but unless you are in it just for the paycheque and your true passion lies elsewhere, that’s got to hurt a bit. And if your heart is not in it, what exactly are you doing all day? Checking Facebook? Laughing at Doge pictures?

A much healthier approach for you if you want to continue to be on the bleeding edge is to accept that all startups can fail, even those in big companies (in their own soft way). The fact that in the latter case the salary keeps landing in your bank account already puts you at an advantage over most of Silicon Valley. Accept both sides of risk, not just the good one, accept that ‘extreme uncertainty’ is a serious business (unlike extreme ironing), and act accordingly.

Oh, and read The Lean Startup book carefully. You will find plenty of examples of tactical failures along the way – failures to accurately predict customer base, feature set, the essence of the product value add, customer interest. While pivots are a normal part of the startup experience, they can be truly unsettling to a corporate developer used to stability and predictability.

Even in a big corporation, you cannot have it both ways. Either be truly lean, agile and ready for all kinds of curve balls, ready to turn on a dime, or move to a more established project already shipping products or services, where incremental innovation is more important than starting from scratch for the third time. In both real and corporate startups, if you are not afraid, it’s not a real startup.

© Dejan Glozic, 2013

Swimming Against The Tide

True story: I visited Ember.js web site and saw three required hipster artifacts: ironic mustaches, cute animals and Ray-Ban Wayfarer glasses (on a cute animal). A tweet was in order and within minutes, it was favored by a competing client side framework by Google (Angular). Who would have guessed client side frameworks are so catty? I can almost picture Angular News guy clicking the ‘Favorite’ button and yelling ‘Oh, Burn!!’ And it wasn’t even a burn, I actually like Ember web site – it is so …. cute.

The reason I visited Ember (and Angular, and Backbone, and Knockout) was to figure out what was going on. There is this scene in a 2002 movie Gangs Of New York where Leonardo DiCaprio leads his gang of Dead Rabbits to fight the competing gang (the Natives), and he has to wade through a river of people running in the opposite direction to avoid cannons fired by the Navy from the harbor. Leonardo and his opponent, a pre-Lincoln Daniel Day-Lewis were so enthralled in their epic fight that they missed the wider context of the New York Draft Riots happening around them. Am I like Leo (minus the looks, fame and fortune), completely missing the wider historic context around me?

Not long ago, I posted a repentant manifesto of a recovered AJAX addict. I swore off the hard stuff and pledged to only consume client-side script in moderation. The good people from Twitter and 37Signals all went through the same trials and tribulations and adopted similar approach (or I adopted theirs). Most recently, Thomas Fuchs, the author of Zepto.js expressed a similar change of heart based on his experiences in getting the fledgling product Charm off the ground. Against that backdrop, the noise of client-side MVC frameworks mentioned above is reaching deafening levels, with all the people apparently not caring about the problems that burned us so much. So what gives?

There are currently two major camps in the Web development right now, and they mostly differ in the role they allocate for the server side. The server side guys (e.g. Twitter, Basecamp, Thomas, yours truly) have been burned by heavy JavaScript clients and want to render initial page on the server, subsequently using PJAX and modest amounts of JavaScript for delicious interactivity and crowd pleasers. Meanwhile, a large population of developers still want to develop one page Web apps that require the script to take over from the browser for the long periods of time, relegating the server to the role of a REST service provider. I don’t want to repeat myself here (kindly read my previous article) but the issues of JavaScript size, parsing time, performance, memory leaks, browser history, SEO didn’t go away – they still exist. Nevertheless, judging by the interest in Angular, Backbone, Ember and other client side JavaScript frameworks, a lot of people think the tradeoffs are worth it.

To be correct, there is a third camp populated mostly by LinkedIn engineering team. They are in a category for themselves because they are definitely not a one page app, yet they do use Dust.js for client side rendering. But they also use a whole mess of in-house libraries for binding pages to services, assembling them, delaying rendering when below the fold, etc. You can read it on their blog – suffice to say that similar to Facebook’s Big Pipe, the chances you can repeat their architecture in your project are fairly slim, so I don’t think their camp is of practical value to this discussion.

Mind you, nobody is arguing the return to the dark ages of Web 1.0. There is no discussion whether JavaScript is needed, only whether all the action is on the client or there is a more balanced division of labor with the server.

I thought long and hard (that is, couple of days tops) about the rise of JavaScript MVC frameworks. So far, this is what I came up with:

  1. Over the last few years, many people have written a lot of crappy, unmaintainable, messy jumble of JavaScript. They now realize the value of architecture, structure and good engineering (client or server).
  2. A lot of people realize that some really smart people writing modern JavaScript frameworks will probably do a better job providing this structure then themselves.
  3. Many projects are simply not large enough to hit the general client side scripting turning point. This puts them in a sweet spot for client side MVC – large enough to be a mess and benefit from structure, not large enough to be a real pig that makes desktop browsers sweat and mobile browsers kill your script execution due to consuming too much RAM.
  4. These projects are also not easily partitioned into smaller contexts that can be loaded as separate Web pages. As a result, they rely on MVC JavaScript frameworks to perform data binding, partitioning, routing and other management.
  5. Modern templating engines such as Mustache or Handlebars can run both on the client and on the server, opening up the option of rendering the initial page server side.
  6. JavaScript community is following the same path that Web 1.0 server side MVC went through: the raise of opinionated and prescriptive MVC frameworks that try to box you into good practices and increase your productivity at the price of control and freedom.
  7. The people using these frameworks don’t really have performance as their first priority.
  8. The people using these frameworks plan to write a separate or native mobile client.

There could be truth in this, or I could be way off base. Either way, my team has no intention of changing course. To begin with, we are allergic to loss of control these frameworks demand – we subscribe to the camp that frameworks are bad. More importantly, we like how snappy our pages are now and want to keep them that way. We intend to keep an eye on the client MVC frameworks and maybe one day we will hit a use case where client side data binding and templates will prove useful (say, if we attempt something like Gmail or Google Docs or Google Calendar). If that happens, we will limit it to that particular use case, instead of going all in.

Meanwhile, @scottjehl perfectly describes my current state of mind thusly:

output-HTML

© Dejan Glozic, 2013

The 12 Amp Limit

640px-Ampere-o-meter-vintage-HDR-0h

Sometimes movies I watch leave a lasting impression for completely unexpected reasons. Case in point – 1995 hit “Apollo 13”. The movie offered plenty of fodder for people who like to play Six Degrees of Kevin Bacon, and offered Tom Hanks in a clean 1970 NASA style haircut. A lot easier to take than the abomination on top of Robert Langdon’s head. Of course, the movie is packed with nail-biting scenes, if you bite your nails even if you know perfectly well that they are going to make it in the end. But that is not why I remember the movie.

Somewhere in the second half, the crew on the ground is trying to devise a plan of onboard system startup that is not going to suck all the juice from the remaining batteries. The battered ship’s power supply can only take up to 12 amps of current before it cries Uncle, and they have a nice analogue instrument with a needle to watch it creep up. Gary Sinise starts clean, carefully adds one by one module and before you know it, 12 amp limit is hit and the metaphorical crew is dead. Ultimately, the solution was found not in the proper startup sequence but in adding another partially shot battery useless on its own, but giving just the needed push when hooked up into the circuit as a booster.

I keep thinking of this scene looking back at software development projects I was on. It all starts new and shiny – real and metaphorical junk cleaned up, new ground broken, everything seems possible, we make crazy progress unencumbered by restrictions and customers. But soon enough, code piles up, requirements keep coming, constraints set in and before you know it, you hit the limit. It could be too many features, too much code, pages taking too long to load, UI too complex, too much customization needed – anything that marks the transition away from innocence into maturity. The problems you need to handle now are not cool, they are adult stuff.

When this limit is hit, different people react differently. Some people are forever chasing the new project high. As soon as it is gone, they start looking for another ‘new’ project in order to recreate the wonderful sense of freedom and freshness, the ‘new project smell’. Others accept the maturity process and ride it all the way. Then there is a group that actually thrives in a mature project environment – they like the structure and the boundaries it provides, engaging in a series of incremental improvements within the constraints of a mature project.

The point I am trying to argue here is that for all the allure of the early days of a project, it is the period after the metaphorical 12 amp limit that separates boys from men. The early days value your ability to dream, to conjure up new, exciting and innovative concepts, new ways of doing things, and things never done before. The next phase is all about execution, about turning that vision into reality, into something that actually delivers on the early promise. Many a startup crashed and burned on the ability to deliver, to scale when faced with the real world problems and customers. The bridge from an early prototype to a real product is the hardest one to cross. All the shortcuts and omissions you made in the rush to the prototype will come to a head and bury you if you are not careful.

For all the head swooning allure of the early days, it is the maturation of a new product that is the most exciting to me, and if you bail as soon as things become too complicated, too hard, not fun any more, you are missing out on the best part. Don Draper from Mad Men was described by a bitter girlfriend as one that ‘only likes the beginnings of things‘. Don’t be the Don Draper of software projects – see the project to the successful maturation and you will be rewarded with a payoff of a job completed, not merely started.

© Dejan Glozic, 2013