Node Summit 2015

NodeSummit panel with representatives from ???
NodeSummit panel with Foundation members.

What a difference a year makes. Astute readers of this blog may remember my sitting on the Node.js fence in the fall of 2013. I eventually jumped off the fence thanks to the wave of presentations at the last year’s NodeSummit indicating that Node.js was ready. During the ensuing year I have become a vocal supporter of Node.js-based micro-services for fun and profit. It is therefore with great anticipation that I flew to San Francisco earlier this week to attend this year’s NodeSummit.

One of the immediate delights of my visit was the fact that unlike last year, IBM’s presence was huge (‘platinum sponsor’ big enough for you?). There have been projects in IBM dabbling in Node.js, and IBM PaaS Bluemix has a first class support for running Node apps. More recently though, Node.js has become a go-to technology for every new IBM project I have seen recently, particularly cloud-hosted (and most new projects fall into that category these days).

The theme of the last year was the claim that ‘Node.js is ready for the enterprise’ (once the Walmart memory leak has been fixed). The mood was one of hope and excitement, with a tinge of whistling in the dark. During the ensuing year, the pilot projects have wrapped up, risk-awerse enterprises have convinced themselves that Node.js is just fine for production, and there is no need to convince anybody any more. This year, we moved on to the details and the logistics of doing it at a massive scale.

Of course, the highlight of the conference was the announcement of the formation of the Node.js Foundation, taking over stewartship of Node from its sole corporate sponsor Joyent (Joyent continues to be the member of the Foundation). It is left to be seen if this move will heal the rift in the community caused by the fork of Node into io.js, but it does point at the maturation of the platform and addresses some of the key complaints regarding the control over its evolution.

Another long-awaited news was the shipping of Node.js 0.12. Exactly a year ago when I attended Node Day organized by PayPal, then and still current Node.js team lead TJ announced that the shipping of 0.12 was ‘immanent’. Well, this was the longest ‘immanent’ anything in the recent history, if that word would to retain any meaning. On the positive side, the shipping of 0.12 followed the passage of all the test case suites for all the supported platforms, turning this into the new quality benchmark to satisfy for all future releases (and the coveted 1.0). It also underlines the difficulty that Node.js platform now finds itself in, being built into the core of more and more production deployments, and under a lot of pressure to evolve the platform while maintaining quality.

One of my favourite talks was by Fred Schott from Box about the realities of running Node.js in a real-world situation. It was heart-warming to follow performance tweaks all the way from mediocre to fast. It is a cautionary tale that, while Node.js is a wonderful platform with a potential for great performance, careless out of the box apps may disappoint. Like any other platform, it requires tuning, profiling and (what a concept) a certain level of mastery. You cannot just copy a ‘Hello, World’ app from Stack Overflow and expect server-melting speeds.

Another personal highlight was that the old ‘server vs client’ rendering debate is alive and well. It was amusing for me to hear two young PayPal team members talk about ditching Node/Dust combo (a normal PayPal staple) for client side rendering using Angular. In my chat with them they revealed that they managed to wrestle decent performance from Angular on mobile only after a lot of work (see my comment about Box’s experience above). Funnily enough, in a later talk Peter Marton shared tricks on isomorphic Node.js apps, where the same template is reused on both sides of the divide (my preferred technique as well). It just shows that there is no right or wrong approach – whatever works in your particular situation.

If I had one complaint about the two days I spent at the conference, it was that it was somewhat light on hard-core technical content. The focus was on high-level panels, ‘this is how we migrated from the monolith to the Node micro-services’ talks and ‘look at all the boxes in our architecture’. Now that we have all congratulated ourselves for the foresight to support Node.js and turn it into such a world-class platform to write modern systems, we should probably take off our celebratory hats and dig into the details of hard core, production Node.

All in all, I enjoyed the summit but left with mixed emotions. Last year I was fully aware of doing familiar things in a new platform, and the most mundane tasks (‘look ma, I am setting a cookie using Node.js!’) felt new and exciting. I think we are leaving that ‘new car smell’ age and entering the period where Node.js will just dissolve into the background – become like air or water, and we will focus on ‘what’ we are doing, rather than the fact we are doing it in Node.js. It may become like (gasp) Java at some point in the future.

For some developers (see my post on another TJ) there is no fun in that (look, ma, I am setting a cookie using Go!), and restless Node.js committers eager to move fast and break things are already busy in the io.js repository. I on the other hand just want to build awesome apps. Node.js is already a great platform for me and as long as it remains stable, reasonably bug free and supported, I am happy. I am sure I share this with many enterprise Node.js enthusiasts.

© Dejan Glozic, 2015

Full Stack Toronto Conference 2014


We at IBM are not strangers to large, well capitalized conferences. As things go in the conference-industrial complex, it is a big deal when one of your keynote speakers is Kevin Spacey, or Imagine Dragons entertain you after hours. So to say that the first Full Stack Toronto Conference was on the opposite side of the spectrum would be an understatement.

How about ‘no food’, ‘no drinks except coffee in the morning’, and ‘no entertainment’ of any kind except finding parking around Ryerson University building? OK, not fair, there was a get together in the nearby Irish pub the first night that I didn’t go to because I was tired.

This conference was really just a meetup making the next step. And on a weekend. Starting on 8:45am. Who does that? Our keynote speaker Anila Arthanari, Director Of Software Development at Infusionsoft, wore a t-shirt ‘I am not a morning person’ expressing the mood of most of the audience. By all accounts, this was supposed to be a flop.

And yet. When you peel the layers of conference pageantry, what remains is the kernel of it all – good talks. When talks are good, people will not mind walking out to a nearby panini store to buy lunch, or walk a block up the Church Street to get a Starbucks hit. Nothing matters if talks are good.

And they were. We had multiple tracks, and I could not go to all the talks, but those I attended were very informative, thought-provoking and immanently applicable. After every talk I had tons of things I wrote down to try after, or catch up on. So what were my key takeaways from the conference?

  1. Lots of people use Angular.js. If you need a client side MVC, you can do worse, with a caveat that version 2.0 is on the slowly approaching horizon, and to say that migration will be interesting would be an understatement.
  2. More and more people use Browserify over RequireJS. Put off by weird configuration syntax, and feeling the need easier code reuse in the case of Node.js, people seem to prefer to just ‘require’ their modules. It makes it easier to go back and forth. I am definitely going to try using it soon. This blog might help as well.
  3. Micro-services are everywhere. In the tongue-in-cheek ‘Show Us Your Stack’ track, multiple presenters described their journey from monoliths to micro-service systems. I like how people are now past the hype and deep in the gory details on standing up such systems in practice. Many were openly asking the audience for their feedback and red flags if they see any.
  4. Not everybody uses Angular.js. I know this is a contradiction, but people who value control and being able to grow into a client side MVC still value Backbone.js and its modular approach. If anything, Angular.js 2.0 promises to be more modular and less arrogant, for the lack of a better word. Here is hoping that in the future, considering Angular.JS will not be such an ‘all or nothing’ dilemma.
  5. Isomorphic and ‘federation of single-page apps’ is a thing. I thought I will be the only one pushing for rendering stuff on both the server and the client using the same templates, but Matthew Conlen from New York Data Company talked about exactly such an approach. Personally, I find it funny that people are happy to partition the API space into micro-services, but don’t feel the need to do the same with the Web apps. As the system grows, a single one page app providing all the UI is going to be the bottleneck of the system. Which is ironic, because user interfaces are the most transient and need to be evolved at a rapid pace. In essence, we are creating a system where API services can move at a rapid speed, but the UI is one big ball of MVC mud.
  6. React can help with isomorphic. Once you decide rendering on both sides of the fence is important to you, React is an attractive proposition because it can do exactly that, and plays nice with Node.js.
  7. Internet Of Things is still in its infancy. It seems like we all feel this weird excitement over turning the lights with Node.js apps and sending messages to robots and receiving MQTT messages that it is now 24C in the room. It is apparent that all this stuff will matter one day and great things will come, but I don’t see what to do with it today other than marvel in the possibilities. I guess once somebody does something really awesome with IoT, we will all slap our collective foreheads and say ‘but of course, so elegant’.

By the way, yours truly presented as well. You can see my slides up on Slideshare, and the source code of my demo on GitHub. You may find it interesting – I got Angular.js to fit into the micro-service driven Web UI, use normal URLs (no horrible hashes or hash bangs), and share a common header with other pages. I also demonstrated SSO using Facebook as the identity provider, lively UI using Web Sockets and isomorphic approach using Dust.js rendered on both client and server. The best part was when an audience member posted a todo into the demo running live on Bluemix, and his entry popped up on screen as I was demoing it. Audience participation, live demo, unexpected proof that the code actually works as designed – priceless!

So there you have it – you can make the attendees feed themselves, only give them coffee in the morning (what is life even), and dispense with most of the usual conference perks, and they will still come if the talks are good. I would say that the first FullStack TO conference focused on the most important thing, and succeeded. Good talks first, creature comforts to follow – good priorities in my book. Looking forward to the next year!

© Dejan Glozic, 2014 2014: Trip Report (Part 3)


This is the third and final installment of my report. There was also part 1 and part 2 you should probably read first for continuity.

Day three of started with a ‘Mad Science Act’, with all the presenters trading the pre-requisite lab coat (to add ‘science’ and also a bit of ‘mad’ to their presentations).

First off, James Halliday (also known as substack). Demonstrating ultra hard core cred with a Linux laptop surrounded by a sea of Macs, substack repeated the cherished Node.js ethos of creating many small modules that can be combined and re-combined ad nauseum, forming your growing tool box. This approach does not need to end with modules – you can apply it to data, in form of data streams. Substack demoed running code where multiple streams can be combined by piping data at different routes of the server. You can read more and play with the demo at the dataplex GitHub repo.

nodeconfeu-23Feross Aboukhadijeh took us on a wild ride of using WebRTC to replicate the P2P BitTorrent communication between browsers, using the WebRTC data channels. The presentation was a lot of fun, with the audience participating by sharing obligatory pictures of cats from their browsers in real time. This feature is made available in the open source library called WebTorrent, together with a BitTorrent bridge (direct connection to BitTorrent is currently not possible due to the need for TCP connection (there needs to be a ‘hybrid’ client that Feross is currently working on).


Travell Perkins, Fidelity Investments CTO had a talk that I could not immediately place in this block and that got me confused until I realized that it was originally supposed to be on Monday, as part of the ‘Node.js in the enterprise’ section (lab coat notwithstanding). I *think* his talk was about databases, possibly his SQLdown NPM module, but sadly I cannot locate my notes on the talk (too much coffee?).

nodeconfeu-25Dominic Tarr (dubbed ‘our resident Jesus’ which is not hard to understand – just look at the picture!) was another of the hard core speakers using a Linux laptop. The notes on his talk were in the same MIA file as for Travell, so only a picture here.

nodeconfeu-26 The last talk in this long-ish section was Mikola Lysenko on the challenges of developing multi-player online games using Web technologies alone. A key problem to solve is replication because multi-player games are distributed but they share state, and this state needs to be replicated in a way that does not ruin the real-time nature of the game. He demonstrated several options in modeling real time physical systems and their pros and cons (which, in case of ‘eventual consistency’ and error correction can yield some comical side effects as seen in the demo). You can read more on his work in his multi-part blog post.

nodeconfeu-27After a needed sugar and caffeine coma break, we switched to the hardware track. First speaker was Colin Vernon, a designer on a strange mission – to bring NPM-style componentization to the hardware world. I was a hobbyist in my youth, and no stranger to the soldering gun. I had a long pause in dealing with hardware other than in a very limited way of ‘adding cards to PC slots’, so this resurgence of interest in hardware definitely brings back memories. Colin introduced us to the amazing world of littlebits, where Internet of Things can be manually put together using small modules that can be combined with the cloud and mobile remote control apps in many interesting ways. This presentation reminded me of how Twitter looked like a bizarre and pointless toy until it turned into something much bugger and more powerful – this could be another example.

nodeconfeu-28Raquel Vélez brought us Node.js-driven Bat-bot that she configured with a pen, with some freedom as to where to go on the canvas and when to draw. The philosophical question was – at which point we can call the result ‘art’. The practical demonstration almost succeeded but the artist committed robocide by falling off the drawing table before finishing the painting (which made the price of the painting skyrocket as a result). These modest starts like this may seem of little value, but Raquel reminded us that Mars Rover is essentially just a much bigger and more complex bot.

nodeconfeu-29Afternoon workshops reduced our collective age to pre-school, with all the toys laying around – drones, Korg bits (you can assemble an analog synth by connecting the bits together) and other goodies to wake up a kid in you.

nodeconfeu-30All in all, it was a great conference, in a format that made TJ Fontaine call it ‘Node.js vacation’ but it was of course much more than that. A lot of time for informal discussions (and environment that encouraged it because, you know, we are on an island with nothing else to do :). Being able to talk to people that wrote Node.js modules we use daily was absolutely a treat, as well as meeting some internet contacts in person for the first time. I came back home with a bag full of memories and lots of ideas for new projects.

I will close with a view of the Waterford Castle ferry that we got to know well during our conference. See you all again next year!


© Dejan Glozic, 2014 2014: Trip Report (Part 2)


Read part 1 of the report.

Before I continue to part 2, a word about timing. Some of you may ask why it took me so long to publish my impressions of an event that happened last week (which, in Internet time, is ‘way in the past’). The thing is, I prefer to live the reality, rather than be outside it and look into it through the phone or tablet screen. As Louis CK would say, ‘the resolution on reality is amazing, it is super HD’, which is way better than even the new iPhone 6. Hence I make records of events as quickly as I can and get back to, you know, be in them.

On with the event. Day 2 of started with the front-end track, with Alex Liu from Netflix dazzling us with the radial approach they took with Node.js and Dust.js when it comes to A/B testing various designs. What is unique in this approach is that A/B testing is not done at a load ballancer level. A traditional approach would see two apps serving the same routes and the ballancer (such as Nginx) routing requests to instance A or B using some routing rule, and then measuring the outcome. Netflix took Dust.js partials to a completely new level by testing out many different combinations of designs within the same app or page, and testing more than A and B (there is C and D and E etc.). Of course, common sense suggest this is very hard to manage manually, hence their approach to use a registry that applies packaging rules and puts together distinct combinations of parts that form a particular test (they are served from CDN for the next person participating in it to cut on the processing latency). That, plus I got a kick of Netflix being among the users of Dust.js that we are also using.

Alex Liu from Netflix on Node.js/Dust.js A/B/C/D/E/F testing.
Alex Liu from Netflix on Node.js/Dust.js A/B/C/D/E/F testing.

You may know Matteo Collina already from my blog posts because he wrote the MQTT NPM module we are using in our own code. This time around, Matteo talked about the new work that is attempting at make communication between micro-services possible without the message brokers. This work is inspired by libchan by Docker team that is written in Go. Matteo and my internet buddy Adrian Rossouw are collaborating on providing a Node.js version as part of the project Graft.

Matteo Collina talks about project Graft.

Jake Verbaten shared the nitty-gritty of writing Node.js services in Uber, where everything you do needs to be up all the time, and run on a multitude of machines. He highlighted all the extra steps that separate anything you make from the day it is ‘productized’. Jake showed the tools Uber uses for this process, including the tool they call ‘potter’ (will be open-sourced at some point). It gets all the scaffolding going for a project, including the repo, continuous integration server, and monitoring.

Jake Verbaten on production-grade Node.js services at Uber.

Thorsen Lorentz went into the details of Chrome JavaScript engine (V8) and JavaScript performance. This is not something you normally thing right away when writing JavaScript, but can come in handy when doing performance tuning. However, his examples left me with mixed feelings. We like our abstractions and the fact that declaring variables, and later assigning values (as opposed to initializing them in the first statement) can have detrimental effect on performance filled me with dread. I don’t want to break the black box approach to Node.js if possible.  Don’t get me wrong, it is perfectly understandable that V8 will work better if helped by certain coding pattern, but it means that I need to become aware of the actual JavaScript engine running my code, which further breaks the Node.js promise of front end and back end unity (unless you restrict yourself to only serving your Chrome clients, that is). Of course, all this is not Thorsen’s fault – we should not shoot the messenger.

Thorsen Lorentz on disturbing secrets about how V8 goes about running your code.

After the break, Bert Belder from StrongLoop kicked off a Node Core track. StrongLoop is a company build around helping others succeed with Node.js, and has the most corporate contributions to the upcoming 0.12 version, as well as the current 0.10 (outside of Joyent, of course). Bert has invited the audience to ask for fixes in node, express, node inspector and other areas.

Bert Belder from StrongLoop on the upcoming 0.12, Node.js community, and what StrongLoop can do to help.

Fedor Indutny held up to the stereotype that Russians are scary good at math by taking us down to follow the white rabbit of TLS encryption. His credentials (see what I did there): he is the author of the TLS module for Node.js. Of course, most of us just want encryption to work and don’t care how, but it was fun to peek behind the curtain at RFC 5246 and find out more about it. He spent the rest of the talk walking us through code snippets of setting up the server with tls.js and exchanging encrypted hello’s.

Fedor Indutny on the wonderful world of TLS encryption.

For the second time at this conference, TJ Fontaine went into the dark allays of tracing Node and finding out what is happening when you launch ‘node app.js’. Apart from replicating Node graphics on T-shirts and the backs of his laptops, TJ’s favorite pastime is debugging v8 core dumps on OSX and Linux using lldb-v8. I got to see more V8 detritus than I saw in a – ever? Golden takeaway – please, please, please name your functions – debugging anonymous functions postmortem is no fun.


After lunch, there were more workshops – more debugging with TJ and hands on with NearForm’s own nscale deployment solution. Unfortunately, my jet lag finally caught up with me and I had to crash.

Our afternoon event involved vising a local hurling club (an Irish sport involving an ash bat and a ball, although hitting a fellow player instead is fine too). We got to make a nice group photo on the green.

Our evening entertainment took us to Waterford on the sea shore. Lots of Guinness (I swear it tastes better here than from a can back home!), and some traditional Irish music, including wonderfully quirky Irish pipes.


Continue on to the third and final installment of the report.

© Dejan Glozic, 2014 2014: Trip Report (Part 1)


Shady’s back, tell a friend! Fresh from the green grass of Ireland where I attended (and presented) at this year’s, I am now back to report on it as promised.

This year’s conference is a second instance of a format started last year by Cian Ó Maidín and the friends from Near Form. The goal is to carefully curate talks across the Node.js community to ensure quality over quantity. I had a great pleasure attending the conference, particularly as I was one of the ‘carefully curated’ speakers this year. What adds a particular flare and sparkle to the event is the location – an actual Irish castle in Waterford, a south-eastern town of Ireland and the oldest of them all.

Getting to the Waterford castle involves an excerpt from the movie Plains, Trains and Automobiles – an itinerary that involves flying to Dublin, catching a bus to Waterford, then switching to a taxi that at some point needs to go over the river Suir via a private car ferry to get to the castle on the island.

The event started in the evening with a welcome reception that involved circus acts – a very ‘tall’ lady (see above) and a guy literately playing with fire.

nodeconfeu-2After-dinner entertainment included a lovely young lady playing a cello and singing in a way that fuses transitional Irish music and modern sensibilities – perfect for Irish hipsters (they do exist, don’t they?). Unfortunately her name escapes me – if you know it (and have a link to her home page), please drop me a comment (Edit: @fritzvd was kind enough to point out her name is Alana Henderson – thanks!).

nodeconfeu-3 was held in a club house of the nearby Golf Club (part of the same Waterford Castle resort). For the next three days, our master of ceremonies was Mikeal, who was well known to most attendees (just look at your apps and if you require ‘request‘, you know him too).

Mikeal tells people to sit down so that we can start.

Conference opened with a welcome addressed by Cian, outlining what is awaiting us for the next three days, and upholding the conference Code of Conduct, which was wonderfully short:

  1. No harassment of any kind is allowed
  2. Please don’t fall into the Suir river (apparently somebody did not long ago)

The technical part of the conference started with Node’s own TJ Fontaine, Node.js core lead, with his ‘State of Node’ address. TJ posed that the industry is in a state that makes this a perfect timing for a JavaScript framework. He also re-iterated some of the key tenets of Node.js philosophy, including non-blocking I/O and ‘do one thing, and do it well’ ethos. Finally, he stressed that the evolution of Node.js is not about what other languages or frameworks are doing, but what is good for Node.js and JavaScript themselves.

TJ Fontaine delivers the State of Node address.

NearForm’s own Richard Rodger kicked off the Micro-Services block with the accumulated experience of deploying micro-services (and in particular Node.js micro-services) in production. He highlighted some natural advantages (scalability, flexibility of deployment) but also disadvantages (added latency). From his real-world experience, he concluded that business logic should be in the services (no core monolith), that developers should resist Tower of Babel (the temptation to use many languages and stacks) and to assume you can design upfront (services are ‘discovered’, not designed). Nevertheless, he reiterated one of the strong suits of micro-services – the fact that you can change your mind, swap databases mid-project (or anything else).

NearForm’s Richard Rodger kicks off the Micro-Services block.

Clifton Cunningham focused on a very hard problem of micro-services – the fact that while multiple services are responsible for various parts of the system, pages still need to share some content. He enumerated options used in the past – client-side stitching using Ajax, front-end server, Server Side Includes (SSIs), Edge-side includes (esi) etc. He presented his team’s take on the problem – an open-source module called Compoxure that has a number of advanced features to deal with the problems normally faced in production – performance, latency, failures and authentication. He also addressed the problem of delivering CSS and JS for this shared content across the micro-services.

Clifton Cunnigham on Compoxure.

Then it was time for me. My take was how my team in IBM DevOps Services decided to pursue micro-services in general, and Node.js in particular as we started evolving our system. I don’t need to go into details – this whole blog is a weekly chronicle of our journey. I added a twist from a position of doing Node.js in a large enterprise – the need to legally vet a large number of Node.js modules (causing legal to hate us) and the complexity of deploying a large number of services in a secure manner (causing OPS to hate us).

Dejan Glozic on Node.js micro-services at IBM (photo courtesy of Louis Faustino via SmugMug).

The last speaker in the Micro-Services block was Fred George, bringing his wealth of experiences building micro-services in a number of real-world projects. He brought forward several examples of projects using different technologies and architecture, unified in the fact that most of the time an event based (asynchronous) approach was a better fit than synchronous. Out of that experience he extracted a concept of a system where all services receive all the messages all the time, but the messages are semantically classifiable, forming ‘rapids’ (all events), ‘rivers’ (themed events) and ‘ponds’ (state/history).

Fred George on rapids, rivers and ponds.

After a coffee and JS cupcakes break, we switched to the ‘Production’ track, starting with Brian McCallister from Groupon walking us from the familiar experience of a company whose monolith has become so large and unwieldy that it eventually made adding new features virtually impossible. Groupon slowly and carefully migrated their two large installations (North America and Europe) to a micro-service system using Node.js, resulting in unblocking of further evolution, with performance and scalability improvements tossed into the mix. This was in a sense a partial continuation of the micro-service track, considering that Groupon’s new system shares many traits with the ones we talked about in the preceding talks.

Brian McCallister on building the I-Tier system at Groupon.

PayPal’s own Jeff Harrell zeroed on a number of anti-patterns of implementing Node.js in a real-world project in PayPal. PayPal made a wholesale transformation to Node.js that is still ongoing, and their large team contributed a number of these anti-patterns. Among them were: bringing baggage from the previous projects, re-creating monolithic applications using Node.js, Googling ‘how to do X in JavaScript’, wrapping everything in promises, sloppy async code, using Node.js for everything, and ignoring the ecosystem.

PayPal's Jeff Harrell on Node.js real world anti-patterns.
PayPal’s Jeff Harrell on Node.js real world anti-patterns.

The last speaker for the day was Aman Kohli from Citi Bank, bringing to us the experience of deploying Node.js to provide back-end services for mobile apps in an environment that is everything but forgiving when it comes to security, adherence to regulations and process. According to Aman, they chose Node because of the async event model, being ideally suited for mobile and sensor apps, the fact that it was approved for internal usage, that it required fewer controls, and due to the good success they had with using Hapi framework for building mobile API services.

Aman Kohli on using Node.js for mobile services in Citi Bank.
Aman Kohli on using Node.js for mobile services in Citi Bank.

At this point it was time to break for lunch. From several choices I picked to attend Kraken.js workshop for my afternoon activity, where I could pick the brains of Jeff Harrell and Erik Toth from PayPal on the philosophy and the plans for this open source suite we already use in our micro-services.

Evening R&R was provided to us with a combination of Irish whiskey tasting (not bad at all, but still prefer Scottish single malt) and a great local folk band treating us with a mix of traditional and Irish-treated covers.

Continue to the part 2 of the report.

© Dejan Glozic, 2014

Odds and Ends

Jack Spade Odds and Ends pouch
Jack Spade Odds and Ends pouch

This week and the next there will be no regular blog posts. I hate to break my routine, but I am going to IBM Innovate 2014, and being an IBMer myself, it is a working conference for me, requiring a lot of preparation time and paying my booth dues.

If you are coming to Innovate, don’t forget to come and hear myself and Dan Berg in our presentation on the new DevOps Pipeline we built for IBM DevOps Services (powered by JazzHub). Here are the coordinates:

ICD-1810 : DevOps Services: Automated Delivery Pipeline for Codename: BlueMix
Innovation – Cloud Development
Date/Time : Wed, 04-Jun, 09:15 AM-10:15 AM
Room : Dolphin-Australia 3
Co-presenter(s):Daniel Berg, IBM

While I am here, let’s review other events where you can see me. The next one comes in July 9-10, when I will present at DevCon5 in New York City. This is my talk:

Node.js Micro-Services: The Water is Fine, Jump In!

Well, duh – what did you expect, just check the number of micro-service posts I made – it is only fitting for me to spread the message in person as well.

Then, in September 7-11 in Ireland comes I don’t have the title of the talk yet but you can expect more micro-service-y goodness.

So there you go – three opportunities to meet me and have a beer – what’s not to like?

See you again in about 10 days when I will resume the regular programming.

© Dejan Glozic, 2014

NodeDay 2014

'Unicorns farting rainbows' from Eran Hammer's presentation 'Node Inc.', artwork by @ChrisMCarrasco
‘Unicorns farting rainbows’ from Eran Hammer’s presentation ‘Node Inc.’, artwork by @ChrisMCarrasco

This post is the continuation of my trip report that started with part 1 on Node.js on the Road. NodeDay was organized and hosted by PayPal in their building in San Jose, CA, on Feb 28th 2014. The focus of the one-day conference was Node.js in the enterprise, which was a great fit for what we are now doing with Node.js in IBM Rational. Add in the opportunity to talk with Bill Scott, Richard Ragan and Jeff Harrell about their use of Dust.js in production, and it all added up to a promising day.


I drove down 101 South from San Francisco to my hotel in San Jose in a calm evening. Last thing I remember before I fell asleep was muted howling of unusually strong wind outside my hotel window. Then I woke up into a morning of storm reports, flash floods, mud slides and falling rocks throughout the region. I must have been very tired.

Luckily, this was largely over by the time I finished breakfast, and PayPal was only 2 miles away.

We got badges and bracelets at the entrance. Badges doubled as agenda – simple and minimalistic, as you would expect from a Node.js event.


The event was kicked off by Bill Scott, Senior Director of UI Engineering, PayPal and our host for the day. The first keynote was by Eran Hammer, Senior Architect of Walmart Labs, who was singlehandedly responsible for pushing me off the fence and into Node.js development. The watershed event was Black Friday 2013, when 55% of Walmart traffic was from mobile devices, and handled by Node.js servers. The uneventful day for the pager-carrying developers cemented their trust that Node.js has ‘arrived’, resulting in Walmart doubling down on Node and spreading it to the rest of the enterprise.

Eran’s focus was on saving the typical enterprise Java developer from living a double life – passionately using Node at home, and wrestling with the legacy projects behind the firewall at work (illustrated with a contrast between unicorns and rainbows at the top of the post, and the Intranet wall of dead puppies – see the rest of the presentation). A common myth is that Open Source is all done by little guys, individuals and startups – just technology and no politics. In reality, Open Source is sustained and flourishing in the enterprise – nobody is using the Open Source more than the enterprise (and Walmart is not exactly a startup either). Why? Because people see the leverage that working for large companies can give them. Hence, we need more ‘Inc.’ in our Node, the community to be one, instead of Enterprise people being on the side.

We (the enterprise) should not feel like we are on the fringes. For example, Joyent has been the best supporter of the project, and one of the biggest consumers of Node. The business of paying for support for Open Source projects is still alive and a valid one. With NPM in the news recently, it is clear that profitability is not a dirty word, it is the only way to sustain the community. Issues such as private NPM repositories that are so important for enterprises were initially hard to take seriously, but it is obvious it is understood now, judging by half a dozen solutions already available. It shows that in many cases, building on what is already available and perhaps sponsoring/adopting core contributors is a cost-effective way to get things done and ensure critical support. Doing everything in-house is an anti-pattern.

Eran Hammer, Senior Architect - Walmart Labs
Eran Hammer, Senior Architect – Walmart Labs

TJ Fountaine, Node.js Project Lead from Joyent has mostly repeated his talk from ‘Node.js on the Road’, which was a good thing to do since there was no overlap in the crowd between the two events (not counting myself of course :-). You can read my report in the previous post with no loss of data.

TJ Fontaine, Node.js Core Lead, Joyent and Bill Scott, Senior Director, UI Engineering - PayPal
TJ Fontaine, Node.js Core Lead – Joyent and Bill Scott, Senior Director, UI Engineering – PayPal

The funniest part of TJ’s keynote is the extent he went to in order to ensure that he is not mixed up with the express.js author TJ Holowaychuk:

Adam Baldwin, Chief Security Officer – &yet, has covered a touchy subject – Node.js security. With all the enthusiasm for the new way of working Node provides, enterprise is very touchy around security and it is a big topic when considering Node adoption. Adam suggests protecting what makes your money, and consider availability as an aspect of security (if the app is down, who cares if it is secure). The process of how you handle the vulnerability is the key (but you will screw it up anyway).

When it comes to security, nodejs-sec mailing list is important because all security-related announcements go there (very few so far, knock on wood). In reality, most of the issues will be in Node modules, not the core – security is built into core, it is not a bolt-on module. The enterprise is responsible for what you require() – there has to be a process in place for vetting modules, and private NPM repositories. Linting is recommended (using npm install precommit-hook).

Security-related test cases are important (some orgs only write test cases for the happy path). One useful library that can help – retire.js (scans your web app or Node app for use of vulnerable JS libraries).

With all said and done, the greatest vulnerability in the enterprise?

Adam Baldwin, Chief Security Officer – &yet
Adam Baldwin, Chief Security Officer – &yet

Joe McCann, COO of The Node Firm has addressed the business case for Node. There are many signs of Node.js maturity – exponential growth in NPM, number of books published, enterprises going into production with Node. He posited that a mission statement for Node should read: “An easy way to build fast and scalable applications”. As to ‘Why Now’, he suggested that legacy stacks are impending innovation and make companies less competitive.

Joe suggested five business tenets when making the case for Node: innovation, productivity, developer joy, hiring/retaining talent and cost savings. He focused on a few success stories:

PayPal had velocity as the key driving point. Node provided boost to the workflow, allowing faster iteration and innovation. After a long history of Java/C++, developers are energized by their jobs again. A frequently re-twitted quote:

Groupon moved from a monolithic application that became a burden and a blocker to a service-oriented architecture. Node.js now handles the same amount of traffic using less hardware (direct cost savings – a music to every CFO’s ears). At the same time, page load times decreased by 50%. The use of a common layout service allowed them to retain the ability to make site-wide design changes quickly.

Walmart achieved Black Friday CPU utilization hovering around 1% (you want to be bored when on pager duty). In addition, quality of candidates after switching to Node.js increased – Node is a magnet for talent. This is important – Node could be a differentiator in hiring/retaining.

In Yahoo!, Node.js services are handling 2M requests/minute, 200 developers write Node, there are 500 internal and 800 external Node modules. Node brought speed and ease of development.

Back to business tenets, innovation is helped by building things faster and more efficiently. Productivity is helped by terse syntax and absence of thick layers of bureaucracy that slow developers down. “Productive developer is a happy developer”, which helps with developer joy and helps stability and retention. As for hiring/retaining talent, there is a magnetism of working on Node that is unique. Finally, cost savings are real and palpable: fewer developers writing apps in less amount of time, using less hardware. Using Node in multiple projects multiplies the savings.

This topics generated lively discussion, and a member of the audience claimed that similar enthusiasm was heard 10 years ago around Ruby on Rails. So what is different with Node? The consensus seems to be that since JavaScript is gravity on the client, using JavaScript ‘all the way down’ provides benefits of simplification and ‘single stack’ that cannot be duplicated in other stacks.

The reality of enterprise architecture is that mixed stacks (Node.js + Java/PHP) are everywhere. Right now, front end in Node.js is a safe bet, but there is growth in systems around Node.

At the request to cover some failures (to counter success stories for fairness), Sam from offered a counter-argument of why some shops cannot get there just yet – some libraries just do not exist yet (but time can help plug this gap). In the same vein, LinkedIn is still a huge Scala/Java shop, and in some use cases Play framework is easier to move their developers forward than to make a massive switch to Node (LinkedIn still uses Node for their mobile services, and they are very fast).

The takeaway is that while Node.js is obviously at an inflection point and it is not hard to find success stories, but it will never be a magic bullet that will solve all the problems. It is imperative to ensure that the problem to be solved is a good fit for Node.

Joe McCann, COO - The Node Firm
Joe McCann, COO – The Node Firm

After lunch and an opportunity to socialize with actual people in the 3D world and full HD (what a concept!), Erik Toth, Principal Software Engineer at PayPal took us through the fuzzy and complicated world of moving a large developer workforce to NodeJS. He provided a set of hard won tips on how to handle resistance and problems along the road. First off, accepting that your rationale might be non-technical is healthy – Node is as much a way of life as a technology. There will be critics, they will parrot questions, represent opinions they’ve read as their own, use outdated information, micro benchamrks, assume you have ulterior motives, they will underestimate you (real engineers use something else). They will assume that because they haven’t heard of something, it doesn’t exist, and all of this without even looking at the technology.

The secret to overcome this? Influencers. You need to have an advocate that absolutely believes in it, and things will start happening. Setting limits is key – focus and avoid some things, avoid inventing problems to have something to solve, solving other peoples’ problems, or solving solved problems. Only fix what is broken. For PayPal, it was primarily velocity and productivity, so there was a need to scale people, product and technology. This requires an ecosystem: GitHub, npm, tooling.

Erik pointed at silos as a serious impediment (enterprises like Game of Thrones). They breed distrust, create shallow alliances based on selfish goals, black-boxes teams and code. He suggested elimination of backlog as a concept (where features go to die).

A key takeaway for individual developers?

Individuals rarely solve real problems – benefit from the work of others. This is not revolutionary, just harder in the enterprise. Finally, the quality of code should not depend on the access rights (“The only thing that should differentiate public and private modules is access rights”). Be proud of your code, expect copypasta, learn semver and teach others.

Oh, and the key takeaway from the Q/A: “We treated Node as something that will go away in 5 years, and we didn’t want to impose custom solutions to those that will come after us”.

Erik Toth, Principal Software Engineer - PayPal
Erik Toth, Principal Software Engineer – PayPal

Ian Livingstone, VP of Engineering of GoInstant told a war story of using Node in two apps, and how the second time was informed by all the mistakes made in the first app. The first app was written in the time when Node was instable. More importantly, growing code base is a maintenance nightmare, which greatly underscores the important of writing small modules instead of large monolithic apps (a common theme throughout the NodeDay).

Ian offered many tips collected the hard way. For example, state must live in a class instance, not in modules. Furthermore, when a class has a state, it must have both ‘initialize’ and ‘destruct’ methods. Another trick that helped was the use of JSON schema, which makes it easy to exhaustively test code. This includes schema for Request and Response (run only in development, limited in production for performance reasons).

On the topic of UI testing, Ian pointed out that multi-user integration tests are required to excersize user interfaces. Selenium was an obvious first choice, but the tests ended up verbose and very fragile, and took a long time to run. NPM to the rescue: a tool called ‘Barista’ (extension to Mocha) proved to be a better solution. Lesson: the power at your finger tips through the Node eco-system is incredible.

The need to avoid one giant code base cannot be stressed enough: like in Java’s “God object”, leaked responsibilities between modules made the code base into a giant ‘house of cards’ that was fragile and very hard to test, greatly diminishing trust in the code. Solution: small, simple and independent modules.

The second time around, applying all these hard-won lessons had a hugely positive effect – it changed the conversation. A small downside – longer install times via NPM (solved by using a proxy backed by S3). Node works better when it has one thing to do, favoring an architecture with many small and simple services. As a convenience, each service has a client library that wraps the transport concerns, making it easy to consume and test. Using upstart and supervisord provided for process lifecycle management.

Takaway – Node.js is power, and many problems have already been solved by the community.

Ian Livingstone, VP of Engineering - GoInstant
Ian Livingstone, VP of Engineering – GoInstant

Trevor Norris, NodeJS Core Maintainer from Mozilla took us on a journey much closer to the metal, under the layers of abstractions we are so used to in our daily work. The aim of the presentation was not to prove that all abstractions are bad, just to make their cost clear. He offered a number of performance tips related to intimate knowledge of V8 engine that powers Node (example: inlined functions can’t contain more than 600 characters, including comments; inlined functions must be declared close to the call site etc.). This level of detail is not very practical to everyday Node developers writing modules at higher level of the stack, but are a reality for Node Core developers that must worry about the performance envelope of the bedrock on which everything else resides.

In that spirit, Trevor continued to provide advise on the most obvious ‘sins of our abstractions’ – the fact that some events like DNS and FS are in fact pseudo-async (they tie up a thread in the background), and again sternly warned about using the new execSync function that is a very odd animal in the ansync world of Node.

Key takeways: write code that you can maintain, write code that doesn’t crash – the slowest code is the code that doesn’t run. And finally, code for your case. Very few developers actually code with Node Core performance requirements in mind.

Trevor Norris, NodeJS Core Maintainer - Mozilla
Trevor Norris, NodeJS Core Maintainer – Mozilla

Richard Rodger, Founder of Near Form really drove the point of keeping Node processes small (a motif of the entire conference). This ensures that services are easy to restart, which makes for a fault-tolerant system. With a number of services running in the system, a communication mechanism is require (messages). Processes need to be small (100 lines of code), a few screens to scroll, that can fit inside your head. Frameworks are discouraged – use modules and composition.

A good message broker that supports pub/sub is very handy in this architecture – it makes the system loosely coupled and extendable. Micro-services also make rewrites cheap, avoiding project management ceremony. They localize bad coding choices, making them less of an issue because they are confined to a single micro-service.

Not all services are created equal – some are critical and require a lot of attention, some do not. By grouping services by quality, focus can be applied where it counts. As always, fixing a bug later in the cycle carries a higher cost, but with micro-services, the curve is less steep.

Richard provided some real-world examples of approaching a problem using micro-service architecture. The approach flows from user stories, to capabilities, services and messages. After that, decisions need to be made on how to run in production: what performance, scale, behavior and message transport are needed. Of course, production rollout need not be extensive – a good start can be to run all the micro-services on one machine and cluster only the ones that actually need it.

As a final comment, Richard agreed that micro-services can be built in any language – Node.js is simply a very suitable option, but not the only one.

Richard Rodger, Founder - NearForm
Richard Rodger, Founder – NearForm
NodeDay audience - full house
NodeDay audience – full house
Mr. JavaScript himself - Douglas Crockford, unaware of my paparazzi tendencies.
Mr. JavaScript himself – Douglas Crockford, unaware of my paparazzi tendencies.

All in all, it was a well organized event and an amazing opportunity to talk to a number of key people in the Node.js community all in the same day. In effect, it was like its topic – I was able to talk to more people and learn more in a shorter period of time and virtually no downtime – like a true Node app. Kudos to Bill Scott for being a gracious host and Lenny Markus for ensuring it all went without a hitch.

© Dejan Glozic, 2014