NodeDay 2014

'Unicorns farting rainbows' from Eran Hammer's presentation 'Node Inc.', artwork by @ChrisMCarrasco
‘Unicorns farting rainbows’ from Eran Hammer’s presentation ‘Node Inc.’, artwork by @ChrisMCarrasco

This post is the continuation of my trip report that started with part 1 on Node.js on the Road. NodeDay was organized and hosted by PayPal in their building in San Jose, CA, on Feb 28th 2014. The focus of the one-day conference was Node.js in the enterprise, which was a great fit for what we are now doing with Node.js in IBM Rational. Add in the opportunity to talk with Bill Scott, Richard Ragan and Jeff Harrell about their use of Dust.js in production, and it all added up to a promising day.


I drove down 101 South from San Francisco to my hotel in San Jose in a calm evening. Last thing I remember before I fell asleep was muted howling of unusually strong wind outside my hotel window. Then I woke up into a morning of storm reports, flash floods, mud slides and falling rocks throughout the region. I must have been very tired.

Luckily, this was largely over by the time I finished breakfast, and PayPal was only 2 miles away.

We got badges and bracelets at the entrance. Badges doubled as agenda – simple and minimalistic, as you would expect from a Node.js event.


The event was kicked off by Bill Scott, Senior Director of UI Engineering, PayPal and our host for the day. The first keynote was by Eran Hammer, Senior Architect of Walmart Labs, who was singlehandedly responsible for pushing me off the fence and into Node.js development. The watershed event was Black Friday 2013, when 55% of Walmart traffic was from mobile devices, and handled by Node.js servers. The uneventful day for the pager-carrying developers cemented their trust that Node.js has ‘arrived’, resulting in Walmart doubling down on Node and spreading it to the rest of the enterprise.

Eran’s focus was on saving the typical enterprise Java developer from living a double life – passionately using Node at home, and wrestling with the legacy projects behind the firewall at work (illustrated with a contrast between unicorns and rainbows at the top of the post, and the Intranet wall of dead puppies – see the rest of the presentation). A common myth is that Open Source is all done by little guys, individuals and startups – just technology and no politics. In reality, Open Source is sustained and flourishing in the enterprise – nobody is using the Open Source more than the enterprise (and Walmart is not exactly a startup either). Why? Because people see the leverage that working for large companies can give them. Hence, we need more ‘Inc.’ in our Node, the community to be one, instead of Enterprise people being on the side.

We (the enterprise) should not feel like we are on the fringes. For example, Joyent has been the best supporter of the project, and one of the biggest consumers of Node. The business of paying for support for Open Source projects is still alive and a valid one. With NPM in the news recently, it is clear that profitability is not a dirty word, it is the only way to sustain the community. Issues such as private NPM repositories that are so important for enterprises were initially hard to take seriously, but it is obvious it is understood now, judging by half a dozen solutions already available. It shows that in many cases, building on what is already available and perhaps sponsoring/adopting core contributors is a cost-effective way to get things done and ensure critical support. Doing everything in-house is an anti-pattern.

Eran Hammer, Senior Architect - Walmart Labs
Eran Hammer, Senior Architect – Walmart Labs

TJ Fountaine, Node.js Project Lead from Joyent has mostly repeated his talk from ‘Node.js on the Road’, which was a good thing to do since there was no overlap in the crowd between the two events (not counting myself of course :-). You can read my report in the previous post with no loss of data.

TJ Fontaine, Node.js Core Lead, Joyent and Bill Scott, Senior Director, UI Engineering - PayPal
TJ Fontaine, Node.js Core Lead – Joyent and Bill Scott, Senior Director, UI Engineering – PayPal

The funniest part of TJ’s keynote is the extent he went to in order to ensure that he is not mixed up with the express.js author TJ Holowaychuk:

Adam Baldwin, Chief Security Officer – &yet, has covered a touchy subject – Node.js security. With all the enthusiasm for the new way of working Node provides, enterprise is very touchy around security and it is a big topic when considering Node adoption. Adam suggests protecting what makes your money, and consider availability as an aspect of security (if the app is down, who cares if it is secure). The process of how you handle the vulnerability is the key (but you will screw it up anyway).

When it comes to security, nodejs-sec mailing list is important because all security-related announcements go there (very few so far, knock on wood). In reality, most of the issues will be in Node modules, not the core – security is built into core, it is not a bolt-on module. The enterprise is responsible for what you require() – there has to be a process in place for vetting modules, and private NPM repositories. Linting is recommended (using npm install precommit-hook).

Security-related test cases are important (some orgs only write test cases for the happy path). One useful library that can help – retire.js (scans your web app or Node app for use of vulnerable JS libraries).

With all said and done, the greatest vulnerability in the enterprise?

Adam Baldwin, Chief Security Officer – &yet
Adam Baldwin, Chief Security Officer – &yet

Joe McCann, COO of The Node Firm has addressed the business case for Node. There are many signs of Node.js maturity – exponential growth in NPM, number of books published, enterprises going into production with Node. He posited that a mission statement for Node should read: “An easy way to build fast and scalable applications”. As to ‘Why Now’, he suggested that legacy stacks are impending innovation and make companies less competitive.

Joe suggested five business tenets when making the case for Node: innovation, productivity, developer joy, hiring/retaining talent and cost savings. He focused on a few success stories:

PayPal had velocity as the key driving point. Node provided boost to the workflow, allowing faster iteration and innovation. After a long history of Java/C++, developers are energized by their jobs again. A frequently re-twitted quote:

Groupon moved from a monolithic application that became a burden and a blocker to a service-oriented architecture. Node.js now handles the same amount of traffic using less hardware (direct cost savings – a music to every CFO’s ears). At the same time, page load times decreased by 50%. The use of a common layout service allowed them to retain the ability to make site-wide design changes quickly.

Walmart achieved Black Friday CPU utilization hovering around 1% (you want to be bored when on pager duty). In addition, quality of candidates after switching to Node.js increased – Node is a magnet for talent. This is important – Node could be a differentiator in hiring/retaining.

In Yahoo!, Node.js services are handling 2M requests/minute, 200 developers write Node, there are 500 internal and 800 external Node modules. Node brought speed and ease of development.

Back to business tenets, innovation is helped by building things faster and more efficiently. Productivity is helped by terse syntax and absence of thick layers of bureaucracy that slow developers down. “Productive developer is a happy developer”, which helps with developer joy and helps stability and retention. As for hiring/retaining talent, there is a magnetism of working on Node that is unique. Finally, cost savings are real and palpable: fewer developers writing apps in less amount of time, using less hardware. Using Node in multiple projects multiplies the savings.

This topics generated lively discussion, and a member of the audience claimed that similar enthusiasm was heard 10 years ago around Ruby on Rails. So what is different with Node? The consensus seems to be that since JavaScript is gravity on the client, using JavaScript ‘all the way down’ provides benefits of simplification and ‘single stack’ that cannot be duplicated in other stacks.

The reality of enterprise architecture is that mixed stacks (Node.js + Java/PHP) are everywhere. Right now, front end in Node.js is a safe bet, but there is growth in systems around Node.

At the request to cover some failures (to counter success stories for fairness), Sam from offered a counter-argument of why some shops cannot get there just yet – some libraries just do not exist yet (but time can help plug this gap). In the same vein, LinkedIn is still a huge Scala/Java shop, and in some use cases Play framework is easier to move their developers forward than to make a massive switch to Node (LinkedIn still uses Node for their mobile services, and they are very fast).

The takeaway is that while Node.js is obviously at an inflection point and it is not hard to find success stories, but it will never be a magic bullet that will solve all the problems. It is imperative to ensure that the problem to be solved is a good fit for Node.

Joe McCann, COO - The Node Firm
Joe McCann, COO – The Node Firm

After lunch and an opportunity to socialize with actual people in the 3D world and full HD (what a concept!), Erik Toth, Principal Software Engineer at PayPal took us through the fuzzy and complicated world of moving a large developer workforce to NodeJS. He provided a set of hard won tips on how to handle resistance and problems along the road. First off, accepting that your rationale might be non-technical is healthy – Node is as much a way of life as a technology. There will be critics, they will parrot questions, represent opinions they’ve read as their own, use outdated information, micro benchamrks, assume you have ulterior motives, they will underestimate you (real engineers use something else). They will assume that because they haven’t heard of something, it doesn’t exist, and all of this without even looking at the technology.

The secret to overcome this? Influencers. You need to have an advocate that absolutely believes in it, and things will start happening. Setting limits is key – focus and avoid some things, avoid inventing problems to have something to solve, solving other peoples’ problems, or solving solved problems. Only fix what is broken. For PayPal, it was primarily velocity and productivity, so there was a need to scale people, product and technology. This requires an ecosystem: GitHub, npm, tooling.

Erik pointed at silos as a serious impediment (enterprises like Game of Thrones). They breed distrust, create shallow alliances based on selfish goals, black-boxes teams and code. He suggested elimination of backlog as a concept (where features go to die).

A key takeaway for individual developers?

Individuals rarely solve real problems – benefit from the work of others. This is not revolutionary, just harder in the enterprise. Finally, the quality of code should not depend on the access rights (“The only thing that should differentiate public and private modules is access rights”). Be proud of your code, expect copypasta, learn semver and teach others.

Oh, and the key takeaway from the Q/A: “We treated Node as something that will go away in 5 years, and we didn’t want to impose custom solutions to those that will come after us”.

Erik Toth, Principal Software Engineer - PayPal
Erik Toth, Principal Software Engineer – PayPal

Ian Livingstone, VP of Engineering of GoInstant told a war story of using Node in two apps, and how the second time was informed by all the mistakes made in the first app. The first app was written in the time when Node was instable. More importantly, growing code base is a maintenance nightmare, which greatly underscores the important of writing small modules instead of large monolithic apps (a common theme throughout the NodeDay).

Ian offered many tips collected the hard way. For example, state must live in a class instance, not in modules. Furthermore, when a class has a state, it must have both ‘initialize’ and ‘destruct’ methods. Another trick that helped was the use of JSON schema, which makes it easy to exhaustively test code. This includes schema for Request and Response (run only in development, limited in production for performance reasons).

On the topic of UI testing, Ian pointed out that multi-user integration tests are required to excersize user interfaces. Selenium was an obvious first choice, but the tests ended up verbose and very fragile, and took a long time to run. NPM to the rescue: a tool called ‘Barista’ (extension to Mocha) proved to be a better solution. Lesson: the power at your finger tips through the Node eco-system is incredible.

The need to avoid one giant code base cannot be stressed enough: like in Java’s “God object”, leaked responsibilities between modules made the code base into a giant ‘house of cards’ that was fragile and very hard to test, greatly diminishing trust in the code. Solution: small, simple and independent modules.

The second time around, applying all these hard-won lessons had a hugely positive effect – it changed the conversation. A small downside – longer install times via NPM (solved by using a proxy backed by S3). Node works better when it has one thing to do, favoring an architecture with many small and simple services. As a convenience, each service has a client library that wraps the transport concerns, making it easy to consume and test. Using upstart and supervisord provided for process lifecycle management.

Takaway – Node.js is power, and many problems have already been solved by the community.

Ian Livingstone, VP of Engineering - GoInstant
Ian Livingstone, VP of Engineering – GoInstant

Trevor Norris, NodeJS Core Maintainer from Mozilla took us on a journey much closer to the metal, under the layers of abstractions we are so used to in our daily work. The aim of the presentation was not to prove that all abstractions are bad, just to make their cost clear. He offered a number of performance tips related to intimate knowledge of V8 engine that powers Node (example: inlined functions can’t contain more than 600 characters, including comments; inlined functions must be declared close to the call site etc.). This level of detail is not very practical to everyday Node developers writing modules at higher level of the stack, but are a reality for Node Core developers that must worry about the performance envelope of the bedrock on which everything else resides.

In that spirit, Trevor continued to provide advise on the most obvious ‘sins of our abstractions’ – the fact that some events like DNS and FS are in fact pseudo-async (they tie up a thread in the background), and again sternly warned about using the new execSync function that is a very odd animal in the ansync world of Node.

Key takeways: write code that you can maintain, write code that doesn’t crash – the slowest code is the code that doesn’t run. And finally, code for your case. Very few developers actually code with Node Core performance requirements in mind.

Trevor Norris, NodeJS Core Maintainer - Mozilla
Trevor Norris, NodeJS Core Maintainer – Mozilla

Richard Rodger, Founder of Near Form really drove the point of keeping Node processes small (a motif of the entire conference). This ensures that services are easy to restart, which makes for a fault-tolerant system. With a number of services running in the system, a communication mechanism is require (messages). Processes need to be small (100 lines of code), a few screens to scroll, that can fit inside your head. Frameworks are discouraged – use modules and composition.

A good message broker that supports pub/sub is very handy in this architecture – it makes the system loosely coupled and extendable. Micro-services also make rewrites cheap, avoiding project management ceremony. They localize bad coding choices, making them less of an issue because they are confined to a single micro-service.

Not all services are created equal – some are critical and require a lot of attention, some do not. By grouping services by quality, focus can be applied where it counts. As always, fixing a bug later in the cycle carries a higher cost, but with micro-services, the curve is less steep.

Richard provided some real-world examples of approaching a problem using micro-service architecture. The approach flows from user stories, to capabilities, services and messages. After that, decisions need to be made on how to run in production: what performance, scale, behavior and message transport are needed. Of course, production rollout need not be extensive – a good start can be to run all the micro-services on one machine and cluster only the ones that actually need it.

As a final comment, Richard agreed that micro-services can be built in any language – Node.js is simply a very suitable option, but not the only one.

Richard Rodger, Founder - NearForm
Richard Rodger, Founder – NearForm
NodeDay audience - full house
NodeDay audience – full house
Mr. JavaScript himself - Douglas Crockford, unaware of my paparazzi tendencies.
Mr. JavaScript himself – Douglas Crockford, unaware of my paparazzi tendencies.

All in all, it was a well organized event and an amazing opportunity to talk to a number of key people in the Node.js community all in the same day. In effect, it was like its topic – I was able to talk to more people and learn more in a shorter period of time and virtually no downtime – like a true Node app. Kudos to Bill Scott for being a gracious host and Lenny Markus for ensuring it all went without a hitch.

© Dejan Glozic, 2014


Node.js on the Road – San Francisco


This post is a first of sorts – as with my review of RESS: The Essentials that opened my Reviews category, it will be the first in the Events. In this context, plural is actually justified because it covers not one but two events: Node.js on the Road on Feb 27th in San Francisco’s hosted by New Relic and NodeDay on Feb 28th in San Jose hosted by PayPal. Both are now available – you can read the NodeDay post here.

I ended up at ‘Node.js on the Road’ by a stroke of luck. Ever since the days when EclipseCon was being hosted in Santa Clara Convention Center, my preferred travel plan was to fly to SFO, then drive down 101 South for the event. The 11:45am Air Canada YYZ-SFO flight is just perfect for this, and it gives me a free afternoon and evening in San Francisco. Just when I was trying to figure out what to do with the available time, along came a tweet about ‘Node.js on the Road’ to be held 6-9pm PT in the New Relic’s downtown SF location on Feb 27th. Seemed like a perfect way to spend the evening before heading South.

Looking for parking on a work day in downtown San Francisco is not for the faint of heart. I ended up parking in an the underground garage of the nearby building. I arrived a bit ahead of time but it turned out that 6pm was ‘when the food was about to arrive, and also the people’, not the actual start of the event. Just as well, because my jet-lagged mind welcomed some time to pipe down and also solved my dinner problem.

Node.js on the Road Agenda
Node.js, food and beer - a perfect combination
Node.js, food and beer – a perfect combination

The even was opened by Bryan Cantrill, SVP Engineering of Joyent, the corporate steward of Node.js. It was a welcome introduction, explaining the format of the evening which was a bit vague to me. He also said something counter-intuitive for many people skeptical of ‘JavaScript on the server’: “Node.js is the most debuggable dynamic environment”. Veterans of Java and JVM would love to disagree, but everybody knows that huge VM dumps created when a real-world production Java program gives up the ghost are a nightmare to analyze, so it is not hard to imagine where he is coming from.

Bryan Cantrill, SVP Engineering, Joyent

First on the list of presenters was TJ Fontaine, the newly minted Node.js Core team lead, replacing Isaac Schlueter who left to become the CEO of the new NPM startup (short for ‘Node Package Manager’, a registry largely responsible for the thriving Node.js ecosystem). His talk was centered around the road ahead – what is left to finish in Node before it can be declared v1.0, how companies and individuals using it can contribute and inform the process. There are multiple ways to make Node better at this point: reporting bugs as we are hitting them in production, contributing source for either enhancements or bug fixes, helping with the documentation (Node currently has a bit of a documentation problem). There are also multiple ways to ensure the community continues to thrive – Stack Overflow, IRCs, mailing lists, meetups. Not all of these options are equally appealing to all (TJ is not a fan of mailing lists or IRCs) – YMMV.

TJ also spent some time to announce what is coming in Node 0.12 which is ‘very immanent’. It was largely a rehash of what was already mentioned in a related blog post. Some are expected (performance improvements), some are too detailed for my current knowledge of Node.js (TLS enhancements, buffers, streams) although I am sure there are developers which are expecting them with anticipation. And then some are controversial. Take execSync – after all the effort to explain that Node.js is asynchronous to the core, being able to run blocking execution seems like a compromise with the devil (to TJ’s defense, he recommended extreme caution when using this function for all the easily understandable reasons).

TJ Fontaine, Node.js Core team lead
TJ Fontaine, Node.js Core Team Lead

Next on the agenda was Dav Glass, Node.js architect from Yahoo!. In my chat with the presenters prior to the event, I learned that the idea is to bring different presenters at each stop to tell the story of their own use of Node.js in real world projects. Yahoo! seems to be turning into a massive Node.js shop, and Dav illustrated it with the ‘We love Node’ slide. Yahoo! runs Node both in production and internally, using a huge Continuous Integration (CI) system called Manhattan. At the moment, some of the boxes in Yahoo! pages are served by Node (others use Java or PHP). As expected, transition to Node in running systems is often a gradual transition, not a big switch.

Dav’s war stories? In his experience, it is rarely Node’s problem – most of the time issues end up being caused by downstream modules deep in the bowels of the running system. Again, since the system needs to keep going, Node needs to be inserted into the giant old infrastructure. Yahoo! is using Node largely unmodified – the only minor changes being done around security and authentication.

Another common theme is the use of NPM to serve internal modules – not all the Node modules are open sourced, requiring a way to serve them internally. One of the internal code quality requirement is that you are not allowed to share a library unless it has 80% test code coverage.

Dav demonstrated some impressive throughput numbers, from processes serving > 1000 rps (requests per second), to some extreme examples of hot rod modules hitting insane figures of 25000-36000 rps.

Dave Glass, Node.js architect, Yahoo
Dav Glass, Node.js Architect, Yahoo! in front of ‘We Love Node!’ slide.

Jeff Harrell is a new hire who came to PayPal 2 years ago as a new Director of UI Engineering to ‘disrupt the PayPal system’. His story has already been somewhat familiar to me due to the write up on the Project Kraken. Jeff added new observations to this story, explaining that the goal of their wholesale move from Java to Node was every bit as driven by developer performance as app performance. In a what is turning into a familiar pattern, their topology has Node front end hooked up to the legacy infrastructure that is either difficult to change, or is not a good fit for Node. PayPal is currently running 20+ Node apps, written by 200 developers (both new hires and Java converts).

According to Jeff, converting Java developers was an interesting process. He observed stages not very different from Kubler-Ross stages of grief, from denial, anger and bargaining to acceptance and finally enthusiasm for Node at the end of the curve. Of course, not everything was perfect: PayPal, being a payment processing company, is a heavy user of SSL, and in Node v0.10, SSL cipher time left much to be desired. This was just a point in time problem, however: in v0.11 and the upcoming v0.12 SSL performance is much improved, and will continue to be a high priority.

Jeff Harrell, Director of UI Engineering, PayPal
Jeff Harrell, Director of UI Engineering, PayPal

The last part of the event was a Q/A panel with all the participants. A lot of questions were for Jeff Harrell – it seems that audience uses PayPal as a good model of a traditional Java shop moving to Node with a clear head, with a desire to improve productivity and make Lean UX real and practical, whereas they have a harder time identifying with the more complex Yahoo!’s systems (a lot of participants work for startups that are years away from reaching Yahoo! scale). Some participants expressed the familiar concern about type safety (or lack thereof) in JavaScript. According to panelists, smaller modules in Node tend to make type safety less of an issue. In fact, Node.js inherits from Unix philosophy – it forces you to think small and combine (and write tests – smaller components are more testable).

As for real-world robustness, Yahoo! has provided Open Source launchers (they are on GitHub) which run one process per core, and restart process if there are problems.

Some participants wanted to know how to get Node introduced into a skeptical organization. Panelists agreed that doing it instead of asking for permission is the way to go (and also making it provable – you can argue technology all day long, but when business people realize you can do it faster, they like the outcome).

Yahoo! is definitely not in that phase right now – Yahoo! has thousands of engineers that all know JavaScript, so convincing them is an easier task. With such a brain power, it is easy to go crazy with inter-process communication and try to squeeze the last ounce of performance using binary protocols, but the recommendation is to start with HTTP and see if anything else is needed.

Q/A panel - TJ, Jeff and Dave
Q/A panel – TJ, Jeff and Dav

All in all, it was an enjoyable and not over-long evening, and a perfect prequel to the NodeDay on Friday. Read about NodeDay in my next post.

Nowhere to park on Spear street
Nowhere to park on Spear street

© Dejan Glozic, 2014