NodeDay 2014

This post is the continuation of my trip report that started with part 1 on Node.js on the Road. NodeDay was organized and hosted by PayPal in their building in San Jose, CA, on Feb 28th 2014. The focus of the one-day conference was Node.js in the enterprise, which was a great fit for what we are now doing with Node.js in IBM Rational. Add in the opportunity to talk with Bill Scott, Richard Ragan and Jeff Harrell about their use of Dust.js in production, and it all added up to a promising day.

TL;DR:

I drove down 101 South from San Francisco to my hotel in San Jose in a calm evening. Last thing I remember before I fell asleep was muted howling of unusually strong wind outside my hotel window. Then I woke up into a morning of storm reports, flash floods, mud slides and falling rocks throughout the region. I must have been very tired.

Luckily, this was largely over by the time I finished breakfast, and PayPal was only 2 miles away.

We got badges and bracelets at the entrance. Badges doubled as agenda – simple and minimalistic, as you would expect from a Node.js event.

The event was kicked off by Bill Scott, Senior Director of UI Engineering, PayPal and our host for the day. The first keynote was by Eran Hammer, Senior Architect of Walmart Labs, who was singlehandedly responsible for pushing me off the fence and into Node.js development. The watershed event was Black Friday 2013, when 55% of Walmart traffic was from mobile devices, and handled by Node.js servers. The uneventful day for the pager-carrying developers cemented their trust that Node.js has ‘arrived’, resulting in Walmart doubling down on Node and spreading it to the rest of the enterprise.

Eran’s focus was on saving the typical enterprise Java developer from living a double life – passionately using Node at home, and wrestling with the legacy projects behind the firewall at work (illustrated with a contrast between unicorns and rainbows at the top of the post, and the Intranet wall of dead puppies – see the rest of the presentation). A common myth is that Open Source is all done by little guys, individuals and startups – just technology and no politics. In reality, Open Source is sustained and flourishing in the enterprise – nobody is using the Open Source more than the enterprise (and Walmart is not exactly a startup either). Why? Because people see the leverage that working for large companies can give them. Hence, we need more ‘Inc.’ in our Node, the community to be one, instead of Enterprise people being on the side.

We (the enterprise) should not feel like we are on the fringes. For example, Joyent has been the best supporter of the project, and one of the biggest consumers of Node. The business of paying for support for Open Source projects is still alive and a valid one. With NPM in the news recently, it is clear that profitability is not a dirty word, it is the only way to sustain the community. Issues such as private NPM repositories that are so important for enterprises were initially hard to take seriously, but it is obvious it is understood now, judging by half a dozen solutions already available. It shows that in many cases, building on what is already available and perhaps sponsoring/adopting core contributors is a cost-effective way to get things done and ensure critical support. Doing everything in-house is an anti-pattern.

TJ Fountaine, Node.js Project Lead from Joyent has mostly repeated his talk from ‘Node.js on the Road’, which was a good thing to do since there was no overlap in the crowd between the two events (not counting myself of course :-). You can read my report in the previous post with no loss of data.

The funniest part of TJ’s keynote is the extent he went to in order to ensure that he is not mixed up with the express.js author TJ Holowaychuk:

Adam Baldwin, Chief Security Officer – &yet, has covered a touchy subject – Node.js security. With all the enthusiasm for the new way of working Node provides, enterprise is very touchy around security and it is a big topic when considering Node adoption. Adam suggests protecting what makes your money, and consider availability as an aspect of security (if the app is down, who cares if it is secure). The process of how you handle the vulnerability is the key (but you will screw it up anyway).

When it comes to security, nodejs-sec mailing list is important because all security-related announcements go there (very few so far, knock on wood). In reality, most of the issues will be in Node modules, not the core – security is built into core, it is not a bolt-on module. The enterprise is responsible for what you require() – there has to be a process in place for vetting modules, and private NPM repositories. Linting is recommended (using npm install precommit-hook).

Security-related test cases are important (some orgs only write test cases for the happy path). One useful library that can help – retire.js (scans your web app or Node app for use of vulnerable JS libraries).

With all said and done, the greatest vulnerability in the enterprise?

Joe McCann, COO of The Node Firm has addressed the business case for Node. There are many signs of Node.js maturity – exponential growth in NPM, number of books published, enterprises going into production with Node. He posited that a mission statement for Node should read: “An easy way to build fast and scalable applications”. As to ‘Why Now’, he suggested that legacy stacks are impending innovation and make companies less competitive.

Joe suggested five business tenets when making the case for Node: innovation, productivity, developer joy, hiring/retaining talent and cost savings. He focused on a few success stories:

PayPal had velocity as the key driving point. Node provided boost to the workflow, allowing faster iteration and innovation. After a long history of Java/C++, developers are energized by their jobs again. A frequently re-twitted quote:

Groupon moved from a monolithic application that became a burden and a blocker to a service-oriented architecture. Node.js now handles the same amount of traffic using less hardware (direct cost savings – a music to every CFO’s ears). At the same time, page load times decreased by 50%. The use of a common layout service allowed them to retain the ability to make site-wide design changes quickly.

Walmart achieved Black Friday CPU utilization hovering around 1% (you want to be bored when on pager duty). In addition, quality of candidates after switching to Node.js increased – Node is a magnet for talent. This is important – Node could be a differentiator in hiring/retaining.

In Yahoo!, Node.js services are handling 2M requests/minute, 200 developers write Node, there are 500 internal and 800 external Node modules. Node brought speed and ease of development.

Back to business tenets, innovation is helped by building things faster and more efficiently. Productivity is helped by terse syntax and absence of thick layers of bureaucracy that slow developers down. “Productive developer is a happy developer”, which helps with developer joy and helps stability and retention. As for hiring/retaining talent, there is a magnetism of working on Node that is unique. Finally, cost savings are real and palpable: fewer developers writing apps in less amount of time, using less hardware. Using Node in multiple projects multiplies the savings.

This topics generated lively discussion, and a member of the audience claimed that similar enthusiasm was heard 10 years ago around Ruby on Rails. So what is different with Node? The consensus seems to be that since JavaScript is gravity on the client, using JavaScript ‘all the way down’ provides benefits of simplification and ‘single stack’ that cannot be duplicated in other stacks.

The reality of enterprise architecture is that mixed stacks (Node.js + Java/PHP) are everywhere. Right now, front end in Node.js is a safe bet, but there is growth in systems around Node.

At the request to cover some failures (to counter success stories for fairness), Sam from Amazon.com offered a counter-argument of why some shops cannot get there just yet – some libraries just do not exist yet (but time can help plug this gap). In the same vein, LinkedIn is still a huge Scala/Java shop, and in some use cases Play framework is easier to move their developers forward than to make a massive switch to Node (LinkedIn still uses Node for their mobile services, and they are very fast).

The takeaway is that while Node.js is obviously at an inflection point and it is not hard to find success stories, but it will never be a magic bullet that will solve all the problems. It is imperative to ensure that the problem to be solved is a good fit for Node.

After lunch and an opportunity to socialize with actual people in the 3D world and full HD (what a concept!), Erik Toth, Principal Software Engineer at PayPal took us through the fuzzy and complicated world of moving a large developer workforce to NodeJS. He provided a set of hard won tips on how to handle resistance and problems along the road. First off, accepting that your rationale might be non-technical is healthy – Node is as much a way of life as a technology. There will be critics, they will parrot questions, represent opinions they’ve read as their own, use outdated information, micro benchamrks, assume you have ulterior motives, they will underestimate you (real engineers use something else). They will assume that because they haven’t heard of something, it doesn’t exist, and all of this without even looking at the technology.

The secret to overcome this? Influencers. You need to have an advocate that absolutely believes in it, and things will start happening. Setting limits is key – focus and avoid some things, avoid inventing problems to have something to solve, solving other peoples’ problems, or solving solved problems. Only fix what is broken. For PayPal, it was primarily velocity and productivity, so there was a need to scale people, product and technology. This requires an ecosystem: GitHub, npm, tooling.

Erik pointed at silos as a serious impediment (enterprises like Game of Thrones). They breed distrust, create shallow alliances based on selfish goals, black-boxes teams and code. He suggested elimination of backlog as a concept (where features go to die).

A key takeaway for individual developers?

Individuals rarely solve real problems – benefit from the work of others. This is not revolutionary, just harder in the enterprise. Finally, the quality of code should not depend on the access rights (“The only thing that should differentiate public and private modules is access rights”). Be proud of your code, expect copypasta, learn semver and teach others.

Oh, and the key takeaway from the Q/A: “We treated Node as something that will go away in 5 years, and we didn’t want to impose custom solutions to those that will come after us”.

Ian Livingstone, VP of Engineering of GoInstant told a war story of using Node in two apps, and how the second time was informed by all the mistakes made in the first app. The first app was written in the time when Node was instable. More importantly, growing code base is a maintenance nightmare, which greatly underscores the important of writing small modules instead of large monolithic apps (a common theme throughout the NodeDay).

Ian offered many tips collected the hard way. For example, state must live in a class instance, not in modules. Furthermore, when a class has a state, it must have both ‘initialize’ and ‘destruct’ methods. Another trick that helped was the use of JSON schema, which makes it easy to exhaustively test code. This includes schema for Request and Response (run only in development, limited in production for performance reasons).

On the topic of UI testing, Ian pointed out that multi-user integration tests are required to excersize user interfaces. Selenium was an obvious first choice, but the tests ended up verbose and very fragile, and took a long time to run. NPM to the rescue: a tool called ‘Barista’ (extension to Mocha) proved to be a better solution. Lesson: the power at your finger tips through the Node eco-system is incredible.

The need to avoid one giant code base cannot be stressed enough: like in Java’s “God object”, leaked responsibilities between modules made the code base into a giant ‘house of cards’ that was fragile and very hard to test, greatly diminishing trust in the code. Solution: small, simple and independent modules.

The second time around, applying all these hard-won lessons had a hugely positive effect – it changed the conversation. A small downside – longer install times via NPM (solved by using a proxy backed by S3). Node works better when it has one thing to do, favoring an architecture with many small and simple services. As a convenience, each service has a client library that wraps the transport concerns, making it easy to consume and test. Using upstart and supervisord provided for process lifecycle management.

Takaway – Node.js is power, and many problems have already been solved by the community.

Trevor Norris, NodeJS Core Maintainer from Mozilla took us on a journey much closer to the metal, under the layers of abstractions we are so used to in our daily work. The aim of the presentation was not to prove that all abstractions are bad, just to make their cost clear. He offered a number of performance tips related to intimate knowledge of V8 engine that powers Node (example: inlined functions can’t contain more than 600 characters, including comments; inlined functions must be declared close to the call site etc.). This level of detail is not very practical to everyday Node developers writing modules at higher level of the stack, but are a reality for Node Core developers that must worry about the performance envelope of the bedrock on which everything else resides.

In that spirit, Trevor continued to provide advise on the most obvious ‘sins of our abstractions’ – the fact that some events like DNS and FS are in fact pseudo-async (they tie up a thread in the background), and again sternly warned about using the new execSync function that is a very odd animal in the ansync world of Node.

Key takeways: write code that you can maintain, write code that doesn’t crash – the slowest code is the code that doesn’t run. And finally, code for your case. Very few developers actually code with Node Core performance requirements in mind.

Richard Rodger, Founder of Near Form really drove the point of keeping Node processes small (a motif of the entire conference). This ensures that services are easy to restart, which makes for a fault-tolerant system. With a number of services running in the system, a communication mechanism is require (messages). Processes need to be small (100 lines of code), a few screens to scroll, that can fit inside your head. Frameworks are discouraged – use modules and composition.

A good message broker that supports pub/sub is very handy in this architecture – it makes the system loosely coupled and extendable. Micro-services also make rewrites cheap, avoiding project management ceremony. They localize bad coding choices, making them less of an issue because they are confined to a single micro-service.

Not all services are created equal – some are critical and require a lot of attention, some do not. By grouping services by quality, focus can be applied where it counts. As always, fixing a bug later in the cycle carries a higher cost, but with micro-services, the curve is less steep.

Richard provided some real-world examples of approaching a problem using micro-service architecture. The approach flows from user stories, to capabilities, services and messages. After that, decisions need to be made on how to run in production: what performance, scale, behavior and message transport are needed. Of course, production rollout need not be extensive – a good start can be to run all the micro-services on one machine and cluster only the ones that actually need it.

As a final comment, Richard agreed that micro-services can be built in any language – Node.js is simply a very suitable option, but not the only one.

All in all, it was a well organized event and an amazing opportunity to talk to a number of key people in the Node.js community all in the same day. In effect, it was like its topic – I was able to talk to more people and learn more in a shorter period of time and virtually no downtime – like a true Node app. Kudos to Bill Scott for being a gracious host and Lenny Markus for ensuring it all went without a hitch.

© Dejan Glozic, 2014