Sharing micro-service authentication using Nginx, Passport and Redis

Abgeschlossen_1
Wikimedia Commons, Abgeschlossen 1, by Montillona

And we are back with the regularly scheduled programming, and I didn’t talk about micro-services in a while. Here is what is occupying my days now – securing a micro-service system. Breaking down a monolith into a collection of micro-services has some wonderful properties, but also some nasty side-effects. One of them is authentication.

The key problem of a micro-service system is ensuring that its federated nature is transparent to the users. This is easier to accomplish in the areas that are naturally federated (e.g. a collection of API end points). Alas, there are two areas where it is very hard to hide the modular nature of the system: composing Web pages from contributions coming from multiple services, and security. I will cover composition in one of the next posts, which leaves us with the topic of the day.

In a nutshell, we want a secure system but not at the expense of user experience. Think about European Union before 1990. In some parts of Europe, you could sit in a car, drive in any direction and cross five countries before the sunset. In those days, waiting in line at the custom checkpoint would get old fast and could even turn into quite an ordeal in extreme cases.

Contrast it to today – once you enter EU, you just keep driving, passing countries as if they are Canadian provinces. Much better user experience.

We want this experience for our micro-service system – we want to hop between micro-services, be secure yet not being actively aware that the system is not a monolith.

It starts with a proxy

A first step in securing a micro-service system starts at a proxy such as Nginx. Placing a multi-purpose proxy in front of our services have several benefits:

  1. It allows us to practice friendly URL architecture – we can proxy nice front end URLs such as http://foobar.io/users or http://foobar.io/projects to separate micro-services (‘users’ and ‘projects’, respectively). It also goes around the fact that each Node.js service runs on a separate port (something that JEE veterans tend to hate in Node.js, since several apps running in the same JEE container can share ports).
  2. It allows us to enable load balancing – we can proxy the same front end location to a collection of service instances running in different VMs or virtual containers (unless you are using a PaaS, at which point you just need to increment the instance counter).
  3. It represents a single domain to the browser – this is beneficial when it comes to sharing cookies, as well as making Ajax calls without tripping the browser’s ‘same origin’ policy (I know, CORS, but this is much easier)

If we wanted to be really cheap, we could tack on another role for the front end proxy – authentication. Since all the requests are passing through it, we can configure a module to handle authentication as well and make all the micro-services behind it handle only authenticated requests. Basic auth module is readily available for Nginx, but anything more sophisticated is normally not done this way.

Use Passport

Most people will need something better than basic auth, and since we are using Node.js for our micro-service system, Passport is a natural choice. It has support for several different authentication strategies, including OAuth 1.0 and OAuth 2.0, and several well known providers (Twitter, Facebook, LinkedIn). You can easily start with a stock strategy and extend it to handle your unique needs (this will most likely be needed for OAuth 2.0 which is not a single protocol but an authentication framework, much to the dismay of Eran Hammer).

Passport is a module that you insert in your Node.js app as middleware, and hook up to the Express session. You need to expose two endpoints: ‘/auth/<provider>’ and ‘/auth/<provider>/callback’. The former is where you redirect the flow in order to start user login, while the latter is where the Auth server will call back after authenticating the user, bringing in some kind of authorization code. You can use the code to go to the token endpoint and obtain some kind of access token (e.g. bearer token in case of OAuth 2.0). With the access token, you can make authenticated calls to downstream services, or call into the profile service and fetch the user info. Once this data is obtained, Passport will tack it on the request object and also serialize it in the session for subsequent use (so that you don’t need to authenticate for each request).

Passport module has a very nice web site with plenty of examples, so we don’t need to rehash them here. Instead, I want us to focus on our unique situation of securing a number of micro-services working together. What that means is:

  1. Each micro-service needs to be independently configured with Passport. In case of OAuth (1.0 or 2.0), they can share client ID and secret.
  2. Each micro-service can have a different callback URL as long as they all have the same domain. This is where we reap the benefit of the proxy – all the callback URLs do have the same domain thanks to it. In order to make this work, you should register your client’s callback_uri with the authentication provider as the shared root for each services’ callback. An actual callback passed to the authentication endpoint for each micro-service can be longer than the registered callback_uri as long as they all share common root.
  3. Each micro-service should use the same shared authentication strategy and user serialization/deserialization.

Using this approach, we can authenticate paths served by different micro-services, but we still don’t have Single Sign-On. This is because Express session is configured using an in-memory session store by default, which means that each micro-service has its own session.

Shared session cookie

This is not entirely true: since we are using the default session key (or can provide it explicitly when configuring session), and we are using single domain thanks to the proxy, all Node.js micro-services are sharing the same session cookie. Look in the Firebug and you will notice a cookie called ‘connect.sid’ once you authenticate. So this is good – we are getting there. As I said, the problem is that while the session cookie is shared, this cookie is used to store and retrieve session data that is in memory, and this is private to each micro-service instance.

This is not good: even different instances of the same micro-service will not share session data, let alone different micro-services. We will be stuck in a pre-1990 Europe, metaphorically speaking – asked to authenticate over and over as we hop around the site.

Shared session store

In order to fix this problem, we need to configure Express session to use an external session store as a service. Redis is wonderfully easy to set up for this and works well as long as you don’t need to persist your session forever (if you restart Redis, you will lose session data and will need to authenticate again).


var ropts = {
   host: "localhost",
   port: 5556,
   pass: "secret"
}

...

    app.use(express.session({ key: 'foobar.sid',
                             store: new RedisStore(ropts),
                             secret: 'secret'}));
    app.use(passport.initialize());
    app.use(passport.session());

I am assuming here that you are running Redis somewhere (which could range from trivial if you are using a PaaS, to somewhat less trivial if you need to install and configure it yourself).

What we now have is a system joined at both ends – Nginx proxy ensures session cookie is shared between all the micro-service instances it proxies, and Redis store ensures actual session data is shared as well. The corollary of this change is that no matter which service initiated the authentication handshake, the access token and the user profile are stored in the shared session and subsequent micro-services can readily access it.

micro-authentication

Single Sign Off

Since we have Redis already configured, we can also use it for pub/sub to propagate the ‘logout’ event. In case there is state kept in Passport instances in micro-services, a system-wide logout for the session ensures that we don’t have a “partially logged on” system after a log out in one service.

I mentioned Redis just for simplicity – if you are writing a micro-service system, you most likely have some kind of a message broker, and you may want to use it instead of Redis pub/sub for propagating logout events for consistency.

Calling downstream services

Not all micro-services will need full Passport configuration. You can configure services that require access token – they can just look for ‘Authorization’ header and refuse to do anything if it is not present. For example, for OAuth 2.0 authentication, the app will need something like:


Authorization: Bearer 0b79bab50daca910b000d4f1a2b675d604257e42

The app can go back to the authentication server and verify that the token is still valid, or go straight to the profile endpoint and obtain user profile using the token (this doubles as token validation because the profile service will protest if the token is not valid). API services are good candidates for this approach, at least as one of the authentication mechanisms (they normally need another way for app-to-app authentication that does not involve an actual user interacting with the site).

What about Java or Go?

This solution obviously works great if all the micro-services are written in Node.js. In a real-world system, some services may be written using other popular stacks. For example, what will happen if we write a Java service and try to participate?

Obviously, running a proxy to the Java micro-service will ensure it too has access to the same session cookie. Using open source Redis clients like Jedis will allow it to connect to the same session store. However, the picture is marred slightly by the fact that Express session signs the cookie with a combination of HMAC-Sha256 and ‘base64’ digest, plus some additional tweaking. This is obviously a very Express-centric approach and while it can be re-created on the Java side, there is this lingering feeling we created a Node-centric system and not a stack-agnostic one.

Java has its own session management system and you can see the JSESSIONID cookie sitting next to the one created by Express. I will need to study this more to see if I can make Java and Node share the session cookie creation and signing in a more stack-neutral way. In a system that is mostly Node.js with a Java service here and there, signing and unsigning the session cookie the way Express likes it may not be a big deal.

In addition, my current experimentation with Java points at creation of a JEE filter (middleware) that checks for the session cookie and redirects to authentication endpoints if a user is not found in the session. True Java authentication solutions are not used, which may or may not be a problem for you if you are a JEE veteran. JEE filters provide for wrapping HTTP requests so methods such as ‘getRemoteUser()’ can be implemented to provide the expected results to Java servlets.

I mentioned Go because it is an up and coming language that people seem to use for writing micro-services more and more these days. I have no idea how to write the session support for Go in this context so I am tossing this in just as food for thought. If you know more, drop me a line.

Closing thoughts

There are many alternative ways to solve this problem, and I profess my mixed feelings about authentication. For me, it is like broccoli – I understand that it is important and good for me, but I cannot wait to get to the ice cream, or whatever is the ice cream equivalent in writing micro-service systems. Spending time with Passport and OAuth 2.0, I had to learn more than I ever wanted to know about authentication, but I am fairly pleased with how the system works now. What I like the most is its relative simplicity (to the extend that authentication can be). My hope is that by avoiding smart solutions, the chances of the system actually working well and not presenting us with difficult edge cases every day are pretty good. I will report back if I was too naive.

© Dejan Glozic, 2014

Advertisements

Nodeconf.eu 2014: Trip Report (Part 1)

nodeconfeu-1

Shady’s back, tell a friend! Fresh from the green grass of Ireland where I attended (and presented) at this year’s nodeconf.eu, I am now back to report on it as promised.

This year’s conference is a second instance of a format started last year by Cian Ó Maidín and the friends from Near Form. The goal is to carefully curate talks across the Node.js community to ensure quality over quantity. I had a great pleasure attending the conference, particularly as I was one of the ‘carefully curated’ speakers this year. What adds a particular flare and sparkle to the event is the location – an actual Irish castle in Waterford, a south-eastern town of Ireland and the oldest of them all.

Getting to the Waterford castle involves an excerpt from the movie Plains, Trains and Automobiles – an itinerary that involves flying to Dublin, catching a bus to Waterford, then switching to a taxi that at some point needs to go over the river Suir via a private car ferry to get to the castle on the island.

The event started in the evening with a welcome reception that involved circus acts – a very ‘tall’ lady (see above) and a guy literately playing with fire.

nodeconfeu-2After-dinner entertainment included a lovely young lady playing a cello and singing in a way that fuses transitional Irish music and modern sensibilities – perfect for Irish hipsters (they do exist, don’t they?). Unfortunately her name escapes me – if you know it (and have a link to her home page), please drop me a comment (Edit: @fritzvd was kind enough to point out her name is Alana Henderson – thanks!).

nodeconfeu-3

Nodeconf.eu was held in a club house of the nearby Golf Club (part of the same Waterford Castle resort). For the next three days, our master of ceremonies was Mikeal, who was well known to most attendees (just look at your apps and if you require ‘request‘, you know him too).

nodeconfeu-4
Mikeal tells people to sit down so that we can start.

Conference opened with a welcome addressed by Cian, outlining what is awaiting us for the next three days, and upholding the conference Code of Conduct, which was wonderfully short:

  1. No harassment of any kind is allowed
  2. Please don’t fall into the Suir river (apparently somebody did not long ago)

The technical part of the conference started with Node’s own TJ Fontaine, Node.js core lead, with his ‘State of Node’ address. TJ posed that the industry is in a state that makes this a perfect timing for a JavaScript framework. He also re-iterated some of the key tenets of Node.js philosophy, including non-blocking I/O and ‘do one thing, and do it well’ ethos. Finally, he stressed that the evolution of Node.js is not about what other languages or frameworks are doing, but what is good for Node.js and JavaScript themselves.

nodeconfeu-6
TJ Fontaine delivers the State of Node address.

NearForm’s own Richard Rodger kicked off the Micro-Services block with the accumulated experience of deploying micro-services (and in particular Node.js micro-services) in production. He highlighted some natural advantages (scalability, flexibility of deployment) but also disadvantages (added latency). From his real-world experience, he concluded that business logic should be in the services (no core monolith), that developers should resist Tower of Babel (the temptation to use many languages and stacks) and to assume you can design upfront (services are ‘discovered’, not designed). Nevertheless, he reiterated one of the strong suits of micro-services – the fact that you can change your mind, swap databases mid-project (or anything else).

nodeconfeu-7
NearForm’s Richard Rodger kicks off the Micro-Services block.

Clifton Cunningham focused on a very hard problem of micro-services – the fact that while multiple services are responsible for various parts of the system, pages still need to share some content. He enumerated options used in the past – client-side stitching using Ajax, front-end server, Server Side Includes (SSIs), Edge-side includes (esi) etc. He presented his team’s take on the problem – an open-source module called Compoxure that has a number of advanced features to deal with the problems normally faced in production – performance, latency, failures and authentication. He also addressed the problem of delivering CSS and JS for this shared content across the micro-services.

nodeconfeu-8
Clifton Cunnigham on Compoxure.

Then it was time for me. My take was how my team in IBM DevOps Services decided to pursue micro-services in general, and Node.js in particular as we started evolving our system. I don’t need to go into details – this whole blog is a weekly chronicle of our journey. I added a twist from a position of doing Node.js in a large enterprise – the need to legally vet a large number of Node.js modules (causing legal to hate us) and the complexity of deploying a large number of services in a secure manner (causing OPS to hate us).

nodeconfeu-9
Dejan Glozic on Node.js micro-services at IBM (photo courtesy of Louis Faustino via SmugMug).

The last speaker in the Micro-Services block was Fred George, bringing his wealth of experiences building micro-services in a number of real-world projects. He brought forward several examples of projects using different technologies and architecture, unified in the fact that most of the time an event based (asynchronous) approach was a better fit than synchronous. Out of that experience he extracted a concept of a system where all services receive all the messages all the time, but the messages are semantically classifiable, forming ‘rapids’ (all events), ‘rivers’ (themed events) and ‘ponds’ (state/history).

nodeconfeu-10
Fred George on rapids, rivers and ponds.

After a coffee and JS cupcakes break, we switched to the ‘Production’ track, starting with Brian McCallister from Groupon walking us from the familiar experience of a company whose monolith has become so large and unwieldy that it eventually made adding new features virtually impossible. Groupon slowly and carefully migrated their two large installations (North America and Europe) to a micro-service system using Node.js, resulting in unblocking of further evolution, with performance and scalability improvements tossed into the mix. This was in a sense a partial continuation of the micro-service track, considering that Groupon’s new system shares many traits with the ones we talked about in the preceding talks.

nodeconfeu-11
Brian McCallister on building the I-Tier system at Groupon.

PayPal’s own Jeff Harrell zeroed on a number of anti-patterns of implementing Node.js in a real-world project in PayPal. PayPal made a wholesale transformation to Node.js that is still ongoing, and their large team contributed a number of these anti-patterns. Among them were: bringing baggage from the previous projects, re-creating monolithic applications using Node.js, Googling ‘how to do X in JavaScript’, wrapping everything in promises, sloppy async code, using Node.js for everything, and ignoring the ecosystem.

PayPal's Jeff Harrell on Node.js real world anti-patterns.
PayPal’s Jeff Harrell on Node.js real world anti-patterns.

The last speaker for the day was Aman Kohli from Citi Bank, bringing to us the experience of deploying Node.js to provide back-end services for mobile apps in an environment that is everything but forgiving when it comes to security, adherence to regulations and process. According to Aman, they chose Node because of the async event model, being ideally suited for mobile and sensor apps, the fact that it was approved for internal usage, that it required fewer controls, and due to the good success they had with using Hapi framework for building mobile API services.

Aman Kohli on using Node.js for mobile services in Citi Bank.
Aman Kohli on using Node.js for mobile services in Citi Bank.

At this point it was time to break for lunch. From several choices I picked to attend Kraken.js workshop for my afternoon activity, where I could pick the brains of Jeff Harrell and Erik Toth from PayPal on the philosophy and the plans for this open source suite we already use in our micro-services.

Evening R&R was provided to us with a combination of Irish whiskey tasting (not bad at all, but still prefer Scottish single malt) and a great local folk band treating us with a mix of traditional and Irish-treated covers.

Continue to the part 2 of the report.

© Dejan Glozic, 2014

HA All The Things

HA all the things

I hate HA (High Availability). Today everything has to be highly available. All of the sudden SA (Standard Availability) isn’t cutting it any more. Case in point: I used to listen to music on my way to work. Not any more – my morning meeting schedule intrudes into my ride, forcing me to participate in meetings while driving, Bluetooth and all. My 8 speaker, surround sound Acura ELS system hates me – built for high resolution multichannel reproduction, it is reduced to ‘Hi, who just joined?’ in glorious mono telephony. But I digress.

You know that I wrote many articles on micro-services because it is our ongoing concern as we are slowly evolving our topology away from monolithic systems and towards micro-services. I have already written about my thoughts on now to scale and provide HA for Node.js services. We have also solved our problem of handling messaging in a cluster using AMQP worker queues.

However, we are not done with HA. Message broker itself needs to be HA, and we only have one node. We are currently using RabbitMQ, and so far it has been rock solid, but we know that in a real-world system it is not a matter of ‘if’ but ‘when’ it will suffer a problem, bringing all the messaging capabilities of the system with it. Or we will mess around with the firewall rules and block access to it by accident. Hey, contractors rupture gas pipes and power cables by accident all the time. Don’t judge.

Luckily RabbitMQ can be clustered. RabbitMQ documentation is fairly extensive on clustering and HA. In short, you need to:

  1. Stand up multiple RabbitMQ instances (nodes)
  2. Make sure all the instances use the same Erlang cookie which allows them to talk to each other (yes, RabbitMQ is written in Erlang; you learn on the first day when you need to install Erlang environment before you install Rabbit)
  3. Cluster nodes by running rabbitmqctl join_cluster –ram rabbit@<firstnode> on the second server
  4. Start the nodes and connect to any of them

RabbitMQ has an interesting feature in that nodes in the cluster can join in RAM mode or in disc mode. RAM nodes will replicate state only in memory, while in disc mode they will also write it to disc. While in theory it is enough to have only one of the nodes in the cluster use disc mode, performance gain of using RAM mode is not worth the risk (performance gain of RAM mode is restricted to joining queues and exchanges, not posting messages anyway).

Not so fast

OK, we cluster the nodes and we are done, right? Not really. Here is the problem: if we configure the clients to connect to the first node and that node goes down, messaging is still lost. Why? Because RabbitMQ guys chose to not implement the load balancing part of clustering. The problem is that clients communicate with the broker using TCP protocol, and Swiss army knives of proxying/caching/balancing/floor waxing such as Apache or Nginx only reverse-proxy HTTP/S.

After I wrote that, I Googled just in case and found Nginx TCP proxy module on GitHub. Perhaps you can get away with just Nginx if you use it already. If you use Apache, I could not find TCP proxy module for it. It it exists, let me know.

What I DID find is that a more frequently used solution for this kind of a problem is HAProxy. This super solid and widely used proxy can be configured for Layer 4 (transport proxy), and works flawlessly with TCP. It is fairly easy to configure too: for TCP, you will need to configure the ‘defaults’, ‘frontend’ and ‘backend’ sections, or join both and just configure the ‘listen’ section (works great for TCP proxies).

I don’t want to go into the details of configuring HAProxy for TCP – there are good blog posts on that topic. Suffice to say that you can configure a virtual broker address that all the clients can connect to as usual, and it will proxy to all the MQ nodes in the cluster. It is customary to add the ‘check’ instruction to the configuration to ensure HAProxy will check that nodes are alive before sending traffic to them. If one of the brokers goes down, all the message traffic will be routed to the surviving nodes.

Do I really need HAProxy?

If you truly want to HA all the things, you need to now worry that you made the HAProxy a single point of failure. I told you, it never ends. The usual suggestions are to set up two instances, one standard and another backup for fail-over.

Can we get away with something simpler? It depends on how you define ‘simpler’. Vast majority of systems RabbitMQ runs on are some variant of Linux, and it appears there is something called LVS (Linux Virtual Server). LVS seems to be perfect for our needs, being a low-level Layer 4 switch – it just passes TCP packets to the servers it is load-balancing. Except in section 2.15 of the documentation I found this:

This is not a utility where you run ../configure && make && make check && make install, put a few values in a *.conf file and you’re done. LVS rearranges the way IP works so that a router and server (here called director and realserver), reply to a client’s IP packets as if they were one machine. You will spend many days, weeks, months figuring out how it works. LVS is a lifestyle, not a utility.

OK, so maybe not as perfect a fit as I thought. I don’t think I am ready for a LVS lifestyle.

How about no proxy at all?

Wouldn’t it be nice if we didn’t need the proxy at all? It turns out, we can pull that off, but it really depends on the protocol and client you are using.

It turns out not all clients for all languages are the same. If you are using AMQP, you are in luck. The standard Java client provided by RabbitMQ can accept a server address array, going through the list of servers when connecting or reconnecting until one responds. This means that in the event of node failure, the client will reconnect to another node.

We are using AMQP for our worker queue with Node.js, not Java, but the Node.js module we are using supports a similar feature. It can accept an array for the ‘host’ property (same port, user and password though). It will work with normal clustered installations, but the bummer is that you cannot install two instances on localhost to try the failure recovery out – you will need to use remote servers.

On the MQTT side, Eclipse Paho Java client supports multiple server URLs as well. Unfortunately, our standard Node.js MQTT module currently only supports one server. I was assured code contributions will not be turned away.

This solution is fairly attractive because it does not add any more moving parts to install and configure. The downside is that the clients becomes fully aware of all the broker nodes – we cannot just transparently add another node as we could in the case of the TCP load balancer. All the client must add it to the list of nodes to connect to for this addition to work. In effect, our code becomes aware of our infrastructure choices more than it should.

All this may be unnecessary for you if you use AWS since Google claims AWS Elastic Load Balancing can serve as a TCP proxy. Not a solution for us IBMers of course, but it may work for you.

Give me PaaS or give me death

This is getting pretty tiring – I wish we did all this in a PaaS like our own Bluemix so that it is all taken care of. IaaS gives you the freedom that can at times be very useful and allow you to do powerful customizations, but at other times makes you wish to get out of the infrastructure business altogether.

I told you I hate HA. Now if you excuse me, I need to join another call.

© Dejan Glozic, 2014

The Year of Blogging Dangerously

391px-Extremely_yummy_raspberry_cheesecake

Wow, has it been a year already? I am faking surprise, of course, because WordPress has notified me well ahead of time that I need to renew my dejanglozic.com domain. So in actuality I said ‘wow, will it soon be a year of me blogging’. Nevertheless, the sentiment is genuine.

It may be worthwhile to look back at the year, if only to reaffirm how quickly things change in this industry of ours, and also to notice some about-faces, changes of direction and mind.

I started blogging in the intent to stay true to the etymological sense of the word ‘blog’ (Web log). As a weekly diary of sorts, it was supposed to chronicle trials and tribulations of our team as it boldly goes into the tumultuous waters of writing Web apps in the cloud. I settled on a weekly delivery, which is at times doable, at other times a nightmare. I could definitely do without an onset of panic when I realize that it is Monday and I forgot to write a new entry.

Luckily we have enough issues we deal with daily in our work to produce enough material for the blog. In that regard, we are like a person who just moved into a new condo after his old apartment went up in flames and went to Ikea. If an eager clerk asks him ‘what do you need in particular’, his genuine answer must be ‘everything – curtains, rugs, new mattress, a table, chairs, a sofa, a coffee table …’.

At least that’s how we felt – we were re-doing everything in our distributed system and we were able to re-use very little from our past lives, having boldly decided to jump ahead as far as possible and start clean.

Getting things out of the system

That does not mean that the blog actually started with a theme or a direction. In the inaugural post The Turtleneck and The Hoodie, I proudly declared that I care both about development AND the design and refuse to choose. But that is not necessarily a direction to sustain a blog. It was not an issue for a while due to all these ideas that were bouncing in my head waiting to be written down. Looking back, I think it sort of worked in a general-purpose, ‘good advice’ kind of way. Posts such as Pulling Back from Extreme AJAX or A Guide to Storage for ADD Types were at least very technical and based on actual research and hands-on experience.

Some of the posts were just accumulated professional experience that I felt the need to share. Don’t Get Attached to Your Code or Dumb Code Good, Smart Code Bad were crowd pleasers, at least in the ‘yeah, it happened to me too’ way. Kind of like reading that in order to lose weight you need to eat smart and go outside. Makes a lot of sense except for the execution, which is the hard part.

344px-'Be_smart..Act_dumb^_-_NARA_-_513932

Old man yells at the cloud

Funnily enough, some of my posts, after using up all the accumulated wisdom to pass on, sound somewhat cranky in hindsight. I guess I disagreed with some ideas and directions I noticed, and the world ignored my disagreement and continued, unimpressed. How dare people do things I don’t approve of!

Two cranky posts that are worth highlighting are Swimming Against the Tide, in which I am cranky regarding client side MVC frameworks, and Sitting on the Node.js Fence, in which I argue with myself on pros and cons of Node.js. While my subsequent posts clearly demonstrate that I resolved the latter dilemma and went down the Node.js route hook, line and sinker, I am still not convinced that all that JavaScript required to write non-trivial Single Page Apps (SPAs) is a very good idea, particularly if you have any ambition to run them on mobile devices. But it definitely sounds funny to me now – as if I was expressing an irritated disbelief that, after publishing all the bad consequences of practicing extreme Ajax, people still keep doing it!

I heart Node.js

Of course, once our team went down Node.js route (egged on and cajoled by me), you could not get me to shut up about it. In fact, the gateway drug to it was my focus on templating solutions, and our choice of Dust.js (LinkedIn fork). By the way, it is becoming annoying to keep adding ‘LinkedIn fork’ all the time – that’s the only version that is actively worked on anyway.

Articles from this period are more-less setting the standard for my subsequent posts: they are about 1500 words long, have a mix of outgoing links, a focused technical topic, illustrative embedded tweets (thanks to @cra who taught me how not to embed tweets as images like a loser). As no story about Node.js apps is complete without Web Sockets and clustering, and both were dully covered.

Schnorr_von_Carolsfeld_Bibel_in_Bildern_1860_006

I know micro-services!

Of course, it was not until I went to attend NodeDay in February that a torrent of posts on micro-services was unleashed. The first half of 2014 was all ablaze with the posts and tweets about micro-services around the world anyway, which my new Internet buddy Adrian Rossouw dully documented in his Wayfinder field guide. It was at times comical to follow food fights about who will provide the bestest definition of them all:

If you follow a micro-services tag for my blog, the list of posts is long and getting longer every week. At some point I will stop tagging posts with it, because if everything is about them, nothing is – I need to be more specific. Nevertheless, I am grateful for the whole topic – it did after all allow me to write the most popular post so far: Node.js and Enterprise – Why Not?

monty-1920-1200-wallpaper

What does the future hold?

Obviously Node.js, messaging and micro-services will continue to dominate our short-term horizon as we are wrestling with them daily. I spoke about them at the recent DevCon5 in NYC and intend to do the same at the upcoming nodeconf.eu in September.

Beyond that, I can see some possible future topics (although I can’t promise anything – it is enough to keep up as it is).

  • Reactive programming – I have recently presented at the first Toronto Reactive meetup, and noticed this whole area of Scala and Akka that is a completely viable alternative to implement micro-services and scalable distributed systems that confirm to the tenets of Reactive Manifesto. I would like to probe further.
  • Go language – not only because TJ decided to go that route, having an alternative to Node.js while implementing individual micro-services is a great thing, particularly for API and back-end services (I still prefer Node.js for Web serving apps).
  • Libchan – Docker’s new project (like Go channels over the network) currently requires Go (duh) but I am sure Node.js version will follow.
  • Famo.us – I know, I know, I have expressed my concerns about their approach, but I did the same with Node.js and look at me now.
  • Swift – I am a registered XCode developer and have the Swift-enabled update to it. If only I could find some time to actually create some native iOS apps. Maybe I will like Swift more than I do Objective-C.

I would like to read this post in a year and see if any of these bullets panned out (or were instead replaced with a completely different list of even newer and cooler things). In this industry, I would not be surprised.

Whatever I am writing about, I would like to thank you for your support and attention so far and hope to keep holding it just a little bit longer. Now if you excuse me, I need to post this – I am already late this week!

© Dejan Glozic, 2014

Node.js Apps and Periodic Tasks

397px-Kitchen_alarm_clock

When working on a distributed system of any size, sooner or later you will hit a problem and proclaim ‘well, this is a first’. My second proclamation in such situations is ‘this is a nice topic for the blog’. Truth to form, I do it again, this time with the issue of running periodic tasks, and the twist that clustering and high availability efforts add to the mix.

First, to frame the problem: a primary pattern you will surely encounter in a Web application is Request/Response. It is a road well traveled. Any ‘Hello, World’ web app is waving you a hello in a response to your request.

Now add clustering to the mix. You want to ensure that no matter what is happening to the underlying hardware, or how many people hunger for your ‘hello’, you will be able to deliver. You add more instances of your app, and they ‘divide and conquer’ the incoming requests. No cry for a warm reply is left unanswered.

Then you decide that you want to tell a more complex message to the world because that’s the kind of person you are: complex and multifaceted. You don’t want to be reduced to a boring slogan. You store a long and growing list of replies in a database. Because you are busy and have no time for standing up databases, you use one hosted by somebody else, already set up for high availability. Then each of your clustered nodes talk to the same database. You set the ‘message of the day’ marker, and every node fetches it. Thousands of people receive the same message.

Because we are writing our system in Node.js, there are several ways to do this, and I have already written about it. Of course, a real system is not an exercise in measuring HWPS (Hello World Per Second). We want to perform complex tasks, serve a multitude of pages, provide APIs and be flexible and enable parallel development by multiple teams. We use micro-services to do all this, and life is good.

I have also written about the need to use messaging in a micro-service system to bring down the inter-service chatter. When we added clustering into the mix, we discovered that we need to pay special attention to ensure task dispatching similar to what Node.js clustering or proxy load balancing is providing us. We found our solution in round-robin dispatching provided by worker queues.

Timers are something else

Then we hit timers. As long as information flow in a distributed system is driven by user events, clustering works well because dispatching policies (most often round-robin) are implemented by both the Node.js clustering and proxy load balancer. However, there is a distinct class of tasks in a distributed system that is not user-driven: periodic tasks.

Periodic tasks are tasks that are done on a timer, outside of any external stimulus. There are many reasons why you would want to do it, but most periodic tasks service databases. In a FIFO of a limited size, they delete old entries, collapse duplicates, extract data for analysis, report them to other services etc.

For periodic tasks, there are two key problems to solve:

  1. Something needs to count the time and initiate triggers
  2. Tasks need to be written to execute when initiated by these triggers

The simplest way to trigger the tasks is known by every Unix admin – cron. You set up a somewhat quirky cron table, and tasks are executed according to the schedule.

The actual job to execute needs to be provided as a command line task, which means your app that normally accesses the database needs to provide additional CLI entry point sharing most of the code. This is important in order to keep with the factor XII from the 12-factors, which insists one-off tasks need to share the same code and config as the long running processes.

 

There are two problems with cron in the context of the cloud:

  1. If the machine running cron jobs malfunctions, all the periodic tasks will stop
  2. If you are running your system on a PaaS, you don’t have access to the OS in order to set up cron

The first problem is not a huge issue since these jobs run only periodically and normally provide online status when done – it is relatively easy for an admin to notice when they stop. For high availability and failover, Google has experimented with a tool called rcron for setting up cron over a cluster of machines.

Node cron

The second problem is more serious – in a PaaS, you will need to rely on a solution that involves your apps. This means we will need to set up a small app just to run an alternative to cron that is PaaS friendly. As usual, there are several options, but node-cron library seems fairly popular and has graduated past the version 1.0. If you run it in an app backed by supervisor or PM2, it will keep running and executing tasks.

You can execute tasks in the same app where node-cron is running, providing these tasks have enough async calls themselves to allow the event queue to execute other callbacks in the app. However, if the tasks are CPU intensive, this will block the event queue and should be extracted out.

A good way of solving this problem would be to hook up the app running node-cron to the message broker such as RabbitMQ (which we already use for other MQ needs in our micro-service system anyway). The only thing node-cron app will do is publish task requests to the predefined topics. The workers listening to these topics should do the actual work:

node-cron

The problem with this approach is that a new task request can arrive while a worker has not finished running the previous task. Care should be taken to avoid workers stepping over each other.

Interestingly enough, a hint at this approach can be found in aforementioned 12-factors, in the section on concurrency. You will notice a ‘clock’ app in the picture, indicating an app whose job is to ‘poke’ other apps at periodic intervals.

There can be only one

A ‘headless’ version of this approach can be achieved by running multiple apps in a cluster and letting them individually keep track of periodic tasks by calling ‘setTimeout’. Since these apps share nothing, they will run according to the local server clock that may nor may not be in sync with other servers. All the apps may attempt to execute the same task (since they are clones of each other). In order to prevent duplication, each app should attempt to write a ‘lock’ record in the database before starting. To avoid deadlock, apps should wait random amount of time before retrying.

Obviously, if the lock is already there, apps should fail to create their own. Therefore, only one app will win in securing the lock before executing the task. However, the lock should be set to expire in a small multiple of times required to normally finish the task in order to avoid orphaned locks due to crashed workers. If the worker has not crashed but is just taking longer than usual, it should renew the lock to prevent it from expiring.

The advantage of this approach is that we will only schedule the next task once the current one has finished, avoiding the problem that the worker queue approach has.

Note that in this approach, we are not achieving scalability, just high availability. Of the several running apps, at least one app will succeed in securing the lock and executing the task. The presence of other apps ensures execution but does not increase scalability.

I have conveniently omitted many details about writing and removing the lock, retries etc.

Phew…

I guarantee you that once you start dealing with periodic tasks, you will be surprised with the complexity of executing them in the cloud. A mix of cloud, clustering and high availability makes running periodic tasks a fairly non-trivial problem. Limitations of PaaS environments compound this complexity.

If you visit TJ’s tweet above, you will find dozen of people offering alternatives in the replies (most of them being variations of *ron). The plethora of different solutions will be a dead giveaway that this is a thorny problem. It is not fully solved today (at least not in the context of the cloud and micro-service systems), hence so many alternatives. If you use something that works well for you, do share in the ‘Reply’ section.

© Dejan Glozic, 2014

Socket.io: Mind the Gap

Wikimedia Commons, 'Mind the Gap', 2008, Clicsouris
Wikimedia Commons, ‘Mind the Gap’, 2008, Clicsouris

Welcome to our regular edition of ‘Socket.io version 1.0 watch’ or ‘Making sure Guillermo Rauch is busy working on Socket.io 1.0 instead of whatever he does to pay the rent that does nothing for me’. I am happy to inform you that Socket.io 1.0 is now available, with the new logo and everything. Nice job!

With that piece of good news, back to our regular programming. First, a flashback. When I was working on my doctoral studies in London, England, one of the most memorable trivia was a dramatic voice on the London Underground PA system warning me to ‘Mind the Gap’. Since then I seldomly purchase my clothes in The Gap, choosing its more upmarket sibling Banana Republic. JCrew is fine too. A few years ago a friend went to London to study and she emailed me that passengers are still reminded about the dangers of The Gap.

We have recently experienced a curious problem in our usage of WebSockets – our own gap to mind, as it were. It involves a topology I have already written about. You will most likely hit it too, so here it goes:

  1. A back-end system uses message queue to pass messages about state changes that affect the UI
  2. A micro-service serves a Web page containing Socket.io client that turns around and establishes a connection with the server once page has been loaded
  3. In the time gap between the page has been served and the client calls back to establish a WebSockets connection, new messages arrive that are related to the content on the page.
  4. By the time the WebSockets connection has been established, any number of messages will have been missed – a message gap of sorts.
mind-the-gap
WebSockets gap: in the period of time from HTTP GET response to the establishment of the WebSockets connection, msg1 and msg2 were missed. The client will receive messages starting from the msg3.

The gap may or may not be a problem for you depending on how you are using the message broker to pass messages around micro-services. As I have already written in the post about REST/MQTT mirroring, we are using MQTT to augment the REST API. This augmentation mirrors the CRUD verbs that result in state change (CUD). The devil is in the details here, and the approach taken will decide whether the ‘message gap’ is going to affect you or not.

When deciding what to publish to the subscribers using MQ, we can take two approaches:

  1. Assume subscribers have made a REST call to establish the baseline state, and only send deltas. The subscribers will work well as long as the took the baseline and didn’t miss any of the deltas for whatever reason. This is similar to showing a movie on a cable channel in a particular time slot – if you miss it, you miss it.
  2. Don’t assume subscribers have the baseline state. Instead, assume they may have been down or not connected. Send a complete state of the resource alongside the message envelope. This approach is similar to breaking news being repeated many times during the day on a news channel. If you are just joining, you will be up to date soon.

The advantages of the first approach are the message payloads. There is no telling how big JSON resources can be (a problem recently addressed by Tim Bray in his fat JSON blog post). Imagine we are tracking a build resource and it is sending us updates on the progress (20%, 50%, 70%). Do we really want to receive the entire Build resource JSON alongside this message?

On the other hand, the second approach is not inconsistent with the recommendation for PUT and PATCH REST responses. We know that the newly created resource is returned in the response body for POST requests (alongside Location header). However, it is considered a good practice to do the same in the requests for PUT and PATCH. If somebody moves the progress bar of a build by using PATCH to update the ‘progress’ property, the entire build resource will thus be returned in the response body. The service fielding this request can just take that JSON string and also attach it to the message under the ‘state’ property, as we are already doing for POST requests.

Right now we didn’t make up our minds. Sending around entire resources in each message strikes us as wasteful. This message will be copied into each queue of the subscribers listening to it, and if it is durable, will also be persisted. That’a a lot of bites to move around while using a protocol whose main selling point is that it is light on the resources. Imagine pushing these messages to a native mobile client over the air. Casually attaching entire JSON resources to messages is not something you want to do in these situations.

In the end, we solved the problem without changing our ‘baseline + deltas’ approach. We tapped into the fact that messages have unique identifiers attached to them as part of the envelope. Each service that is handling clients via WebSockets has a little buffer of messages that are published by the message broker. When we send the page the client, we also send the ID of the last known message embedded in HTML as data. When WebSockets connection is established, the client will communicate (emit) this message ID to the server, and the server will check the buffer if new messages have arrived since then. If so, it will send those messages immediately, allowing the client to catch up – to ‘bridge the gap’. After it has been caught up, the message traffic resumes as usual.

As a bonus, this approach works for cases where the client drops the WebSockets connection. When connection is re-established, it can use the same approach to catch up on the messages it has missed.

The fix: the service sends the ‘message marker’ (last message id). Client echoes the marker when connecting with WebSockets. Detecting the hole in message sequence, the service immediately sends the missing messages allowing the client to catch up.

As you can see, we are still learning and evolving our REST/MQTT mirroring technique, and we will most likely encounter more face-palm moments like this. The solution is not perfect – in an extreme edge case, the WebSockets connection can take so long that the service message buffer fills up and old messages start dropping off. A solution in those cases is to refresh the browser.

We are also still intrigued with sending the state in all messages – there is something reassuring about it, and the fact that the similarity to PATCH/PUT behavior only reinforces the mirroring aspect is great. Perhaps our resources are not that large, and we are needlessly fretting over the message sizes. On the other hand, when making a REST call, callers can use ‘fields’ and ’embed’ to control the size of the response. Since we don’t know what any potential subscriber will need, we have no choice but to send the entire resource. We need to study that approach more.

That’s it from me this week. Live long, prosper and mind the gap.

© Dejan Glozic, 2014

Micro-service APIs With Some Swag (part 2)

London Cries: A Man Swaggering, Paul Sandby, 1730
London Cries: A Man Swaggering, Paul Sandby, 1730

Read part 1 of the article.

Last week I delved into the problem of presenting a unified API doc from a distributed system composed of micro-services. In the second installment, we will get our hands dirty with help of Swagger by Wordnik.

A quick recap: the problem we are trying to solve is how to document all the APIs of the system when these APIs are an aggregation of endpoints contributed by individual micro-services. To illustrate the possible solution, we will imagine a distributed system consisting of two micro-services:

  • Projects – this service provides APIs to query update and delete projects (very simplified, of course).
  • Users – this service provides APIs to query and create user profiles. Projects API will make a reference to users when listing project members.

For simplicity, we will choose only a handful of properties and methods. Project model is defined in Swagger like this:

"Project": {
   "id": "Project",
   "description": "A single project model",
   "properties": {
      "name": {
      "type": "string",
      "description": "name of the project",
      "required": true
   },
   "description": {
      "type": "string",
      "description": "description of the project"
   },
   "avatar": {
      "type": "string",
      "description": "URL to the image representing the project avatar"
   },
   "owner": {
      "type": "string",
      "description": "unique id of the projects' owner",
      "required": true
   },
   "members": {
      "type": "array",
      "description": "array of unique ids of project members",
      "items": {
         "type": "string"
      }
   }
}

Users are provided by another service and have the following model, defined the same way:

"User": {
   "id": "User",
   "description": "A single user model",
   "properties": {
      "id": {
         "type": "string",
         "description": "unique user id",
         "required": true
      },
      "name": {
      "type": "string",
         "description": "user name",
         "required": true
      },
      "email": {
         "type": "string",
         "description": "user email",
         "required": true
      },
      "picture": {
         "type": "string",
         "description": "thumbnail picture of the user"
      }
   }
}

The key to the proposed solution lies in Swagger’s feature that allows the composite API document to be composed of APIs coming from multiple places. The entry point to the document definition will look like this:

{
    "apiVersion":"1.0",
    "swaggerVersion":"1.2",
    "apis":[
        {
            "path": "http://localhost:3000/api-docs/projects.json",
            "description":"Projects"
        },
        {
            "path": "http://localhost:3001/api-docs/users.json",
            "description":"Users"
        }
    ],
    "info":{
        "title":"30% Turtleneck, 70% Hoodie API Example",
        "description":"This API doc demonstrates how API definitions can be aggregated in a micro-service system, as demonstrated at <a href=\"http://dejanglozic.com\">http://dejanglozic.com</a>."
    }
}

Each API group with its set of endpoints can be sources from a different URL, allowing us the flexible solution to provide this resource by the actual micro-service that owns that API portion.

Each individual API document will list the endpoints and resources it handles. For each endpoint and verb combination, it will list parameters, request/response bodies, as well as data models and error responses. This and much more is fully documented in Swagger 1.2 specification.

{
  "path": "/users/{id}",
  "operations": [
    {
      "method": "GET",
      "summary": "Returns a user profile",
      "notes": "Implementation notes on GET.",
      "type": "User",
      "nickname": "getUser",
      "authorizations": {},
      "parameters": [
        {
          "name": "id",
          "description": "Unique user identifier",
          "paramType": "path",
          "type": "string"
        }
      ],
      "responseMessages": [
        {
          "code": 404,
          "message": "User with a provided id not found"
        }
      ]
    }
  ]
}

Swagger handles most of its specification through the parameter list, which is fairly clever. In addition to query parameters, they can be used to define path segments, as well as request body for POST, PUT and PATCH. In addition, request and response body schemas are linked to the model specifications further down the document.

Needless to say, I skipped over huge swaths of the specification dealing with authentication. Suffice to say that Swagger specification currently supports basic Auth, API key and OAuth 2 as authentication options.

At this point of the article, I realized that I cannot show you the actual JSON files without making the article long and unwieldy. Luckily, I also realized I actually work for IBM and more importantly, IBM DevOps Services (JazzHub). So here is what I have done:

The entire Node.js app is now available as a public project on DevOps Services site:

ids-swagger

Once you explored the code, you can see it running in IBM Bluemix. You can drill into the Swagger API UI (despite what the UI is telling you, you cannot yet ‘Try it’ – the actual API is missing, this is just a doc; for a real API, this part will also work).

bm-swagger

I hope you agree that showing a running app is better than a screenshot. From now on, I will make it my standard practice to host complete demo apps in DevOps Services and run them in Bluemix for your clicking pleasure.

© Dejan Glozic, 2014