Micro-service APIs With Some Swag (part 1)

London Cries: A Man Swaggering, Paul Sandby, 1730
London Cries: A Man Swaggering, Paul Sandby, 1730

Every aspect of the API matters to some Client.

Jim des Rivieres, Evolving Eclipse APIs

It is fascinating that the quote above is 14 years old now. It was coined by the Benevolent Dictator of Eclipse APIs Jim des Rivieres in the days when we defined how Eclipse Platform APIs were to be designed and evolved. Of course, APIs in question were Java, not the REST variety that is ruling the API economy these days. Nevertheless, the key principles hardly changed.

Last week when I wrote about the switch to micro-services by SoundCloud, I noted that APIs are predominantly a public-facing concern in monolithic applications. There is no arms-length relationship between providers and consumers of functional units, enabling a low-ceremony evolution of the internal interfaces. They are the ‘authorized personal only’ rooms in a fancy restaurant – as long as the dining room is spotless, we will ignore the fact that the gourmet meals are prepared by a cute rat that sounds a lot like Patton Oswald.

Put another way, APIs are not necessary in order to get the monolithic application up and running. They are important the moment you decide to share your data with third-party developers, write a mobile app, or enable partner integrations. Therefore, monolithic applications predominantly deal with public API.

Things are much different for a micro-service based distributed system. Before any thought is put in how the general public will interact with the system, micro-services need to figure out how they will interact with each other.

In the blog post about Node.js clustering, I pointed out that Node is inherently single-threaded, and clustering is required just to stretch to all the cores of a single server, never mind load balancing across multiple VMs. This ‘feature’ essentially makes clustering an early consideration, and switching from vertical to horizontal scaling (across multiple machines) mostly a configuration issue. Presumably your instances have already been written to share-nothing and do not really care where they are physically running.

Micro-service APIs are very similar in that regard. They force developers to start with a clean API for each service, and since a complex system is often built with several teams working in parallel, it would turn into a total chaos without clean contracts between the services. In micro-service systems, APIs are foundational.

Internal APIs – an oxymoron?

In the previous post, I put forward a few rules of writing a micro-service based distributed system that concern APIs. Here they are again:

  • Rule 3: APIs should be the only way micro-services talk to each other and the outside world.
  • Rule 4: Internal APIs should be documented and otherwise written as if they will be exposed to the open Internet at any point.
  • Rule 5: Public APIs are a subset of internal APIs with stricter visibility rules, rate limiting and separate authentication.

The aforementioned Jim des Rivieres used to say that “there is no such a thing as internal API”. Interfaces are either firm contracts exhibiting all the qualities of APIs or they can change at any time without warning. There is no mushy middle ground. I tend to agree with him when it comes to monolithic systems, where ‘internal’ refers to ‘written for systems’ internal use only’. However, in distributed systems ‘internal’ refers to traffic between services, or between systems behind the firewall. It is more to do with ‘things we say to the members of our own family’, presumably versus ‘things we say to the outside world’.

In this context, ‘internal APIs’ is a legitimate thing because ‘internal’ refers to the visibility rules, not the quality of the API contract. Rule #4 above explicitly states that – there is nothing different about internal APIs except visibility.

Presenting unified API front

If APIs are the only way micro-services should communicate with each other and the outside world, the consumers need to be presented with a cleanly documented contract. Documenting the APIs cannot be an afterthought – it needs to be built with the micro-service, sometimes even before the documented endpoints actually work.

The fact that our distributed system is composed of micro-services is a great feature for us and our ability to quickly evolve and deploy the system with little of no downtime. However, API consumers can’t care less about it – they want one place to go to see all the APIs.

There are multiple ways of skinning that particular cat, but we have decided to do as follows:

  1. Proxy all the APIs to the common path (e.g. https://example.com/api)
  2. Expose the API version in the URL (I know, I know, we can yell at each other until the cows come home about how that is great or stupid, but many popular APIs are doing it and so are we). Thus the common path gets a version (e.g. https://example.som/api/v1)
  3. Reserve a segment after the version for each micro-service that exposes APIs (e.g. /projects, /users etc.).
  4. Provide API specification using a popular Open Source API doc solution

On the last point, we looked around and considered several alternatives, finally settling on Swagger by Wordnik. It is a popular solution, with a vibrant community, fairly well defined API spec, a reusable live API UI that can be included in our UI, and with a path forward towards version 2.0 that promises to address currently missing features (the current version is 1.2).

A micro-service based system using Swagger to define APIs could look like this:


Each micro-service that provides APIs will make a Swagger API doc resource available, describing all the endpoints, verbs, parameters and request/response bodies.  Documentation micro-service can render these in two ways – using Swagger Live UI and rendering static docs.

Swagger Live UI is available as an Open Source project and allows users to not only read the rendered documentation, but enter values and try it out in place. To see it in action, try out the Pat Store sample.

The UI is all client side, which makes it stack-agnostic and fit for being served by a multitude of platforms, but if you are aggregating your definitions like we do, you need go around browser’s Single-Origin limitation. You can either proxy the API definitions or use CORS. In our case, it helps that we proxy all the services to the single external URL root, which is on the same domain as the doc UI – problem solved.

I can stop now while I am ahead – this being the part 1 of a multi-part article. In the next installment, I will walk you through an example of two micro-services – one providing API for Projects, another for Users. We will spec out the API, document the spec using Swagger, write a Node.js app to serve the UI from these definitions, and also render an alternative static version of the API doc.

See you next week, off to write some API micro-services.

© Dejan Glozic, 2014


SoundCloud is Reading My Mind

Marvelous feats in mind reading, The U.S. Printing Co., Russell-Morgan Print, Cincinnati & New York, 1900
Marvelous feats in mind reading, The U.S. Printing Co., Russell-Morgan Print, 1900

“Bad artists copy. Good artists steal.”

– Pablo Picasso

It was bound to happen. In the ultra-connected world, things are bound to feed off of each other, eventually erasing differences, equalizing any differential in electric potentials between any two points. No wonder the weirdest animals can be found on islands (I am looking at you, Australia). On the internet, there are no islands, just a constant primordial soup bubbling with ideas.

The refactoring of monolithic applications into distributed systems based on micro-services is slowly becoming ‘a tale as old as time’. They all follow a certain path which kind of makes sense when you think about it. We are all impatient, reading the first few Google search and Stack Overflow results ‘above the fold’, and it is no coincidence that the results start resembling majority rule, with more popular choices edging out further and further ahead with every new case of reuse.

Luke Wroblewski of Mobile First fame once said that ‘two apps do the same thing and suddenly it’s a pattern’. I tend to believe that people researching the jump into micro-services read more than two search results, but once you see certain choices appearing in, say, three or four stories ‘from the trenches’, you become reasonably convinced to at least try them yourself.

If you were so kind as to read my past blog posts, you know some of they key points of my journey:

  1. Break down a large monolithic application (Java or RoR) into a number of small and nimble micro-services
  2. Use REST API as the only way these micro-services talk to each other
  3. Use message broker (namely, RabbitMQ) to apply event collaboration pattern and avoid annoying inter-service polling for state changes
  4. Link MQ events and REST into what I call REST/MQTT mirroring to notify about resource changes

Then this came along:

As I was reading the blog post, it got me giddy at the realization we are all converging on the emerging model for universal micro-service architecture. Solving their own unique SoundCloud problems (good problems to have, if I may say – coping with millions of users falls into such a category), SoundCloud developers came to very similar realizations as many of us taking a similar journey. I will let you read the post for yourself, and then try to extract some common points.

Stop the monolith growth

Large monolithic systems cannot be refactored at once. This simple realization about technical debt actually has two sub-aspects: the size of the system at the moment it is considered for a rewrite, and the new debt being added because ‘we need these new features yesterday’. As with real world (financial) debt, the first order of business is to ‘stop the bleeding’ – you want to stop new debt from accruing before attempting to make it smaller.

At the beginning of this journey you need to ‘draw the line’ and stop adding new features to the monolith. This rule is simple:

Rule 1: Every new feature added to the system will from now on be written as a micro-service.

This ensures that precious resources of the team are not spent on making the monolith bigger and the finish line farther and farther on the horizon.

Of course, a lot of the team’s activity involves reworking the existing features based on validated learning. Hence, a new rule is needed to limit this drain on resources to critical fixes only:

Rule 2: Every existing feature that requires significant rework will be removed and rewritten as a micro-service.

This rule is somewhat less clear-cut because it leaves some room for the interpretation of ‘significant rework’. In practice, it is fairly easy to convince yourself to rewrite it this way because micro-service stacks tend to be more fun, require fewer files, fewer lines of code and are more suitable for Web apps today. For example, we don’t need too much persuasion to rewrite a servlet/JSP service in the old application as a Node.js/Dust.js micro-service whenever we can. If anything, we need to practice restraint and not fabricate excuse to rewrite features that only need touch-ups.

Micro-services as BBQ. Mmmmm, BBQ…

An important corollary of this rule is to have a plan of action ahead of time. Before doing any work, have a ‘cut of beef’ map of the monolith with areas that naturally lend themselves to be rewritten as micro-services. When the time comes for a significant rework in one of them, you can just act along that map.

As is the norm these days, ‘there’s a pattern for that’, and as SoundCloud guys noticed, the cuts are along what is known as bounded context.

Center around APIs

As you can read at length on the API evangelist’s blog, we are transforming into an API economy, and APIs are becoming a central part of your system, rather than something you tack on after the fact. If you could get by with internal monolith services in the early days, micro-services will force you to accept APIs as the only way you communicate both inside your system and with the outside world. As SoundCloud developers realized, the days of integration around databases are over – APIs are the only contact points that tie the system together.

Rule 3: APIs should be the only way micro-services talk to each other and the outside world.

With monolithic systems, APIs are normally not used internally, so the first APIs to be created are outward facing – for third party developers and partners. A micro-service based system normally starts with inter-service APIs. These APIs are normally more powerful since they assume a level of trust that comes from sitting behind a  firewall. They can use proprietary authentication protocols, have no rate limiting and expose the entire functionality of the system. An important rule is that they should in no way be second-class compared to what you would expose to the external users:

Rule 4: Internal APIs should be documented and otherwise written as if they will be exposed to the open Internet at any point.

Once you have the internal APIs designed this way, deciding which subset to expose as public API stops becoming a technical decision. Your external APIs look like internal with the exception of stricter visibility rules (who can see what), rate limiting (with the possibility of a rate-unlimited paid tier), and authentication mechanism that may differ from what is used internally.

Rule 5: Public APIs are a subset of internal APIs with stricter visibility rules, rate limiting and separate authentication.

SoundClound developers went the other way (public API first) and realized that they cannot build their entire system with the limitations in place for the public APIs, and had to resort to more powerful internal APIs. The delicate balance between making public APIs useful without giving out the farm is a decision every business need to make in the API economy. Micro-services simply encourage you to start from internal and work towards public.


If there was a section in SoundCloud blog post that made me jump with joy was a section where they discussed how they arrived at using RabbitMQ for messaging between micro-services, considering how I write about that in every second post for the last three months. In their own words:

Soon enough, we realized that there was a big problem with this model; as our microservices needed to react to user activity. The push-notifications system, for example, needed to know whenever a track had received a new comment so that it could inform the artist about it. At our scale, polling was not an option. We needed to create a better model.


We were already using AMQP in general and RabbitMQ in specific — In a Rails application you often need a way to dispatch slow jobs to a worker process to avoid hogging the concurrency-weak Ruby interpreter. Sebastian Ohm and Tomás Senart presented the details of how we use AMQP, but over several iterations we developed a model called Semantic Events, where changes in the domain objects result in a message being dispatched to a broker and consumed by whichever microservice finds the message interesting.

I don’t need to say much about this – read my REST/MQTT mirroring post that describes the details of what SoundCloud guys call ‘changes in the domain objects result in a message’. I would like to indulge in a feeling that ‘great minds think alike’, but more modestly (and realistically), it is just common sense and RabbitMQ is a nice, fully featured and reliable open source polyglot broker. No shocking coincidence – it is seen in many installations of this kind. Let’s make a rule about it:

Rule 6: Use a message broker to stay in sync with changes in domain models managed by micro-services and avoid polling.

All together now

Let’s pull all the rules together. As we speak, teams around the world are suffering under the weight of large unwieldy monolithic applications that are ill-fit for the cloud deployment. They are intrigued by micro-services but afraid to take the plunge. These rules will make the process more manageable and allow you to arrive at a better system that is easier to grow, deploy many times a day, and more reactive to events, load, failure and users:

  1. Every new feature added to the system will from now on be written as a micro-service.
  2. Every existing feature that requires significant rework will be removed and rewritten as a micro-service.
  3. APIs should be the only way micro-services talk to each other and the outside world.
  4. Internal APIs should be documented and otherwise written as if they will be exposed to the open Internet at any point.
  5. Public APIs are a subset of internal APIs with stricter visibility rules, rate limiting and separate authentication.
  6. Use a message broker to stay in sync with changes in domain models managed by micro-services and avoid polling.

This is a great time to build micro-service based systems, and collective wisdom on the best practices is converging as more systems are coming online. I will address the topic of APIs in more detail in one of the future posts. Stay tuned, and keep reading my mind!

© Dejan Glozic, 2014

For Once, Being Reactive is Good

5 Gum - React
5 Gum – React

Apple said Monday that it sold more than 300,000 iPads on the first day of its launch, ushering a new era of people buying things in order to find out what they are.


SNL Weekend Update, season 35, episode 18

All my life, I thought ‘reaction’ was a bad word. Ever since the French Revolution, being ‘reactionary’ could get you into a lot of trouble. More recently (and less detrimental to your health and limb count), being in ‘reactionary’ mode is considered merely an anti-pattern. We were all in situations in life where we felt like we were merely reacting to changes foisted upon us, like tall grass helplessly flailing on a windy day. We all want to be the wind, not the grass.

As you could read only everywhere, including my own blog post, Agile movement has been declared dead (although ‘agility’ is still fine and dandy, thank you). Being communal people and in need of an idea to gather around, and not liking the traditional organized religions’ early hours, we looked for a more suitable replacement.

Not that others were not trying, and even before Agile’s passing. For example, Adam Wiggins of Heroku extraction has channeled his inner L. Ron Hubbard by establishing his Church of 12 Factors (kudos for pulling it off without one reference to space ships). It it is chock-full of Cloud-y goodness and is actually quote good and useful. I think Adam is now beating himself up for not waiting a bit and slapping ‘micro-services’ all over the manifest, because that is totally what ’12 factors’ are about.

According to Adam, 12-factor apps:

  • Use declarative formats for setup automation, to minimize time and cost for new developers joining the project;
  • Have a clean contract with the underlying operating system, offering maximum portability between execution environments;
  • Are suitable for deployment on modern cloud platforms, obviating the need for servers and systems administration;
  • Minimize divergence between development and production, enabling continuous deployment for maximum agility;
  • And can scale up without significant changes to tooling, architecture, or development practices.

So what went wrong? It is still a worthwhile document, and I keep revisiting it often, but it lacks the galvanizing force that true movements have. Maybe there are just too many factors (even actual religions knew to stop at 10), or maybe because it sounds too much like inspirational lifestyle articles such as 12 Lifestyle Factors That Make You Feel Depressed.

Then the Agile thing happened, and it was time to get cracking. It was not a long wait – behold The Reactive Manifesto.

Now, I make it sound like it all happened in a neat chronological order (something my buddy Adrian Rossouw would organize in a Wayfinder timeline), but it did not. The first version of the manifesto was published by Jonas Boner and friends and described on the Typesafe blog in July 2013. It was uploaded to GitHub and the community was invited to help tweak the document. The current version (1.1) dates September 2013 and is signed by thousands of believers (I meant ‘supporters’). In Jonas’ own words, the motivation for putting the manifesto forward was:

The primary motivation for this manifesto is to come up with a name for these new type of applications (similar to NOSQL, Big Data, SOA and REST) and to define a common vocabulary for describing them — both in terms of business values and technical concepts. Names matter and hopefully this will help bring the communities together and make it easier for users and vendors to talk about things and understand each other.


Up to now the usual way to describe this type of application has been to use a mix of technical and business buzzwords; asynchronous, non-blocking, real-time, highly-available, loosely coupled, scalable, fault-tolerant, concurrent, reactive, event-driven, push instead of pull, distributed, low latency, high throughput, etc. This does not help communication, quite the contrary, it greatly hinders it.

The four traits

In its core, The Reactive Manifesto puts forward four reactive traits a modern distributed system should possess (notice the small number of key tenets – that’s how it’s done). Let’s see how these qualities intersect with the modern micro-service based systems I was writing about in the last few months:

  1. Reactive to events – a modern micro-service based distributed system is asynchronous by nature, with each service sitting dormant until an event wakes it up. Lest it turns out we are talking REST only, a loosely coupled system using some kind of message broker is a better fit because it offers further decoupling. Publishing into a pub/sub topic does not require any knowledge of the possible consumers of the message in a way that A->B REST calls do. And of course, while the authors of the manifest seem to be coming from the Scala background (with Play framework also playing a part), it is easy to notice that Node.js is an even better fit. Its asynchronous nature ‘all the way down’ ensures non-blocking way of reacting to events.
  2. Reactive to load – a corollary of the micro-service based system is the freedom to scale out each service independently of the rest of the system. The ability to instrument the nodes and cluster hot spots while living the less popular services as-is is of great help in the cloud environments. Cloud resources are finite and cost money. Knowing which nodes to cluster (and more importantly, where that would be overkill) is essential to arriving at a system that is reactive to load while still keeping the monthly bill reasonable.
  3. Reactive to failure – when there are many moving parts, failure is inevitable. Successful ‘born on the net’ companies with complex distributed systems not only guard against failure, they openly embrace it (who hasn’t heard about the Netflix’s Chaos Monkey).
  4. Reactive to users – this part is a bit confusing. You would think that the four reactive traits are like ‘four pillars of heaven’ or ‘the four elements’ (minus Mila Jovovic). As it turns out, the previous three reactive properties are preconditions to the system being reactive to users – by providing real time, engaging, performant user experiences. Being reactive to events, load and failure will simply increase your chances to be reactive to users in a way that will keep them from leaving in frustration.

Reactive reactions

When you research a topic, and the second Google search hit after the actual topic is “Reactive Manifesto bulls**t”, you cannot resist clicking on the link. In it, Paul Chiusano argues that Reactive Manifest is not only not right, it’s not even wrong. It looks like the ultimate insult is being banished into the binary system limbo, where you are neither right nor wrong, you are, well, nothing (it’s like Louis CK joke that because he owes $50, he needs to raise $50 just to be broke).

Of course, there are positive reactions and people who don’t really care – the usual spectrum.

Here is what I think: I am actually positive about the intent of Reactive Manifesto. First off, it redefines ‘reactive’ as something positive. Of course it didn’t invent anything new, but that is not the first time in history somebody came and put a name on something we were already doing but didn’t know it. Remember JJG and Ajax? He didn’t invent it – he just put a name on a technique that is the bedrock of any modern client side application. How about ‘micro-services’? Many people fail to see how they are different from SOA or just plain ‘services’ or ‘distributed systems’ – a lot of haters in that camp too.

But here is what my first reaction was: coming from a former communist country, my first association was with a guy whose beard is every hipster’s dream – Karl Marx. When he and his buddy Engels came forward with the Communist Manifesto, they didn’t invent alienation, oppression and capitalism’s seedy underbelly. They just articulated them succinctly and gave something to millions of disgruntled workers around the world to rally around. It didn’t turn out all that well in hindsight, but notice what I am getting at here – manifestos don’t invent new things, they put a names on concepts, helping the adherents galvanize around them and form movements.

So to those who say ‘The Reactive Manifesto’ didn’t invent anything new, you are right: that was not the intention. Go read your history or  Google ‘manifesto’.

The ‘What, Why, How’ trifecta

I would like to circle back to ’12 factors’ and ‘micro-services’ and claim that we now have a fairly complete set of answers to big questions we may ask while we build modern distributed systems:

  1. What are we building? A micro-service based distributed system.
  2. Why are we building it? Because we want it to be reactive to events, load, failure and users.
  3. How are we building it? Using the techniques and recommendations outlined in the 12-factors.

There you go – no need to choose one over the other – a simple ‘1 + 4 + 12’ formula that will bring you happiness and make you rich.

As for me, I am taking a leaf from the SNL book – I am going to the first Toronto Reactive Meetup. In fact, I am raising SNL by not only participating but actually speaking at it – a new era of people presenting on topics to find out what they are.

If you are in Toronto on June 24, join us – in addition to reading my musings, you can see me deliver them in full HD.

© Dejan Glozic, 2014