Micro-service APIs With Some Swag (part 2)

London Cries: A Man Swaggering, Paul Sandby, 1730
London Cries: A Man Swaggering, Paul Sandby, 1730

Read part 1 of the article.

Last week I delved into the problem of presenting a unified API doc from a distributed system composed of micro-services. In the second installment, we will get our hands dirty with help of Swagger by Wordnik.

A quick recap: the problem we are trying to solve is how to document all the APIs of the system when these APIs are an aggregation of endpoints contributed by individual micro-services. To illustrate the possible solution, we will imagine a distributed system consisting of two micro-services:

  • Projects – this service provides APIs to query update and delete projects (very simplified, of course).
  • Users – this service provides APIs to query and create user profiles. Projects API will make a reference to users when listing project members.

For simplicity, we will choose only a handful of properties and methods. Project model is defined in Swagger like this:

"Project": {
   "id": "Project",
   "description": "A single project model",
   "properties": {
      "name": {
      "type": "string",
      "description": "name of the project",
      "required": true
   },
   "description": {
      "type": "string",
      "description": "description of the project"
   },
   "avatar": {
      "type": "string",
      "description": "URL to the image representing the project avatar"
   },
   "owner": {
      "type": "string",
      "description": "unique id of the projects' owner",
      "required": true
   },
   "members": {
      "type": "array",
      "description": "array of unique ids of project members",
      "items": {
         "type": "string"
      }
   }
}

Users are provided by another service and have the following model, defined the same way:

"User": {
   "id": "User",
   "description": "A single user model",
   "properties": {
      "id": {
         "type": "string",
         "description": "unique user id",
         "required": true
      },
      "name": {
      "type": "string",
         "description": "user name",
         "required": true
      },
      "email": {
         "type": "string",
         "description": "user email",
         "required": true
      },
      "picture": {
         "type": "string",
         "description": "thumbnail picture of the user"
      }
   }
}

The key to the proposed solution lies in Swagger’s feature that allows the composite API document to be composed of APIs coming from multiple places. The entry point to the document definition will look like this:

{
    "apiVersion":"1.0",
    "swaggerVersion":"1.2",
    "apis":[
        {
            "path": "http://localhost:3000/api-docs/projects.json",
            "description":"Projects"
        },
        {
            "path": "http://localhost:3001/api-docs/users.json",
            "description":"Users"
        }
    ],
    "info":{
        "title":"30% Turtleneck, 70% Hoodie API Example",
        "description":"This API doc demonstrates how API definitions can be aggregated in a micro-service system, as demonstrated at <a href=\"http://dejanglozic.com\">http://dejanglozic.com</a>."
    }
}

Each API group with its set of endpoints can be sources from a different URL, allowing us the flexible solution to provide this resource by the actual micro-service that owns that API portion.

Each individual API document will list the endpoints and resources it handles. For each endpoint and verb combination, it will list parameters, request/response bodies, as well as data models and error responses. This and much more is fully documented in Swagger 1.2 specification.

{
  "path": "/users/{id}",
  "operations": [
    {
      "method": "GET",
      "summary": "Returns a user profile",
      "notes": "Implementation notes on GET.",
      "type": "User",
      "nickname": "getUser",
      "authorizations": {},
      "parameters": [
        {
          "name": "id",
          "description": "Unique user identifier",
          "paramType": "path",
          "type": "string"
        }
      ],
      "responseMessages": [
        {
          "code": 404,
          "message": "User with a provided id not found"
        }
      ]
    }
  ]
}

Swagger handles most of its specification through the parameter list, which is fairly clever. In addition to query parameters, they can be used to define path segments, as well as request body for POST, PUT and PATCH. In addition, request and response body schemas are linked to the model specifications further down the document.

Needless to say, I skipped over huge swaths of the specification dealing with authentication. Suffice to say that Swagger specification currently supports basic Auth, API key and OAuth 2 as authentication options.

At this point of the article, I realized that I cannot show you the actual JSON files without making the article long and unwieldy. Luckily, I also realized I actually work for IBM and more importantly, IBM DevOps Services (JazzHub). So here is what I have done:

The entire Node.js app is now available as a public project on DevOps Services site:

ids-swagger

Once you explored the code, you can see it running in IBM Bluemix. You can drill into the Swagger API UI (despite what the UI is telling you, you cannot yet ‘Try it’ – the actual API is missing, this is just a doc; for a real API, this part will also work).

bm-swagger

I hope you agree that showing a running app is better than a screenshot. From now on, I will make it my standard practice to host complete demo apps in DevOps Services and run them in Bluemix for your clicking pleasure.

© Dejan Glozic, 2014

Micro-service APIs With Some Swag (part 1)

London Cries: A Man Swaggering, Paul Sandby, 1730
London Cries: A Man Swaggering, Paul Sandby, 1730

Every aspect of the API matters to some Client.

Jim des Rivieres, Evolving Eclipse APIs

It is fascinating that the quote above is 14 years old now. It was coined by the Benevolent Dictator of Eclipse APIs Jim des Rivieres in the days when we defined how Eclipse Platform APIs were to be designed and evolved. Of course, APIs in question were Java, not the REST variety that is ruling the API economy these days. Nevertheless, the key principles hardly changed.

Last week when I wrote about the switch to micro-services by SoundCloud, I noted that APIs are predominantly a public-facing concern in monolithic applications. There is no arms-length relationship between providers and consumers of functional units, enabling a low-ceremony evolution of the internal interfaces. They are the ‘authorized personal only’ rooms in a fancy restaurant – as long as the dining room is spotless, we will ignore the fact that the gourmet meals are prepared by a cute rat that sounds a lot like Patton Oswald.

Put another way, APIs are not necessary in order to get the monolithic application up and running. They are important the moment you decide to share your data with third-party developers, write a mobile app, or enable partner integrations. Therefore, monolithic applications predominantly deal with public API.

Things are much different for a micro-service based distributed system. Before any thought is put in how the general public will interact with the system, micro-services need to figure out how they will interact with each other.

In the blog post about Node.js clustering, I pointed out that Node is inherently single-threaded, and clustering is required just to stretch to all the cores of a single server, never mind load balancing across multiple VMs. This ‘feature’ essentially makes clustering an early consideration, and switching from vertical to horizontal scaling (across multiple machines) mostly a configuration issue. Presumably your instances have already been written to share-nothing and do not really care where they are physically running.

Micro-service APIs are very similar in that regard. They force developers to start with a clean API for each service, and since a complex system is often built with several teams working in parallel, it would turn into a total chaos without clean contracts between the services. In micro-service systems, APIs are foundational.

Internal APIs – an oxymoron?

In the previous post, I put forward a few rules of writing a micro-service based distributed system that concern APIs. Here they are again:

  • Rule 3: APIs should be the only way micro-services talk to each other and the outside world.
  • Rule 4: Internal APIs should be documented and otherwise written as if they will be exposed to the open Internet at any point.
  • Rule 5: Public APIs are a subset of internal APIs with stricter visibility rules, rate limiting and separate authentication.

The aforementioned Jim des Rivieres used to say that “there is no such a thing as internal API”. Interfaces are either firm contracts exhibiting all the qualities of APIs or they can change at any time without warning. There is no mushy middle ground. I tend to agree with him when it comes to monolithic systems, where ‘internal’ refers to ‘written for systems’ internal use only’. However, in distributed systems ‘internal’ refers to traffic between services, or between systems behind the firewall. It is more to do with ‘things we say to the members of our own family’, presumably versus ‘things we say to the outside world’.

In this context, ‘internal APIs’ is a legitimate thing because ‘internal’ refers to the visibility rules, not the quality of the API contract. Rule #4 above explicitly states that – there is nothing different about internal APIs except visibility.

Presenting unified API front

If APIs are the only way micro-services should communicate with each other and the outside world, the consumers need to be presented with a cleanly documented contract. Documenting the APIs cannot be an afterthought – it needs to be built with the micro-service, sometimes even before the documented endpoints actually work.

The fact that our distributed system is composed of micro-services is a great feature for us and our ability to quickly evolve and deploy the system with little of no downtime. However, API consumers can’t care less about it – they want one place to go to see all the APIs.

There are multiple ways of skinning that particular cat, but we have decided to do as follows:

  1. Proxy all the APIs to the common path (e.g. https://example.com/api)
  2. Expose the API version in the URL (I know, I know, we can yell at each other until the cows come home about how that is great or stupid, but many popular APIs are doing it and so are we). Thus the common path gets a version (e.g. https://example.som/api/v1)
  3. Reserve a segment after the version for each micro-service that exposes APIs (e.g. /projects, /users etc.).
  4. Provide API specification using a popular Open Source API doc solution

On the last point, we looked around and considered several alternatives, finally settling on Swagger by Wordnik. It is a popular solution, with a vibrant community, fairly well defined API spec, a reusable live API UI that can be included in our UI, and with a path forward towards version 2.0 that promises to address currently missing features (the current version is 1.2).

A micro-service based system using Swagger to define APIs could look like this:

swagger

Each micro-service that provides APIs will make a Swagger API doc resource available, describing all the endpoints, verbs, parameters and request/response bodies.  Documentation micro-service can render these in two ways – using Swagger Live UI and rendering static docs.

Swagger Live UI is available as an Open Source project and allows users to not only read the rendered documentation, but enter values and try it out in place. To see it in action, try out the Pat Store sample.

The UI is all client side, which makes it stack-agnostic and fit for being served by a multitude of platforms, but if you are aggregating your definitions like we do, you need go around browser’s Single-Origin limitation. You can either proxy the API definitions or use CORS. In our case, it helps that we proxy all the services to the single external URL root, which is on the same domain as the doc UI – problem solved.

I can stop now while I am ahead – this being the part 1 of a multi-part article. In the next installment, I will walk you through an example of two micro-services – one providing API for Projects, another for Users. We will spec out the API, document the spec using Swagger, write a Node.js app to serve the UI from these definitions, and also render an alternative static version of the API doc.

See you next week, off to write some API micro-services.

© Dejan Glozic, 2014