Odds and Ends

Jack Spade Odds and Ends pouch
Jack Spade Odds and Ends pouch

This week and the next there will be no regular blog posts. I hate to break my routine, but I am going to IBM Innovate 2014, and being an IBMer myself, it is a working conference for me, requiring a lot of preparation time and paying my booth dues.

If you are coming to Innovate, don’t forget to come and hear myself and Dan Berg in our presentation on the new DevOps Pipeline we built for IBM DevOps Services (powered by JazzHub). Here are the coordinates:

ICD-1810 : DevOps Services: Automated Delivery Pipeline for Codename: BlueMix
Innovation – Cloud Development
Date/Time : Wed, 04-Jun, 09:15 AM-10:15 AM
Room : Dolphin-Australia 3
Co-presenter(s):Daniel Berg, IBM

While I am here, let’s review other events where you can see me. The next one comes in July 9-10, when I will present at DevCon5 in New York City. This is my talk:

Node.js Micro-Services: The Water is Fine, Jump In!

Well, duh – what did you expect, just check the number of micro-service posts I made – it is only fitting for me to spread the message in person as well.

Then, in September 7-11 in Ireland comes nodeconf.eu. I don’t have the title of the talk yet but you can expect more micro-service-y goodness.

So there you go – three opportunities to meet me and have a beer – what’s not to like?

See you again in about 10 days when I will resume the regular programming.

© Dejan Glozic, 2014

Should We Fight or Embrace the DOM?

Dom Tower, Utrecht, 2013, Anitha Mani (Wikimedia Commons)
Dom Tower, Utrecht, 2013, Anitha Mani (Wikimedia Commons)

Now, my story is not as interesting as it is long.

Abe Simpson

One of the privileges (or curses) of experience is that you amass a growing number of cautionary tales with which you can bore your younger audience to death. On the other hand, knowing the history of slavery came in handy for Captain Pickard to recognize that what Federation was trying to do by studying Data is create an army of Datas, not necessarily benefit the humanity. So experience can come in handy once in a while.

So let’s see if my past experience can inform a topic de jour.

AWT, Swing and SWT

The first Java windowing system (AWT) was based on whatever the underlying OS had to offer. The original decision was to ensure the same Java program runs anywhere, necessitating a ‘least common denominator’ approach. This translated to UIs that sucked equally on all platforms, not exactly something to get excited about. Nevertheless, they embraced the OS, inheriting both the shortcomings and receiving the automatic improvements.

The subsequent Swing library took a radically different approach, essentially taking on the responsibility of rendering everything. It was ‘fighting the OS’, or at least side-stepping it by creating and controlling its own reality. In the process, it also became responsible for keeping up with the OS. The Eclipse project learned that fighting the OS is trench warfare that is never really ‘won’. Using an alternative system (SWT) that accepted the windowing system of the underlying OS turned out to be a good strategic decision, both in terms of the elusive ‘look and feel’, and for riding the OS version waves as they sweep in.

The 80/20 of custom widgets

When I was working on the Eclipse project, I had my own moment of ‘sidestepping’ the OS by implementing Eclipse Forms. Since browsers were not ready yet, I wrote a rudimentary engine that gave me text reflows, hyperlinks and images. This widget was very useful when mixed with other normal OS widgets inside Eclipse UI. As you could predict, I got the basic behavior fairly quickly (the ’80’ part). Then I spent couple of years (including help from younger colleagues), doing the ‘last mile’ (the ’20’) – keyboard, accessibility, BIDI. It was never ‘finished’ and it was never quite as good as the ‘real’ browser, and not nearly as powerful.

One of the elements of that particular custom widget was managing the layout of its components. In essence the container was managing a collection of components but the layout of the components was delegated to the layout manager that could be set on the container. This is an important characteristic that will come in handy later in the article. I remember the layout class as one of the trickiest and hardest to get done right and fully debug. After it was ‘sort of’ working correctly, everybody dreaded touching it, and consequently forgot how it worked.

DOM is awesome

I gave you this Abe Simson moment of reflection to set the stage for a battle that is raging today between people who want to work with the browser’s DOM, and people who think it is the root of all evil and should be worked around. As is often these days, both points of view came across my twitter feed from different directions.

In the ’embrace the DOM’ corner, we have the Web Components crowd, who are thinking that DOM is just fine. In fact, they want us to expand it to turn it into a universal component model (instead of buying into the ‘bolt on’ component models of widget libraries). I cannot wait for it: I always hated the barrier of entry for Web libraries. In order to start reusing components today, you first need to buy into the bolt-on component model (not unlike needing to buy another set top box in order to start enjoying programming from a new content provider).

‘Embracing the DOM’ means a lot of things, and in a widely retweeted article by Reto Schläpfer about React.js, he argued that the current MV* client side framework treat DOM as the view, managing data event flow ‘outside the DOM’. Reto highlights the example of React.js library as an alternative, where the DOM that already manages the layout of your view can be pressed into double-duty of serving as the ‘nervous system’.

This is not entirely new, and has been used successfully elsewhere. I wrote previously on DOM event bubbling used in Bootstrap that we used successfully in our own code. Our realization that with it we didn’t feel the need for MVC is now echoed by React.js. In both cases, layout and application events (as opposed to data events) are fused – layout hierarchy is used as a scaffolding for the event paths to flow, using the build-in DOM behavior.

For completeness, not all people would go as far as claim that React.js obviates the need for client-side MVC – for example, Backbone.js has been shown to play nicely with React.js.

DOM is awful

In the other corner are those that believe that DOM (or at least the layout part of it) is broken beyond repair and should be sidestepped. My micro-service camarade de guerre Adrian Rossouw seems to be quite smitten with the Famo.us framework. This being Adrian, he approached this is the usual comprehensive way, collecting all the relevant articles using wayfinder.co (I am becoming increasingly spoiled/addicted to this way of capturing Internet wisdom on a particular topic).

Studying Famo.us is an archetype of a red herring – while its goal is to allow you build beautiful apps using JavaScript, transformations and animation, the element relevant to this discussion is that they sidestep DOM as the layout engine. You create trees and use transforms, which Famo.us uses to manage the DOM as an implementation detail, mostly as a flat list of nodes. Now recall my Abe Simpson story about SWT containers and components – doesn’t it ring similar to you? A flat list of components and a layout manager on top of it controlling the layout as manifestation of a strategy pattern.

Here is what Famo.us has to say about they approach to DOM for layouts and events:

If you inspect a website running Famo.us, you’ll notice the DOM is very flat: most elements are siblings of one another. Inspect any other website, and you’ll see the DOM is highly nested. Famo.us takes a radically different approach to HTML from a conventional website. We keep the structure of HTML in JavaScript, and to us, HTML is more like a list of things to draw to the screen than the source of truth of a website.

Developers are used to nesting HTML elements because that’s the way to get relative positioning, event bubbling, and semantic structure. However, there is a cost to each of these: relative positioning causes slow page reflows on animating content; event bubbling is expensive when event propagation is not carefully managed; and semantic structure is not well separated from visual rendering in HTML.

They are not the only one with ‘the DOM is broken’ message. Steven Wittens in his Shadow DOM blog post argues a similar position:

Unfortunately HTML is crufty, CSS is annoying and the DOM’s unwieldy. Hence we now have libraries like React. It creates its own virtual DOM just to be able to manipulate the real one—the Agile Bureaucracy design pattern.

The more we can avoid the DOM, the better. But why? And can we fix it?


CSS should be limited to style and typography. We can define a real layout system next to it rather than on top of it. The two can combine in something that still includes semantic HTML fragments, but wraps layout as a first class citizen. We shouldn’t be afraid to embrace a modular web page made of isolated sections, connected by reference instead of hierarchy.

Beware what you are signing up for

I would have liked to have a verdict for you by the end of the article, but I don’t. I feel the pain of both camps, and can see the merits of both approaches. I am sure the ‘sidestep the DOM’ camp can make their libraries work today, and demonstrate how they are successfully addressing the problems plaguing the DOM in the current browser implementations.

But based on my prior experience with the sidestepping approach, I call for caution. I will also draw on my experience as a father of two. When a young couple goes through the first pregnancy, they focus on the first 9 months, culminating in the delivery. This focus is so sharp and short-sighted that many of the couples are genuinely bewildered when the hospital hands them their baby and kicks them to the hospital entrance, with newly purchased car seat safely secured in the back. It only dawns on them at that point that baby is forever – that the bundle of joy is now their responsibility for life.

With that metaphor in mind, I worry about taking over the DOM’s responsibility for layout. Not necessarily for what it means today, but couple of years down the road when both the standards and the browser implementations inevitably evolve. Will it turn into a trench warfare that cannot be won, a war of attrition that drains resources and results in abandoned libraries and frameworks?

Maybe I can figure that one out after a nap.

© Dejan Glozic, 2014

The Rise of the Full-Stack Architect

Pebble stack, Wikimedia Commons, Zzubnik
Pebble stack, Wikimedia Commons, Zzubnik

Full-Stack Web Architect needed with the experience with web services, back end Web platforms, databases, cloud based hosting such as Heroku and AWS, visual design, UX/UI, experience of mobile web development, all in agile environment.


An actual 2014 posting for a job based in London, UK

Ta-da – the moment has come. In post #2 of this blog, I promised to formally apologize for calling myself ‘an architect’. Only 39 weeks later, I am making good on the promise. Procrastinating much?

The hate on architects ebbs and flows, but never fully goes away. This creates a horrible problem for experienced developers who are fascinated with technology and are not overjoyed with becoming managers, yet clearly need to be in a leadership position of some kind because they are kicking them out of the developers’ kindergarten (what do you mean I am too old to hang around the playground, and why is it ‘creepy’?).

Management has been a traditional career growth path, and some still believe it is the only smart choice for former developers (kids, if you never before saw click bait, here it is):

From my chair, everything points in a different direction. Wherever I look (including my own company), hierarchies are being flattened, leading by example and by inspiration is replacing pulling the rank, and some companies are eliminating management altogether. So management is receeding as a leadership model to aspire to, but architects are being openly mocked:

So what is an experienced developer with so much to offer to younger colleagues to do? Looks like we are stranded between Scylla and Charybdis of the software leadership career paths. But it is a false dilemma. Bill was clear about this in his blog post – you should aspire to be a leader, to have a vision that inspires people and makes them follow you of their own will, not because they report to you. Somebody else can handle TPS reports, and you can focus on leading a growing group of people in making awesome products, and enjoying coming to work every morning because working in such an environment is intoxicating. Then you move the hell out of their way and let them do the work – they can do it better than you anyway.

OK, figured it all out – have a breathtaking vision, generate a growing following, solve the world hunger, and still have time to catch up on the new episodes of Silicon Valley on HBO. Got it.

Meanwhile on planet Earth…

For us lesser souls, some more attainable pointers:

  1. Be a wise sage – your experience with waves of technology allows you to notice repetition – new generations of developers re-solving problems from 5 or 10 years ago. When a puppy developer comes to you with a very cool client side MVC framework, point at the geological layers of MVC in the past technologies. Do that not to disparage the new framework, but to look at it for its own merit, not because MVC is a revolutionary new concept (you will know it is not, but chances are your junior colleague will not). Warning: don’t be an old fart for whom ‘things are not as good as they used to be’. That’s nonsense – a lot of things are many times better now, so be on the lookout for the first signs of being stuck in your own ways and nip it in the bud.
  2. Look over their heads into the future – there is always so much to do every day, and so few hours before your caffein-induced code turns into garbage. Allow developers to code like beasts, solving the ‘now’, while you look ahead, planning where you need to be in 3 months, 6 months, a year. Don’t try to go too far – we are all in a transition, and things are insane now.  Long term predictions have become somewhat useless lately, so ensure you have enough vision to prevent the team go idle, and be prepare to tweak the vision when course-correcting data becomes available.
  3. Resolve technical disputes – as fiery technical debates would indicate, there are many ways to skin any technical cat (a metaphor, I don’t condone skinning actual cats). One of your key jobs as a technical leader is to ensure the entire system works, and that can only happen if all the parts of the system speak the same language to each other. In other words, as long as everybody uses the same protocol, and that protocol does not suck too much, it is better than when parts of the system use superior but incompatible protocols.
  4. Prefer practical over pure – architects are often derided for making everything more complicated than it needs to be. Don’t do it. Look at the most popular APIs today – they may not satisfy REST police but they are clean, they work, they are stable and well documented and the deviations from the ideal actually solve some real world problems for developers.
  5. Maintain street creed – you can only lead developers if they respect you as one of their own, and that can only happen if you continue to code. Don’t put yourself on the critical path because you will spend too much time heads down, which will prevent you from doing (2), but you need to know what you are talking about. That requires that for every library or framework or protocol you want to recommend, you need to educate yourself, write some real code (sorry, more than just Hello, World) – give it a serious test drive.
  6. Don’t practice drive-by architecting – nothing irks developers more than an architect that designs a system, provides all the specifications, drops the finished docs on their poor heads and disappears, leaving them to suffer from the consequences of the poor choices. You need to see your architecture live in the running code, solving problems. Nothing makes me happier than developers gobbling up the new architecture and moving to it as fast and they can because it addresses their long standing problems. As a colleague of mine would say ‘the devil is in the pudding’ – if the system using your architecture is not faster, more scalable, more maintainable, your architecture sucks.
  7. Be a connector – just because people don’t officially report to you, it does not mean that you are free from practicing your soft skills. You need to talk to design, engineering, operations, product management – all speaking their own distinct languages. More importantly, you must be able to do it without strangling somebody with a Cat 6 cable. That privilege is reserved for developers, you need to be above it.
  8. Have cloud, DevOps and Web-scale in mind – practically all modern systems today are distributed systems running in data centers. They need to be evolved using Continuous Integration, features need to be promptly deployed using Continuous Deployment, services need to be clusterable, scalable and redundant. Architecting a system today without keeping these in mind is a recipe for a career spiral of death.

If you follow the suggestions above, you are really ‘system-developer-in-cheif’. Note that you are not a team lead even though it may sound like it – you are a ‘system lead’ serving a number of teams that already have team leads. You are a ‘how everything fits together lead’, which is too long to write, so for practical reasons, we will call you an ‘architect’. There – we closed the full circle.

What about full-stack?

In the olden days, developers have been segregated by their skills, because different tiers of a complex system were build with such a divergent technology that it was really hard to keep it all in your head. SQL experts were not very good at building backend systems with SOA and other bloatware, which was very different from writing Web sites with server-side MVC, which was very different from client side Ajax. Conversely, architects followed suit – they specialized as well.

With the dawn of the ‘JavaScript everywhere’, with NoSQL databases storing and passing around unstructured JSON, Node.js servers with JavaScript templating engines running on both sides of the fence, and MVC client libraries, a developer can theoretically write a whole system – soup to nuts. I say ‘theoretically’ because while the language context switching has been removed, the problem domains are still very different. There is actually a growing need for a breed of architects to ‘make sense of it all’ and devise and evolve a system that does not explode from the very wealth of choices that are now around us at every level.

Ad sense

Look at the ad at the top again – nothing in it surprises me, and I understand the motivation, but boy – that guy or girl is going to be one shiny unicorn, and I don’t know if there are very many prancing around with all those skills.

You know what, lets have fun: let’s see if I could get that job by a very scientific method of dissecting the ad and trying to match it with my blog posts. For this exercise we will be using a simplification that I actually possess in-depth knowledge about topics I write about (which is not a foregone conclusion, but let’s not go there):

As Magnus Pyke would yell – ‘Science!’. If this IBM thing does not work out, I should give them a call. My point was that there is a dire need for ‘making-sense-of-it-all-in-cheif’ in every large team. Again, this is too long, so we will call this job ‘full-stack architect’. If your company is large enough, there could be additional career ladder titles such as ‘Senior Technical Staff Member’, ‘Distinguished Engineer’, ‘Fellow’, but I really dig full-stack architect, so I will call myself that from now on.

Here is hoping that one day soon some future George Constanza will lie to people that he is ‘Art Vandelay, Full-Stack Architect’.

© Dejan Glozic, 2014

REST and MQTT: Yin and Yang of Micro-Service APIs


It seemed that the worst was over – I haven’t heard a single new portmanteau of celebrity names in a while (if you exclude ‘Shamy’ which is a super-couple name of Sheldon and Amy from The Big Bang Theory but being a plot device, I don’t think it counts). Then when I researched for this blog post I stumbled upon project QEST, a mashup of MQTT and REST. Et tu, Matteo Collina?

What Matteo did in the project QEST is an attempt to bridge the world of apps speaking REST and the world of devices speaking MQTT with one bilingual broker. I find the idea intriguing and useful in the context of the IoT world. However, what I am trying to achieve with this post is address the marriage of these two protocols in the context of micro-service-based distributed systems. In a sense, we are re-purposing a protocol not primarily created for this but that exhibits enough flexibility and simplicity to fit right in.

You keep saying that

I think I have written about usefulness of message brokers in micro-service systems often enough to reasonably expect it to be axiomatic by now. From the point of view of service to service interaction, REST poses a problem when services depend on being up to date with data they don’t own and manage. Being up to date requires polling, which quickly add up in a system with enough interconnected services. As Martin Fowler has pointed out in the article on the event collaboration pattern, reversing the data flow has the benefits of reacting to data changes, rather than unceasingly asking lest you miss a change.

However, the problem with this data flow reversal when implemented literally is that onus of storing the data is put on the event recipients. Storing the data in event subscribers allows them to be self-sufficient and resilient – they can operate even if the link to the event publisher is temporarily severed. However, with every second of the link breakage they operate on potentially more and more stale data. It is a case of ‘pick your poison’ – with apps using the request-response collaboration pattern, a broken link will mean that no collaboration is happening, which may or may not be preferred to acting on outdated information.

As we are gaining more experience with micro-service-based systems, and with the pragmatic assumption that message broker can fail, we are finding event collaboration on its own insufficient. However, augmenting REST with messaging results in a very powerful combination – two halves of one complete picture. This is how this dynamic duo works:

  1. The service with a REST API will field requests by client services as expected. This establishes the baseline state.
  2. All the services will simultaneously connect to a message broker.
  3. API service will fire messages notifying about data changes (essentially for all the verbs that can cause the change, in most cases POST, PUT, PATCH and DELETE).
  4. Clients interested in receiving data updates will react to these changes according to their functionality.
  5. In cases where having the correct data is critical, client services will forgo the built-up baseline + changes state and make a new REST call to establish a new baseline before counting on it.

How is this different from a pure implementation of Event Collaboration pattern?

  1. Messages are used to augment, not replace REST. This is in contrast to, say, Twitter streaming API where you need to make a choice (you will either use REST or stream the tweets using an HTTP connection that you keep open).
  2. While message brokers are reliable and there are ways to further increase this durability (delivery guarantees, durable queues, quality of service etc.), REST is still counted on establishing a ‘clean slate’. Of course, REST can fail too, but if it does, you have no data, as opposed to old and therefore incorrect data.
  3. Client services are not required to store data. For this to work, they still need to track the baseline data they obtained through the REST call and be able to correlate messages to this baseline. For example, if a client service rendered a Web page from the data obtained from a REST API, it should be able to detect that a message it received will affect this web page and use something like Web Sockets to update the page accordingly.

OK, but what is the actual contract?

Notice how I have mentioned the word ‘API’ multiple times, while I keep talking about ‘messaging’ in a non-committal way. And yet, there is no ‘generic’ API – by definition it requires clear contract in the way client services can interact with the API service. If we are to extend REST Yin with the messaging Yang, it has to be a true companion and become part of the API contract.

This is where MQTT comes in. As an Oasys standard, it is vendor-neutral in the same way as REST. While the protocol spec itself is detailed and intricate, most of the experience of using the protocol is ‘publishers publish messages into topics and subscribers subscribe to said topics’. That’s it.

A very useful characteristic of MQTT topic structure is that it can contain delimiters (‘/’), which opens up a possibility to sync up REST URLs and topics. This prompted some developers such as Matteo to go for full parity (essentially using the REST URL as a topic). I don’t think we need to go that far – as long as the segments that matter match, we don’t need to have the same root. I don’t think that the entire URL makes sense as a topic other than symbolically, unless you are writing a ‘superbroker’ – a server that is both a broker and a REST server (and a floor vax and a desert topping). Or an MQTT-REST bridge. Our approach is purely that of API mirroring – a convention that still expects from services to connect to a MQTT broker of their choice.

REST/MQTT API in action

So how does our approach look in practice? Essentially, you start with a normal REST API and add MQTT messages for REST endpoints that result in a state change (POST/PUT/PATCH/DELETE).

For example, let’s say we have an API service responsible for serving people profiles. The REST endpoints may look something like this:

GET /people – this returns an array of JSON objects, one per person

GET /people/:id – this returns a single JSON object of a person with the provided id, something like:

  "id": "johndoe",
  "name": "John Doe",
  "email": "jdoe@example.com"

PATCH /people/:id – this updates select fields of the person (say, name and email – we don’t support changing the id). The sequence diagram of using such an API service may look like this:


The sequence starts with the client service B making an HTTP GET request to fetch a resource for John Doe. API service will return JSON for the requested person as expected. After that, another service (client A) issues a PATCH request to update John Doe’s email address. API service will execute the request, return updated JSON for John Doe in the response, then turn around and publish a message to notify subscribers that ‘/people/johndoe’ has changed. This message is delivered to both clients that are subscribed to ‘people/+’ topics (i.e. changes to all people resources). This allows service B to react to this change.

Topics and message bodies

Since MQTT is now part of the formal API contract, we must document it for each REST endpoint that causes state change. There is no hard and fast rule on how to do this, but we are using the following conventions:

POST endpoints publish a message into the matching MQTT topic with the following shape:

  "event": "created",
  "state": { /* JSON returned in the POST response body */ }

PUT and PATCH endpoints use the following shape:

  "event": "modified",
  "changes": { "email": "johndoe@example.com" }

The shape above is useful when only a few properties have changed. If the entire object has been replaced, an alternative would be:

  "event": "modified",
  "state": { /* JSON returned in the PUT response body */ }

Finally, a message published upon a DELETE endpoint looks like this:

  "event": "deleted"

Handling i18n

If the API service is returning JSON with translatable strings, it is customary to honor ‘Accept-Language’ HTTP header if present and return the string in the appropriate locale. Alternatively, ‘lang’ query parameter can be used either on its own or as an override of the header. This all seems straightforward.

The things get complicated when you reverse the flow. API service publishing a message cannot know in advance which languages will be needed by the subscribers. We don’t have a fully satisfactory answer for this, but our current thinking is to borrow from JSON-LD and include multiple versions of translatable strings in the message body, in a way that is done in Activity Streams 2.0 draft:

  "object": {
    "type": "article",
    "displayName": {
      "en": "A basic example",
      "fr": "Un exemple basique"


While others have attempted to create a formal bridge between the REST and MQTT worlds, when building a system using micro-services we are content with achieving REST/MQTT API mirroring through convention. We find the two protocols to be great companions, packing a mighty one-two punch that maintains API testability and clear contract while making the system more dynamic, and providing for looser coupling and more sustainable future growth.

© Dejan Glozic, 2014