The Web UI Integration Continuum

Puzzle Krypt-2

One of the key roles of the Jazz Platform I am working on right now is that of integration. From the early days of the Eclipse Platform, I was fascinated with composable systems, and how this composition manifests itself in the user interface. Integration on the Web is simultaneously natural and frustrating. It is natural because the entirety of the Web is a gargantuan aggregate of parts, but it is also frustrating when this aggregation is attempted inside one of those parts. As far as the browser is concerned, you are giving it a page, the whole thing, HEAD, BODY, the works. It is not concerned with how you source the content of this page, and will in fact fight you bitterly if you want to pull a buffet style assembly of the page, sourcing parts from different domains. For the browser, the buck stops at your page, and if you disagree, you will be consigned to the iframe hell (and yes, I know about Web Components, HTML Imports and Project Polymer, but let’s talk again when there is less need for all the polyfills in the modern browsers).

As the architect responsible for Web UI integration of the Platform, I tried to get some better perspective on this topic, pulling back from the details as far as I could (until the letters on my monitor became too small to read). Everywhere I looked on the Web, I could see all kinds of integration examples, and as looking for patterns is one of my hobbies, I tried to group them in the smallest number of buckets. I have noticed that we as a species are obsessed with trying to fit anything we can lay our hands on into the four quadrants, probably owing it to our reliance on cardinal directions from the dawn of civilization. Try as I might, I could not come up with the four quadrants analogy of my own. The picture that emerged in my head turned more into the four distinct points on a continuum in the upper right quadrant:

modes-of-integration
The Web UI integration continuum

The coordinates of the space are semantic (how much the elements being integrated depend on a particular vocabulary) and compositional (how much the elements have an expectation of composition in some kind of a container). The four distinct points or modes on the continuum are:

  1. Data – in this mode, participants are asked to contribute data via services following an agreed upon vocabulary. The meaning of the data is fully known to the platform and it is responsible for rendering the data in various ways depending on the context. The platform serves as a data or status integrator. The value is in being able to see the data from various places put together and rendered by the platform, or made available to other participants to render the aggregation.
  2. Links – in this mode, participants are asked to contribute hyperlinks to various places (again, via services but this time we are returning navigation elements, not status data). The platform serves as a navigation integrator. The value is in having a one-stop shop for navigating to participants’ pages, but in a way that is fully dynamic (otherwise why not just use any of the gazillion bookmarks management tools out there).
  3. Activities – in this mode, participants are providing a timeline of activities. For the sake of generalization, we will assume that these activities are in the Activity Streams format, and we will promote an RSS feed into an activity stream where each activity’s verb is ‘post’. The platform serves as the activity integrator, curating those that are tailored to the authenticated user and producing a merged stream that can be filtered and rendered in many different ways. Incidentally, this is what various feed aggregators have been doing for eons (say, Feedly I currently use after my messy divorce with Google Reader).
  4. Widgets – in this mode, participants are providing Web widgets (a general term of a building block that does something useful, e.g. OpenSocial gadget) that the platform will arrange in some manner. The platform has little knowledge of the content of the widgets – it serves as a widget integrator. The value is in composition – putting together a page where building blocks are sourced from a number of different participants, and the platform manages the screen real estate (and may offer drag and drop customization as a bonus).

Note how the modes of integration are arranged in a continuum that goes from semantic to compositional. This gradation indicates how the emphasis is moving from pure data with little or no instructions on how the integration is to be presented, to expecting all the integration to be purely a screen real-estate transaction with little knowledge of what is in the box. The reason this is a continuum is that few real world use cases fall neatly into just one category. Links may contain a title, a description, an indication whether to navigate in place or open a new window. Activities in activity streams contain a lot of presentation content, including strings, images and snippets of HTML that will be rendered in the activity bounding box (the activity feed renderer views these snippets as black(ish) boxes, modulo markup sanitation for security reasons). Widgets that expect to be physically placed on a page may still provide some metadata allowing the container to have some basic knowledge of what they are, even though the gist of the contract is still to carve out an area of the page and let them do their thing.

What this all means in practice comes when you want to instruct a participant in the integration platform to contribute information. Let’s say that we want to hook up a server into the platform UI. We could:

  1. Provide a status REST service that when asked, returns JSON with the current state of the server “I am OK”, “I am kinda bummed”, with additional information such as memory, CPU utilization, etc., according to the agreed upon server status vocabulary. The platform can render a happy face for good server, sad face for a sad server and a hover over the server icon can show the basic stats, all sourced from the service. This is data or status integration. We can easily mix a whole mess of servers on the same page (tons of happy and sad faces, as they go on living their server lives).
  2. Servers can post an activity stream, letting us know when things change (memory low, too many processes, restarting, online again). We can merge these activities into admin’s activity feed, making him forever haunted by the needy servers and their DOS attack phobias.
  3. Servers can contribute links to their home and admin pages we can hook up into our menus and directories (alternatively, links can be sneaked in via the REST service in (1), combining status data and URLs to probe further).
  4. Servers can provide unique performance widgets providing detailed rendering of the status, CPU and memory utilization in a way we don’t need to understand other than allocate some real estate for them. Maybe we just cannot do justice to the happy or sad server faces, and only the server itself can render all the depth and complexity of its inner state.
  5. We can combine 1 through 4 in one awesome server card, containing current status, navigation for further reading, most recent server activities, and some advanced status widgets provided by the server itself. All four modes of integration folded in one neat package.

Since I formulated this model, I have tried to back-test it with all the scenarios of Web UI integration I came across. Here are some examples:

  1. Activities – Facebook, Twitter, LinkedIn, all activity feeds basically. An RSS feed variety – every feed integrator known to man. My favorite is activities that contain object types for which there are formal or informal preview widgets (say, posting a link to a youtube video that ends up being rendered with the video widget embedded in place). A great example of activity integration that brings widget integration for the ride.
  2. Status integration – a gazillion services currently in use that you can call to fetch some interesting data an render on your page (either fetching on the server and inserting content in place, or making a proxied XHR to the service and render on the client).
  3. Links integration – various federated directories, distributed site maps, any type of contribution into extensible menus, launch pads etc. Any time your web page computes the links to put in menus by collecting the links from participants (say, Jazz Home menu).
  4. Widgets integration – any web site that supports widgets – all dashboards/portals, any page where widgets can be added to the side bar, and just about any page that supports ads, Facebook ‘Like’ button, WordPress I am using to write this blog, upcoming Web Components etc.

So far I didn’t find an example of Web UI integration that cannot be nicely categorized as one of these, or a combination therein. Try it yourself – if you find a use case that defies this categorization, I will post a rendition of a big eyed sad puppy face in a follow-up blog, declaring you the winner of the dare.

© Dejan Glozic, 2013

Making Gourmet Pizza in the Cloud

A while ago I had lunch with my wife in a Toronto midtown restaurant called “Grazie”. As I was perusing the menu, a small rectangle at the bottom attracted my attention. It read:

We strongly discourage substitutions. The various ingredients have been selected to complement each other. Substitutions will undermine the desired effect of the dish. Substitutions will slow down our service.

When I read it first, I filed it under ‘opinionated chef’ (as if there is any other kind) and also remembered another chef named Poppie from a Seinfeld episode, who opined on this topic thusly:

POPPIE: No, no. You can’t put cucumbers on a pizza.

KRAMER: Well, why not? I like cucumbers.

POPPIE: That’s not a pizza. It’ll taste terrible.

KRAMER: But that’s the idea, you make your own pie.

POPPIE: Yes, but we cannot give the people the right to choose any topping they want! Now on this issue there can be no debate!

Fast forward to 2013. This whole topic came back to my volatile memory as I was ruminating about what the cloud platform model means for web app developers. In the years before the cloud, we would carefully consider our storage solution, our server and client technology stack and other elements with an eye on how the application is going to be deployed. You wanted to minimize support and maintenance cost, administration headaches and also wanted to achieve economy of scale. A large web application is composed of components, and the economy of scale can be achieved if one architectural style is followed throughout, one database type, one web framework that supports components etc.

In this architecture, it is just not feasible to allow freedom for components to have a say in this matter. Maintaining three different databases or forcing three Web frameworks to cooperate is just not practical, even though these choices may be perfect for the components in question. The ‘good enough’ fit was a good compromise for consistency and ease of support and maintenance.

In contrast, a modern web application is now becoming a loose collection of smaller apps running in a cloud platform and communicating via agreed upon protocols. One of the key selling points of the cloud platform is that it is easy to write and publish an app and have it running within minutes. I am sure early successes are important for learning, and many apps will be tactical and time-boxed – up for a couple of months, then scrapped. But a much more important aspect of the new architecture to me is the opportunity to carefully curate the components of your federated application, the way an opinionated chef selects ingredients for a gourmet pizza.

Let me explain. A modern cloud platform such as EC2, Heroku or CloudFoundry provides for you a veritable buffet of databases (MySQL, Postgres, MongoDB, SQL Server), caching solutions (Memcache, Redis), message queues (RabbitMQ), language environments (Java, Scala, Python, Ruby, JavaScript), frameworks (Django, Spring, Node/Express, Play, Rails). You can then package a carefully curated collection of client-side JavaScript frameworks to include in our app. You have the freedom to choose, and not only in one way:

  1. You have the freedom to select the right database, server and client side that is the best fit for the problem that a single app is trying to solve. You can write computationally intensive app in Java or Scala, write front end app with a lot of I/O and back-and-forth chatter in Node.js, and another in Django or RubyOnRails because you simply like it and it makes you more productive. You don’t have to use whatever has been chosen by the entire team even though it does not fit your exact needs.
  2. You then have the freedom to compose your federated application out of individual apps cooperating as services, each one optimized for its particular task. As I said earlier, this would have been prohibitively expensive and impractical in the past, but is now attainable for smaller and smaller providers. The cloud platform has subsumed the role of the economy of scale vehicle that was previously required from large applications deployed on premise.

The vision of the polyglot future has been already declared for programming languages by Dave Fecak et al and also for storage solutions by Martin Fowler. A common thread in both is ‘best fit for the task’. In the former, additional angle was employment prospects of polyglot developers, while in the latter was a caveat that polyglot persistence comes with the expense of additional complexity.

I think that my point of view is subtly different: in the cloud platforms, complexity of managing many different systems is mitigated by the fact that the platform is responsible for most of it. On the other hand, while CFOs are salivating over the prospects of slashed IT expenses in hosted solutions, I am more intrigued by another unexpected side-effect: using the right tool for the right job and using multiple tools within the same project is now attainable for smaller players. All of a sudden, an application running multiple servers, each using a different database type is trivially possible for even the smallest startups.

We will never be asked again to put cucumber on our pizza just because the team as a whole has decided so, even though we hate cucumber. Conversely, we CAN put cucumber just on our pizza even though the rest of the team doesn’t like it if works wonderfully in our unique combination. In the new world of cloud platforms, we can all make gourmet pizza-style applications, each carefully put together, each uniquely delicious, and each surprisingly affordable. The opinionated chef from Grazie now seems almost prophetic: in the cloud, we DO want to select various ingredients to complement each other, because substitutions will undermine the desired effect and slow down our service(s).

You can almost pass the restaurant’s menu as the white paper.

© Dejan Glozic, 2013