The Web UI Integration Continuum

Puzzle Krypt-2

One of the key roles of the Jazz Platform I am working on right now is that of integration. From the early days of the Eclipse Platform, I was fascinated with composable systems, and how this composition manifests itself in the user interface. Integration on the Web is simultaneously natural and frustrating. It is natural because the entirety of the Web is a gargantuan aggregate of parts, but it is also frustrating when this aggregation is attempted inside one of those parts. As far as the browser is concerned, you are giving it a page, the whole thing, HEAD, BODY, the works. It is not concerned with how you source the content of this page, and will in fact fight you bitterly if you want to pull a buffet style assembly of the page, sourcing parts from different domains. For the browser, the buck stops at your page, and if you disagree, you will be consigned to the iframe hell (and yes, I know about Web Components, HTML Imports and Project Polymer, but let’s talk again when there is less need for all the polyfills in the modern browsers).

As the architect responsible for Web UI integration of the Platform, I tried to get some better perspective on this topic, pulling back from the details as far as I could (until the letters on my monitor became too small to read). Everywhere I looked on the Web, I could see all kinds of integration examples, and as looking for patterns is one of my hobbies, I tried to group them in the smallest number of buckets. I have noticed that we as a species are obsessed with trying to fit anything we can lay our hands on into the four quadrants, probably owing it to our reliance on cardinal directions from the dawn of civilization. Try as I might, I could not come up with the four quadrants analogy of my own. The picture that emerged in my head turned more into the four distinct points on a continuum in the upper right quadrant:

modes-of-integration
The Web UI integration continuum

The coordinates of the space are semantic (how much the elements being integrated depend on a particular vocabulary) and compositional (how much the elements have an expectation of composition in some kind of a container). The four distinct points or modes on the continuum are:

  1. Data – in this mode, participants are asked to contribute data via services following an agreed upon vocabulary. The meaning of the data is fully known to the platform and it is responsible for rendering the data in various ways depending on the context. The platform serves as a data or status integrator. The value is in being able to see the data from various places put together and rendered by the platform, or made available to other participants to render the aggregation.
  2. Links – in this mode, participants are asked to contribute hyperlinks to various places (again, via services but this time we are returning navigation elements, not status data). The platform serves as a navigation integrator. The value is in having a one-stop shop for navigating to participants’ pages, but in a way that is fully dynamic (otherwise why not just use any of the gazillion bookmarks management tools out there).
  3. Activities – in this mode, participants are providing a timeline of activities. For the sake of generalization, we will assume that these activities are in the Activity Streams format, and we will promote an RSS feed into an activity stream where each activity’s verb is ‘post’. The platform serves as the activity integrator, curating those that are tailored to the authenticated user and producing a merged stream that can be filtered and rendered in many different ways. Incidentally, this is what various feed aggregators have been doing for eons (say, Feedly I currently use after my messy divorce with Google Reader).
  4. Widgets – in this mode, participants are providing Web widgets (a general term of a building block that does something useful, e.g. OpenSocial gadget) that the platform will arrange in some manner. The platform has little knowledge of the content of the widgets – it serves as a widget integrator. The value is in composition – putting together a page where building blocks are sourced from a number of different participants, and the platform manages the screen real estate (and may offer drag and drop customization as a bonus).

Note how the modes of integration are arranged in a continuum that goes from semantic to compositional. This gradation indicates how the emphasis is moving from pure data with little or no instructions on how the integration is to be presented, to expecting all the integration to be purely a screen real-estate transaction with little knowledge of what is in the box. The reason this is a continuum is that few real world use cases fall neatly into just one category. Links may contain a title, a description, an indication whether to navigate in place or open a new window. Activities in activity streams contain a lot of presentation content, including strings, images and snippets of HTML that will be rendered in the activity bounding box (the activity feed renderer views these snippets as black(ish) boxes, modulo markup sanitation for security reasons). Widgets that expect to be physically placed on a page may still provide some metadata allowing the container to have some basic knowledge of what they are, even though the gist of the contract is still to carve out an area of the page and let them do their thing.

What this all means in practice comes when you want to instruct a participant in the integration platform to contribute information. Let’s say that we want to hook up a server into the platform UI. We could:

  1. Provide a status REST service that when asked, returns JSON with the current state of the server “I am OK”, “I am kinda bummed”, with additional information such as memory, CPU utilization, etc., according to the agreed upon server status vocabulary. The platform can render a happy face for good server, sad face for a sad server and a hover over the server icon can show the basic stats, all sourced from the service. This is data or status integration. We can easily mix a whole mess of servers on the same page (tons of happy and sad faces, as they go on living their server lives).
  2. Servers can post an activity stream, letting us know when things change (memory low, too many processes, restarting, online again). We can merge these activities into admin’s activity feed, making him forever haunted by the needy servers and their DOS attack phobias.
  3. Servers can contribute links to their home and admin pages we can hook up into our menus and directories (alternatively, links can be sneaked in via the REST service in (1), combining status data and URLs to probe further).
  4. Servers can provide unique performance widgets providing detailed rendering of the status, CPU and memory utilization in a way we don’t need to understand other than allocate some real estate for them. Maybe we just cannot do justice to the happy or sad server faces, and only the server itself can render all the depth and complexity of its inner state.
  5. We can combine 1 through 4 in one awesome server card, containing current status, navigation for further reading, most recent server activities, and some advanced status widgets provided by the server itself. All four modes of integration folded in one neat package.

Since I formulated this model, I have tried to back-test it with all the scenarios of Web UI integration I came across. Here are some examples:

  1. Activities – Facebook, Twitter, LinkedIn, all activity feeds basically. An RSS feed variety – every feed integrator known to man. My favorite is activities that contain object types for which there are formal or informal preview widgets (say, posting a link to a youtube video that ends up being rendered with the video widget embedded in place). A great example of activity integration that brings widget integration for the ride.
  2. Status integration – a gazillion services currently in use that you can call to fetch some interesting data an render on your page (either fetching on the server and inserting content in place, or making a proxied XHR to the service and render on the client).
  3. Links integration – various federated directories, distributed site maps, any type of contribution into extensible menus, launch pads etc. Any time your web page computes the links to put in menus by collecting the links from participants (say, Jazz Home menu).
  4. Widgets integration – any web site that supports widgets – all dashboards/portals, any page where widgets can be added to the side bar, and just about any page that supports ads, Facebook ‘Like’ button, WordPress I am using to write this blog, upcoming Web Components etc.

So far I didn’t find an example of Web UI integration that cannot be nicely categorized as one of these, or a combination therein. Try it yourself – if you find a use case that defies this categorization, I will post a rendition of a big eyed sad puppy face in a follow-up blog, declaring you the winner of the dare.

© Dejan Glozic, 2013

Feed Your Web UI Without Choking It

Today’s topic is penguins, how they feed their young and how to apply that to Web development. Now watch my improbable feat of connecting these two topics. Come for the penguins, stay for the Web UI!

I am a father and as all parents know, part of the job description when your kids reach certain age includes suffering through many an animated movie (I am looking at you, Hoodwinked!). On one occasion though, I actually enjoyed the movie, and it happened to be Happy Feet. Those of you who watched it know that young Mumble is fed by his father Memphis eating a big fish, then regurgitating the half-digested food into Mumble’s beak, like so:

Antarctic adelie penguins (js) 21

This is not from the movie but I don’t want to wrestle with Warner Bros over the copyright for the images.

As I watched the scene, I started thinking about what I like to call ‘naive Ajax’. I already gave Ajax its own blog post so no need to repeat myself here. Suffice to say is that when Ajax was young, making an XHR call to fetch some data was fairly straightforward and powerful. At that time, there were readily available services of the monster variety (Big Bad Services or BBS©). These services often returned horrible XML, and in the ‘all you can eat’ amounts. Developers didn’t want to bother with writing the services themselves, preferring to stay on the client writing JavaScript. They just called the Big Bad Service, suffered through the XML, threw most of it away and built their DOM nodes out of the useful parts.

I think you are starting to see the improbable analogy here: the unprocessed, big data service is like feeding the little Mumbo raw fish – too much for its little beak. Let me count the ways why that is suboptimal:

  1. Big services normally serve verbose XML with tons of namespace information. If you are lucky and the service supports gzip, it may not be a big problem (XML is insanely compressible), but if it does not, a lot of the bytes are sent down the wire unnecessarily.
  2. XML is not the optimal format to process on the client. You can let the browser parse it into a DOM tree, and traverse the DOM. However, that means that your JavaScript code needs to be larger because it contains nontrivial amount of logic just to traverse the data and sift it out.
  3. The data may not be fully digestible (see what I did here) – you may need to derive some of it from what you have received, or make more than one XHR request and sort out the joins in-browser.
  4. Very often you get way more than you need, which is wasteful because you waited and wasted the bandwidth and traversed a big DOM only to throw it away.

It would be foolish to assume that this is the thing of the past and that modern JSON services are much better and do not suffer from this problem. You may as easily choke your Web UI with a ton of JSON data – the only difference is that you will get huge JavaScript objects instead of an XML DOM tree. The accent is on ‘impedance mismatch’, not on the exchange format.

What is the ‘right way’ then? Use what I call ‘Happy Feet Services’. Write your own service on the presentation server (the one serving the rest of the Web app resources) in a service language of your choice. Make that service call the Big Bad Services, process the response, correlate the data, toss the unused parts away – it will do a much better job on the server. Once you finally bring the data set down to a super tasty little nugget, send it to the browser as gzipped JSON. It will arrive quickly because it is small, it will be turned into a JavaScript object by the browser, and your client JavaScript does not need to be very large because the data set is readily usable – no processing required.

If you feel lucky (punk!), you can go one step further and make your Happy Feet Service even more useful: make it dual-purpose using content negotiation. If you ask for “application/json”, return JSON as described. If you ask for “text/html”, send out the same data already rendered and ready to go as HTML markup – all you need to do is pipe it into a DIV using innerHTML property. You will have dynamic Ajax app with very little JavaScript.

For this realization alone, I would say I got my money’s worth. The only thing I learned from Hoodwinked! is not to give coffee to a squirrel. I knew that already.

© Dejan Glozic, 2013

Progressive Disclosure and Progressive Reduction

nomagic

‘Make things as simple as possible, but not simpler’ is the user-friendly paraphrase of a quote from a lecture Albert Einstein delivered at Oxford in 1933. It is a wonderful irony that Einstein was proven wrong (by the composer Roger Sessions) in that he didn’t make the quote itself ‘as simple as possible’, necessitating subsequent improvements.

He also said other things. For example, when some kids where playing on his patio where concrete did not fully set, he allegedly yelled at them, later justifying the yelling with “I love children in ze abstract, not in ze concrete”. We will leave the application of the Einstein’s Gulf to his personal life for another time and focus instead on the problem of simplicity.

My designer colleague Kim Peter shared a blog post recently about LayerVault’s approach to this balancing act called progressive reduction. In a nut shell, LayerVault proposes initial user experience that is helpful and descriptive, only to be made ‘tighter’ and simpler as the user gains proficiency. This progress is supposed to be automatic (by tracking activity) – no explicit declaration from a user required (‘I am so an expert now, honestly’). The concept also provides for users being downgraded due to lack of recent activity.

Progressive reduction example (LayerVault).

The concept is not new but pairing it with usage monitoring caught my attention. I discussed it with my colleagues and got mixed reaction. As a team, we became allergic to solutions that seem overtly ‘magical’, and UI that changes between two visits without the explicit gesture definitely falls into that class (I am looking at you, Microsoft Word 2003 menus).  However, designers are inexplicably drawn to the siren call of ‘the perfect interface’ where things cannot be ‘made simpler’. Look at all the gushing over the evolution of the Starbucks logo, resulting in the removal of the company name (Starbucks reckons we are all expert coffee drinkers by now and need no hand holding in locating the nearest source of developer’s nectar).

Progressive reduction reminded me of another user experience technique – progressive disclosure. When I was on the Eclipse team, we had been tasked with solving a somewhat different problem IBM was experiencing with some of its large desktop applications (you can read the presentation about it by a former Eclipse colleague Kim Horne; favorite part: user.brain.OutOfMemoryError). They had so many features that the out of the box experience had some of the ‘747 cockpit’ feel to it. The fix of that bug was to hide most of the features and slowly reveal them as the user goes about his daily tasks (‘only if you answer these questions three, the other side of the UI you’ll see‘).

We spent a lot of time discussing whether the true fix was to ruthlessly edit the UI, or declare that all the dials and switches are required in order to actually fly the 747. On the topic of feature editing, I remember that Kevin Haaland, my manager and Eclipse Platform project manager at that time, made a distinction between true, or systemic and fake or accidental complexity, of which only the former actually matters, the latter being a by-product of things being cobbled together without enough thought about the end result. I also developed a distaste for our lack of backbone in making a design decisions, resulting in delegating too many choices to the users via a massive preference dialog (300 individual preference panels in one instance). Even the 747 cockpit has evolved to become much more streamlined and pilot-friendly – I guess we need to look for another metaphor. Oh, and progressive disclosure was ‘magical’ too – the features in the UI appeared on their own, not as a result of an explicit action.

It is no accident that progressive disclosure is not mentioned as frequently nowadays with Web or mobile applications. For the former, people are experiencing your site through the peephole of one page at a time, so it can be enormous and you only need to worry about the complexity of a single page (unless you include links to 300 other pages). And on mobile devices, it is really hard to create a mobile app with too many features due to the natural restrictions of the medium.

Perhaps another reason it is not spelled out is because it graduated into the ‘duh’ status – of course you will provide a clean initial interface and hide secondary and tertiary functionality behind menus, slots, drawers and other UI elements for stashing things away. It is just good design, nothing to see here, moving on.

Progressive reduction is a different beast altogether and is concerned with reconfiguring the experience with the level of expertise as it changes.

I think the thought process that led to progressive reduction was as follows:

  1. We worked hard with our target audience to ensure all the features in the product are actually wanted and sought after
  2. We like clean UI so most of the features are stashed away so that the initial experience is as sparse as a Scandinavian living room
  3. We now have an opposite problem – users unfamiliar with the interface not quickly finding the features they want
  4. We better put some in-context hand holding – training wheels – until the user is confident with the UI
  5. Once the user is confident with the UI, he will start resenting the hand holding
  6. Users are too busy to click on the ‘dismiss the training wheels’ button, so the UI should ‘sense’ when to reconfigure itself

I am fine with most of this progression, except the claim number 6. In-context guidance is a welcome concept as long as it right beside the UI it is related to and can be easily dismissed. But I have an issue with the claim that we need to take the act of explicit dismissal away from the users and base it instead on usage. A UI that flexes from descriptive to streamlined can easily be toggled between these modes with a gesture in a more predictable fashion, and with a minimal workload overhead (and no overhead of tracking and storing usage data). For example, task guides in the Jazz Platform prototype can be explicitly dismissed and they stay closed until summoned again:

An Example of a Jazz Platform prototype task guide
An Example of a Jazz Platform prototype task guide

My misgivings with the automated part notwithstanding, I liked the article and I think the takeaway message is to separate the modes of presentation (beginner, expert) and how the UI switches between the modes (explicit action or some kind of magic). As you can guess, I am biased towards ‘no surprises’ UI and an explicit gesture to make the UI progressively simpler (beginner>normal>expert modes) is perfectly fine with me. However, I don’t want the UI itself deciding what level of simplicity I deserve at any point in time, and if I was a bad boy and should be downgraded to the training wheels for the failure of visiting the page often enough.

© Dejan Glozic, 2013

Avro Arrow, Tick-Tock and Small Batches

I have just returned from a month-long vacation, enjoying an overdue change of context. Among other things, it helped me free up enough brain cycles to play one of my favorite games – ‘look for a pattern’.

It appears (as my Jazz Platform colleagues tell me) that I have a giant database for a brain. Anything that enters is promptly tagged and also grouped together with existing entries that somehow match. This gift allows me to entertain/annoy friends and family with speeches that start with “that reminds of me of a movie/song/novel/article in which…”. I would like to take credit for this ability but there is nothing I did to develop it. Although I am diligently working on sabotaging it through my newly acquired taste for single malt whiskeys.

My colleagues are always amused that no matter how tangential the detour is, there is always a point where I bring the narrative back to what Ward Cunningham liked to call ‘aha moment’, where it all suddenly makes sense. Allow me to try to bring you to such a moment via a very meandering path.

In 1997 I had a lot of pleasure watching a two-part docudrama about a Canadian attempt at greatness via a bold new supersonic fighter jet design called Avro Arrow (also a reminder of how much weight Dan Aykroyd put on since The Blues Brothers). The movie caused a small (albeit polite) burst of patriotism in Canada. My takeaway from the movie was a detail where they attempted to build a completely new airframe, and a completely new engine at the same time. It was never done before – normally a new design would use tried and tested engines (Rolls-Royce, in this instance). My favorite quote of the government official displeased with this risk taking: “It’s a little bit like wearing two left shoes. It may be distinctive, but it’s not too bright”. File under: minimize number of variables.

Many years later, and completely unrelated, I stumbled upon a project model called Tick-Tock practiced by Intel. Using this model, Intel starts with an established micro-architecture and shrinks the process technology (a ‘tick’), followed by a new micro-architecture once the process is proven (a ‘tock’). If something goes wrong, it is easier to pinpoint why (and fire the right people, I guess). Variable minimization again.

Finally, this particular set of neurons in my index of a brain lit up again when I hit upon The power of small batches by Eric Ries (an excerpt from his book/movement/religion The Lean Startup). To quickly paraphrase, in a context of extreme uncertainty, working on many features simultaneously is not a good strategy because it does not help us learn when things fail, and does not help us build on successes (again, because we don’t know which one of the many things we did was the turning point). Making small number of changes, validating with users and either pivoting or persevering has more chances to result in project success, particularly when we are building something completely new. Variable minimization.

And there you have it: a bold new (albeit ill-fated) fighter jet project, a model for managing complexity of CPU evolution, and the light at the end of the tunnel for sleep-deprived and overly caffeinated startup founders. In all these stories, a common theme is the importance of keeping the number of variables down so that informed decisions can be made about the course of the project as early as possible.

Now, few of us will start a new fighter jet program or build a new CPU in our spare time, so these examples may seem entertaining but impractical. But they translate quite well into any technology project where something completely new is attempted. And that is exactly what we did in the Rational Jazz Platform project. As we were building the first prototype, we wanted to take the opportunity to reset the technology base, but it was deemed too risky considering that we were just figuring out what we were building in the first place. For this reason, we build the prototype using Jazz Foundation code base we intimately knew (our ‘tick’). Once the demo was finished and presented at IBM Innovate 2013, we reset and are now writing real code on a completely new stack (our ‘tock’).

With the experience of the prototype behind us, I am glad we didn’t go the ‘distinctive, but not too bright’ route of doing both at the same time. If we found out certain features were wrong and are not needed, it would really suck if we spent a lot of time building the stack to support them first.

Jolly good, let’s celebrate with a glass of Glenlivet, neat.

© Dejan Glozic, 2013