Providing for Startups: the New California Gold Rush

California clipper, Wikimedia Commons

The California Gold Rush (1848–1855) lasted only 7 years. In that amount of time, it brought 300,000 people to California, invigorated the US economy, played a key role in ushering California to statehood in 1850, and caused a precipitous population decline of Native Californians.

One of the most profound lessons of this period in history stems from the analysis of winners and losers. It is well known that merchants made far more money than miners during the Gold Rush. One of them is a ubiquitous American brand Levi Straus & Co that is still selling rugged and durable work pants that put them on the map (‘wild horses could not tear them apart’). The great thing about supplying prospectors was the steady business: they needed shovels, tools, food, clothing, lodging – irrespective of whether they ended up finding gold or not. This is not much different from investment banks charging fees for every trade. They reap the profit by making the market, not by going long or short.

A similar dynamic is now on clear display in Silicon Valley. Less that 100 miles from Sacramento and Coloma, the ground zero of the original Gold Rush, modern merchants are popping up at a great speed. Their common goal is to supply startups with everything they need on their quest to the magical inflection point, when your traction graph turns into the coveted hockey stick.

Modern merchants

There are currently many examples of companies built with the express purpose of supplying startups running their 1.0 experiments. To qualify, a company needs to have a business model that does not directly sell a service or product to customers. Instead, their customers are startups and their business model is that of enablement.

Consider these examples – there are many more, but I am using some I personally came across or had to evaluate for our own product:

  • Optimizely – one of the key properties of building a 1.0 product is the extreme uncertainty. Most startups are throwing platefuls of spaghetti at the wall, hoping some will stick. The technical process of throwing the spaghetti is called ‘A/B testing’, and Optimizely allows you to create experiments, expose your users at A or B variant of any part of your product, and find the winner. You don’t quite know why A or B is better but Optimizely can tell you which one to pick.
  • Appcues – common startup headache is getting users to understand your value proposition, use your product and stay with it long enough to drive up engagement metrics. Appcues will allows you to create product tours, and even follow users through tasks, observing their progress and suggesting next steps.
  • LaunchDarkly – 1.0 products are a mix of key features that work well, and newly built features that are a bit more shaky but may still offer value to some users that opt in. Controlling the feature flags, dynamically updating them, building up white lists and other more sophisticated ways of controlling your product gets complicated quickly, and many startups delegate this to LaunchDarkly.
  • Stripe – once you achieve engagement and a growing list of returning users, it’s time to monetize your idea. That means managing payments – what Stripe does.
  • Freshdesk – once you get paying customers, you need to provide customer support. Freshdesk offers support ticket management and various tools to streamline the support process.

Why do these companies even exist? Because most startups are based on a single, hopefully groundbreaking idea. Operationalizing that idea and bringing it to users is like a series of concentric circles, with layers of table stakes around the pure idea. Most startups want to get the idea out as soon as possible (they are racing against the VC funding runway), and need to outsource most of the enabling facilities.

Building most 1.0 products is like building a tent city – you need to do it fast, and be able to dismantle and move to a different location on a moment’s notice (should pivoting become necessary). It is simply not cost-effective to invest in building sturdy brick houses at this point in time.

The dangers of outsourcing

I am not a fan of any of these products. This should come at no surprise as I am not working for a startup. I work for a large company that can fund a new idea until it runs out of patience, not money. But you don’t need to work for a large company to share my sentiment – all you need to do is work for a startup that saw the hockey stick and is now building version 2.0 of your product, having validated the idea and secured long-term funding and even some revenue.

Modern merchants are Trojan horses

This is common sense – most of the companies I listed above are in the Cloud, and you engage with them by creating an external account. In order for their value-add to find a way to your product, you need to place some glue JavaScript in your pages, invoking the APIs with your account’s API key. Not all of the supporting apps work like this – some can be called on the server side, but sufficient number of them act too close to a Trojan Horse for comfort.

For example, Optimizely allows you to run A/B experiments by controlling portions of your site. You can inject any DOM element directly through the client side API you add to your own page. It does not even have to be malicious – a junior Offering Manager can make a mistake and instantly break your product in production, bypassing Dev/QA/Prod quality control process you so carefully constructed.

In the interest of full disclosure, Optimizely does acknowledge this problem and now offers Optimizely FullStack option, allowing server side A/B experiments, where developers fully control the page elements they are testing.

Appcues is actually making it clear in their Enterprise marketing page:

Empower your teams to create compelling user experiences without engaging engineering and development resources.

Appcues Enterprise

Obviously, the only way you can ‘create compelling user experiences’ without ‘engaging development resources’ is by bypassing your engineering and injecting UI into the pages directly, from a third-party server, at real time. Developers are going to love that.

Once the neck-breaking days of 1.0 are over, you will want some quality control of your development process. This will become really hard if your product is a Frankenstein’s monster, containing a mix of your original code and many third party services. At any given point, interruption in these services can affect your site (I am not even talking about actual Cloud compute and storage providers – that’s one layer below). It is also hard to enact any kind of development process progression. You can run any number of automated Q/A tests in your pre-prod stage – a significant portion of your product’s behaviour is controlled outside of your build approval process.

Data security issues and global operations

As these modern Gold Rush merchants assist you in vital operation of your 1.0 product, they have access to sensitive areas – PII (personally identifiable information), users, data they generate, account transactions, engagement metrics etc. Of course, each of them will tell you that security is very important to them, and will have dedicated resources about their security posture. Nevertheless, your product’s security guarantee is dependent on the sum of the security guarantees of each and every third party service you use. This may be acceptable as you are getting your loving product off the ground, but may become an issue once you start growing your 2.0, or try to launch the coveted Enterprise edition. Enterprise customers are known to ask prickly questions about security, and may frown at your use of any of the services that got you thus far. It may even become impossible to secure GDPR compliance with some of them, simply because they don’t have servers in the right geography, or are not GDPR compliant.

Even if you manage to retain satisfactory security posture, some of these services may become a limiting factor of your growth plans. Many of these products are built by California startups themselves, and lack worldwide deployment. This means that even if you start deploying your product to various data centers around the globe, calling back to US West coast servers may invalidate most of the performance gains of deploying closer to non-US customers.

Cloud dependency

Regardless of how you engage with these products (client side or full stack), they exist as third party servers you need to talk to. Most Enterprise editions of Cloud products install on premises. As a rule, most startups born on the Cloud dream about the point in the future when they will offer Enterprise edition and finally make it big. Unfortunately this entails being able to make your entire product installable behind Enterprise firewalls (my daily life involves using Enterprise version of a number of the popular Cloud products such as GitHub, Slack and Travis behind my company’s firewall).

Once you get to this point, you will rue the day you became so dependent on dozens of modern Gold Rush merchants. You will have to slowly replace each and every one, or disable them in the Enterprise edition. Luckily, some of them will simply not be needed: for example, financial transaction management is useless because Enterprise versions are typically sold through sellers or paid online, then downloaded and installed.

Enterprise customers frown upon any outbound traffic attempt from their servers (some insist on air-gap environments), which means that most of the usage tracking tooling will need to be disabled (or replaced with batch tooling that collects usage, then sends it as a digest, not real time, assuming it is enabled by the Enterprise customers).

A pattern I have observed in recent years is that companies with Enterprise versions of their products use their Cloud counterparts to pilot and launch new features, and run experiments there. Once hardened, they are made available in the Enterprise version, minus all the supporting Cloud software. Features typically lag the Cloud version by 2-3 months, subject to customers’ maintenance windows.

Parting thoughts

I don’t want to leave you with an idea that products I mentioned have no place in the modern application architecture. My advice is to simply understand what they are – ready-made parts of a 1.0 US West Coast-based Cloud product prioritizing rapid growth. If that fits your startup, gobble them up.

If, on the other hand, your product is more mature, has established customers, you are not a startup any more, want to regain control over your development process, security and performance, need worldwide deployment, or are building Enterprise edition of your product, understand that you will most likely have to re-build most of that functionality in-house.

Last but not least, be on the lookout for organizational warning signs. If the Design Lead pushes for Optimizely so that she can run the A/B experiments without having to deal with the impossible Development Lead, software is the least of your problems. The last thing you need is your organizational power struggles to drive your product architecture.

© Dejan Glozic, 2019

Celestial Bodies and Micro-Front End Reuse

Source: Wikimedia Commons

If you are writing a modern production system of any complexity, chances are you are using microservice architecture for it. If that system serves the Web browser, sooner or later you will discover that you must break up the UI into parts served by multiple micro-services (or micro front-ends).

The promise of microservices is that they can be built by separate teams, allowing teams relative independence and ability to move at their own speed, without the need to for intergalactic coordination. However, sooner or later executives (God bless them) will notice that some of the user interfaces contain elements that look similar across the system, and the dreaded word “Reuse” will raise its head.

It can sound downright heretical to say anything against reuse in software industry. That code reuse saves time and improves quality feels so innately right that it takes gumption to raise against it. What would be more logical than to create a reusable component for things that look similar and make all micro front-ends reuse them?

And yet.

Movement on a planetary scale

To try to illustrate the problem we can hit with indiscriminate reuse, let’s look at the movement of planets in a solar system. As planets rotate around the star, they follow their own orbits that place them at various distances to each other. If we simplify the geometry and make all orbits perfect circles and in the same plane, the planets will still have different rotational speed, dictated by various factors (yes, ‘Gravity’ violated many of them). This rotation will place them at various positions to each other, with two extremes:

  1. Conjunction – when two planets are lined up on the same side relative to the star and closest to each other
  2. Opposition – when two planets are lined up on opposite sides relative to the star

As planets are in constant motion, they will at any moment be between these two points.

Back to the topic

OK, so let’s bring it back to the microservice system. The reason UI component reuse is being discussed in your meetings is because a design pass was made, and the first iteration of the whole system’s user experience has been put together, and some repetition is observed. This page in micro front-end A looks very similar to the page in micro front-end B. That picker is the same in three pages. Let’s solve it once and for all for everybody. Reuse, people! Sensible leadership all around.

It is easy to fall into this trap because reuse works great for, say, libraries. Of course you will not write that set of mathematical transformations over and over everywhere. In our Node.js microservice system, the list of modules each app depends on is getting longer and longer.

I just typed npm list > deps in one of our largest microservices (I am tempted to call it Gustav for its girth). The flattened list of all dependencies is 3689 lines long. We should call it “miniservice” or just “service” by now. I am not writing 3000 modules by hand any time soon. Reuse is fine and dandy here.

Similarly, all our micro front-ends are using the same platform API layer. The APIs are ensuring the state of the microservice system is consistent. Again, I am not saving the same state in two APIs and keeping it in sync somehow – that would be crazy.

All our micro front-ends are using the same style guide and the same set of React components written to follow that guide with a set of basic visual building blocks (no, we are not using Bootstrap; yes, we know 15% of all web sites use it – good for them). Without it, our UI will look like a ransom note:

Say NO to casual UI reuse

This brings us to reusing actual parts of pages between micro front-ends. If you follow the reasoning above, it is the logical next step. This is where our astrophysics comes into play.

The pages that look similar today are like planets in conjunction. They are at their closest position in the microservice planetary system. The danger is that we are forgetting the constant movement. It may be hard to believe right now, but these pages will move apart. Why? Complex systems rarely evolve in perfect coordination. You cut your hair and a month later the cut starts losing shape, because the hair refuses to grow exactly the same everywhere. Diverging priorities, architectural changes, business pressures, reorgs all conspire against your microservices keeping their relative distance to each other.

The whole point of a microservice system is to insulate against this corporate chaos by allowing self-sufficient teams to cut through the haze and continue to make progress. Wonton UI reuse will tie the microservices together and restrict their freedom of movement. This is an elaborate way to write a distributed monolith, the worst kind. In our planetary analogy, you are applying super-natural gravitational forces between planets that completely mess up their trajectories and may result in the fiery collapse of the entire system. Or ‘Gravity 2’.

Doing it right

The heretical thought presented in this article is that some duplication and bespoke UI code is good if the outcome is preserved agility and independence of the microservice teams. Of course, thousands of lines of duplicated code is its own kind of a waste, agility or not. This is where good practices are essential, such as using back end for front end to push most of business logic to the server, leaving mostly presentation logic on the client. The business logic can then reuse libraries and modules across microservices (within reason, you need avoid the danger of increased coupling so don’t go overboard).

If you do it right, you will already use common style, common APIs, shared libraries and common widgets. The rest is just composition, where you can create a bespoke UI that perfectly fits your page and nobody will break you by pushing a change without enough testing at 2am.

Any exceptions?

Yes, there are exceptions. Some UI components are just too tricky to implement even once, let alone multiple times. You will know those when you see them – the microservice teams will come together and agree that one of them will build it for all of the teams because life is too short to fix that horrible bug 5 times.

The threshold to clear will be:

  1. Is this truly a single component, or multiple components merged together, selected with a maze of flow control and configuration parameters? If latter, don’t.
  2. Is the component simple enough and with a single purpose, more likely to survive inevitable zigs and zags of business requirements and design changes?
  3. Is the team writing the shared component up to the task and ready to write exhaustive functional and UI tests to guard against regressions and breakages?
  4. Is the problem complex enough to overcome the overhead of reuse?

Once you are true to these conditions, reusable components will be few and far between. If a once useful reused component becomes a sore point in a year, fork and take the control back.

You will reap the consequences

In complex microservice systems, some of the truisms of software development need to be put to the test and re-examined. The true test is not if applying them works at the current moment (at the planetary conjunction), but if the system will survive the opposing forces of changing business requirements, architecture evolution, aging code and organizational changes.

As engineers, we must remember that business defines requirements, design defines the UX, but we are responsible for turning the requirements and the design into clickable reality. It is our responsibility to choose that reality not only based on what our system looks like today, but what it may look like in 6, 12 or 18 months. Resist the urge to reach for quick shortcuts through casual reuse, and plan for what situation that reuse will likely put you after a few architectural and organization changes.

Like planets, your microservices will never stand still.

© Dejan Glozic, 2017





Microservice Grid and Micro Frontends

Adam naming the animals, Wellcome Library, London. Source: Wikimedia Commons.

This post is not here to bring you anything new as it is for me to run a victory lap. If there is anything I blogged about with any consistency, it was about microservices. There are many articles in this blog addressing various problems you may face when attempting a microservice system of a non-trivial scale. One area I was particularly keen on was decomposition of the front end in a way similar to what we did in the back.

You would think that it goes without saying, but a lot of microservice noise was focused on the backend. I understand that: breaking the big server into a number of smaller ones behind a routing proxy is easier to understand. REST API is REST API is REST API. A number of endpoints per microservice neatly collate into a complete system with use of session-less API services.

But what’s good for the goose ought to be good for the gender. Slapping a massive, unruly front end server in front of a nice microservice API system does not help us much, other than to highlight the irony of the half-hearted approach. That’s why I focused a lot of this blog on composing the UI microservices as well. From your favorite article on microservice authentication, to UI composition, to isomorphic microservices using Node.js and React.

Far from being armchair generals, we put our money where our mouth is, applying these ideas in production, learning more in each successive generation. The last name for this approach I used was microservice grid:

microservice-gridThe picture above represents the essence of our current approach. It is uniform in that horizontally it represents tiers (front end, back end, optional hosted legacy apps in containers). Three buses provide glue – UI domain for bringing all paths under the same URL, API domain for collating all API endpoints, and an event bus to glue the microservices using a message broker of some kind.

Vertically, each column of the grid represents a feature. What I like about this approach is that unlike with big monolith apps, adding features does not make everything bigger and slower. Adding a new feature vertical does not affect the performance of the existing verticals.

So that’s what you call it

One of my favorite hobbies now is going to Technology Radar published by ThoughtWorks and looking for how they are naming a technique we are already using in production. For example, the pattern of providing a single API gateway for your client side code is now called Back End for Front end. But what gave me no end of pleassure was that now the whole approach of this blog that I spent so much time writing about has an official name: micro frontends.

We’ve seen significant benefit from introducing microservice architectures, which have allowed teams to scale delivery of independently deployed and maintained services. However, teams have often struggled to avoid the creation of front-end monoliths—large and sprawling browser applications that are as difficult to maintain and evolve as the monolithic server-side applications we’ve abandoned. We’re seeing an approach emerge that our teams call micro frontends. In this approach, a web application is broken up by its pages and features, with each feature being owned end-to-end by a single team. Multiple techniques exist to bring the application features—some old and some new—together as a cohesive user experience, but the goal remains to allow each feature to be developed, tested and deployed independently from others. The BFF – backend for frontends approach works well here, with each team developing a BFF to support its set of application features.

Technology Radar, Nov 2016, ThoughtWorks

Like with Adam in the engraving above, things exist before they are named, but Nomen est Omen – naming our approach in a way that offers a nice symmetry with micro services gives it a renewed meaning and is a success in my book.

Therefore, go forth and build your microservice grids, your micro frontends, your Backends for Frontends, you princes of Maine, you kings of New England.

© Dejan Glozic, 2017

The Web of Hooks


It’s a new year, which means it is time to update the copyright at the bottom. I have been busy lately and I didn’t want to write fluffy blogs just to fill the space. But this time I actually have real content, so here it goes.

If you followed my blog in the past, you know I was very bullish on message brokers in the context of microservice architecture. I claimed that REST APIs alone are not sufficient to sustain a successful microservice system. Event collaboration pattern is necessary to ensure a scalable and robust system where microservices that own resource lifecycle don’t need to be burdened with executing all the code driven by that lifecycle. You can read more about it in my article about REST/MQ mirroring.

Not so fast

There is a fly in this particular ointment, however. While HTTP REST is well understood, messaging protocols are numerous. You may know from my writing that I liked MQTT for a while due to its simplicity and wide support. However, MQTT is actually not a great protocol for gluing your microservice system. It is designed to be lightweight and run on the smallest of IoT devices, and lacks some critical features such as anycast support and manual acknowledgement of messages.

Another popular protocol (AMQP) suffered a schism of sorts. The version supported in a popular RabbitMQ broker (0.91) is a very different protocol from an actual open standard AMQP 1.0 that is not as widely supported. This is a pity because I really like AMQP 1.0. It is like MQTT but with anycast and manual acknowledgment – perfect as a microservice system glue, yet still very simple to work from within clients.

And of course, there is Apache Kafka, a very powerful but odd choice given its origins in high-throughput distributed log aggregation. Kafka is unapologetically Java-centric, has a proprietary protocol and bolt-on REST APIs to connect client languages are still fairly low level. For example, where AMQP 1.0 guarantees quality of service and requires implementations to provide a buffer of messages that are re-delivered to a client that crashed, Kafka simply allows you to maintain a pointer in the queue but it is your job to work up the queue and catch up after restarting. You pay for performance by working at a level fairly low for general purpose application messaging.

Authentication pains

Choosing the messaging solution and protocol is only one part of the problem. If you want your system to be extensible and allow third-party integrations, you need to make it fairly easy for new clients to connect. On the other hand, you need to secure the clients because you don’t want a rogue app to eavesdrop on the events in the system without authorization (otherwise you have this egg on the bot face). Maintaining a large list of clients connected to the same topics can also be an scalability issue for some brokers.

For all these reasons, it has been widely acknowledged that message brokers are not a good match for external integrations. A cursory scan of popular cloud applications with large ecosystems all point at a more client-friendly alternative – WebHooks.

WebHooks to the rescue

You would think that if something is so popular and has a catchy name, there is an actual well written protocol for it. Wrong! WebHooks is the least common denominator you could imagine. In a nutshell, this is all there is:

  • You publish a list of valid events for which you will notify clients
  • You provide an API (and/or UI) for clients to register URLs for one or more of those events
  • When an event happens, you execute HTTP POST on the URLs registered for that event type.

That’s all. Granted, message brokers have topics you can publish and subscribe, and the actual messages they pass around are free-form, so this is not very different. But absent from WebHooks are things such as anycast, Quality of Service, manual acknowledgement etc.

I don’t intend to go into the details of what various implementations of WebHooks in apps such as GitHub, Slack etc. provide because thankfully Giuliano Iacobelli already wrote such an article. My interest here is to apply this knowledge to a microservice system we are building and try to anticipate pros and cons of going with WebHooks.

What it would take

First thing that comes to mind is that in order to support WebHooks, we would need to write a new WebHook service. Its role would be to accept registrations, and store URL and event type mappings for subsequent invocation. Right there, my first thought is about the difference between external and internal clients. External clients would most likely use the UI to register a URL of their integration. This is how you register your script in GitHub so that it runs on every commit, for example.

However, with internal clients we would have a funny problem: every time I restart a microservice instance, I would need to register somewhere in startup. That would make a POST endpoint a nonstarter, because I don’t want to keep creating new registrations. Instead, a PUT with a client ID would work better, where an existing registration for the same ID would just be updated if already there.

Other than that, the service would offer a POST for a new message into the provided event type that would be delivered to all registered URLs for that type. Obviously it would need to guard against 404s, 502s and URLs that take too long to return response, giving up on them after a set timeout.

The best of both worlds

The set timeout brings back the topic of the quality of service, implying that WebHooks are great for external integration but not that great for reliable glue of a microservice system. Why don’t we marry the two then? We could continue to use message broker for reliable delivery of internal messages, and hook it up to a WebHook service that would notify external integrations without the need to support our particular protocol, or get too much access into the sensitive innards. Hooking up a WebHook service to a message broker would have the added benefit of buffering the service itself so that it can be restarted and updated without interruption and missed events.


In the diagram above, our microservice system has the normal architecture with a common routing proxy providing a single domain entry into the microservices. The microservices use normal message broker clients to publish to topics. A subset of these topics deemed suitable for external integrations is also listened to by the WebHook service, and for each of those messages it reaches into the stored list of registrations and calls HTTP POST on the registered URLs. If the WebHook service crashes, a reputable message broker will maintain a buffer of messages to re-deliver them upon restart. For performance reasons, WebHook service can choose to keep a subset of registrations in the in-memory cache depending on how frequently they are used.


Obviously registering a URL with an HTTP PUT is much easier to implement, and providing a single POST endpoint to handle the event lowers the barrier of entry for external integrations. In fact, hooking up code to react to a single POST could very well be done using serverless architecture.

Are we losing something in the process? Inserting another service into the flow will add a bit of a delay but external notifications are normally for events that are not happening many times a second, so the tiny delay is more than acceptable tradeoff. In addition, if the client providing WebHook URL is itself load-balanced, this delivery will be hardcoded to anycast (the event notification will only hit one of the instances in the cluster).

Finally, this creates two classes of clients – ‘inner circle’ and external, segregated clients. Inner circle is hooked up directly to the message broker, while the external clients go through the service. In addition to this being an acceptable price to pay for easier integration, it is useful to be able to only expose a subset of events externally – some highly sensitive internal events may only be available to ‘trusted’ clients subscribing to message broker topics and having internal credentials.

Since the WebHook service will normally not keep retrying to deliver an event to an unresponsive URL, it is possible to miss an event. If this is a problem, external system would need to fashion a ‘belt and suspenders’ fortification, where event driven approach is augmented with a periodic REST API call to ‘compare notes’ and ensure the baseline it is working against is up to date.

© Dejan Glozic, 2017

Components Are (Still) Hard on the Web


Here’s Johnny! I know, I know. It’s been a while I posted something. In my defence, I was busy gathering real world experience – the kind that allows me to forget the imposter syndrome for a while. We have been busy shipping a complex microservice system, allowing me to test a number of my blog posts in real life. Most are holding up pretty well, thank you very much. However, one thing that continues to bother me is that writing reusable components is still maddeningly hard on the Web in 2016.

Framing the problem

First, let’s define the problem. In a Web app of a sufficient complexity, there will be a number of components that you would want to reuse. You already know we are a large Node.js shop – reusing modules via NPM is second nature to us. It would be so nice to be able to npm install a Web component and just put it in your app. In fact, we do exactly that with React components. Alas, it is much harder once you leave the React world.

First of all, let’s define what ‘reusing a Web component’ would mean:

  1. You can put a component written by somebody else somewhere on your page.
  2. The component will work well and will not do something nasty.
  3. The component will not look out of place in your particular design.
  4. You will know what is happening inside the component, and will be able to react to its lifecycle.

First pick a buffet

Component (or widget) reuse was for the longest time a staple of desktop UI development. You bought into the component model by using the particular widget toolkit. That is not a big deal on Windows or MacOs – you have no choice if you want to make native applications. The same applies to native mobile development. However, on the Web there is no single component model. Beyond components that are intrinsic part of HTML, in order to create custom components you need to buy into one of the popular Web frameworks first. You need to pick the proverbial buffet first before you can sample from it.

In 2016, the key battle is currently between abstracting out and embracing the Web Platform. You can abstract the platform (HTML, CSS, DOM) using JavaScript. This is the approach used by React (and my team by extension). Alternatively, you can embrace the platform and use HTML as your base (what Web Components, Polymer and perhaps Angular 2 propose). You cannot mix and match – you need to pick your approach first.

OK, I lied. You CAN mix and match, but it becomes awkward and heavy. React abstracts out HTML but if you use a custom component instead of the built-in HTML, React will work fine. All the React traits (diff-ing the two incremental iterations of virtual DOM, then applying the difference to the actual DOM) works for custom components as well. Therefore, it is fine to slip in a Web Component into a React app.

The opposite does not really work well – consuming a React component in Angular or Polymer is awkward and not worth it really. Not that the original direction is worth it necessarily – you need to load Web Components JavaScript AND React JavaScript.

Don’t eat the poison mushroom

One of the ways people loaded components in their pages in the past was by using good old iframes. Think what you will about them, but you could really lock the components down that way. If you load a component into your own DOM, you need to really trust it. Single origin policy and CORS are supposed to help you prevent a component leaking data from your page to the mother ship. Nevertheless, particularly when it comes to more complex components, it pays to know what they are doing, go through the source code etc. This is where open source really helps – don’t load a black box component into your DOM.

The shoes don’t match the belt

One of the most complex problems to deal with when consuming a Web component of any type is the design. When you are working in a native SDK, the look and feel of the component is defined by the underlying toolkit. All iOS components have the ‘right’ look and feel out of the box when you consume them. However, Web apps have their own themes that create combinatorial explosion. A reusable component needs to do one of the following things:

  1. Be configurable and themeable so that you can either set a few parameters to better blend it into your style guide, or provide entire template to really dial it in
  2. Be generic and inoffensive enough to be equidistant from any parent theme
  3. Be instantly recognizable (think youtube player) in a way that makes it OK that it has its own look and feel.

A very complex reusable component with a number of elements can be very hard to dial in visually by consumers. In corporations, this may reduce the number of themes you want to support. A large component may take it upon itself to support 2-3 supported and widely used design style guides. Then all you need to do is provide a single parameter (style guide name) to make the component use the right styles across the board.

What is going on inside?

Adding a component into your page is not only a matter of placement and visual style. Virtually all reusable components are interactive. A component can be self-contained (for example, all activity in a youtube player is confined to its bounding box), or expected to interact with the parent. If the component must interact with the parent, you need to consider the abstraction chain. Consider the simple countdown timer as a reusable component. Here is how the abstraction chain works:


The timer itself uses two low-level components – ‘Start’ and ‘Stop’ buttons. Inside the timer, the code will add click listeners for both buttons. The listeners will add semantic meaning to the buttons by doing things according to their role – starting and stopping the timer.

Finally, when this component is consumed by your page, only one listener is available – ‘onTimerCountdown()’. Users will interact with the timer, and when the timer counts down to 0, the listener you registered will be notified. You should be able to expect events at the right semantic level from all reusable components, from the simplest calendars to large complex components.

If a component can be made part of a larger document, two things you will care the most is serialization and dirty state. When users interact with the component and make a modification, you want to be told that the component is changed. This should trigger the dirty state of the parent. When the user clicks ‘Save’, you should be able to serialize the component and store this state in the larger document. Inversely, on bootstrap you should be able to pass the serialized state to the component to initialize itself.

Note that the actual technology used does not matter here – even the components embedded using iframes can use window.postMessage to send events up to the parent (and accept messages from the parent). While components living in your DOM will resize automatically, iframe-ed components will need to also send resizing events via window.postMessage to allow the parent to set the new size of the iframe.

The long tail

More complex reusable components don’t only have client-side presence. They have a need to call back to the server and fetch the data they need. You can configure such a component in two ways:

  1. You can fetch the data it requires for the component. In that case, the component is fully dependent on the container and it is container’s responsibility to perform all the XHR calls to fetch the data and pass it to the component. This approach may be best for pages that want full control of the network calls. As an added bonus, you can fit such a component into a data flow such as Flux, where some of the data may be coming from Web Socket driven server-side push, not just XHR request.
  2. You can proxy the requests that the component is performing. This approach is also acceptable because it allows the proxy to control which third-party servers are going to be whitelisted.
  3. You can configure CORS so that the component can make direct calls on its own. This needs to be done carefully to avoid the component siphoning data from the page to servers you don’t approve.

On all of these cases you may still want to be told about the events inside the web component using the component events as discussed above.

Frameworks are just the beginning

So there you go – all the problems you need to wrestle with when trying to reuse components in a larger project. Chances are the component is written in the ‘wrong’ framework, but trying to make the component load in your page is only the beginning. Fitting the component into the page visually, figuring out what is happening in it events-wise, and feeding it data from the server is the real battle. Unless you are trying to load a calendar widget, this is where you will spend most of your time.

© Dejan Glozic, 2016 for Tardy React Status Indicators

The White Rabbit, Wikimedia Commons
The White Rabbit, Wikimedia Commons

In one of my favorite movies “The Blues Brothers”, the wife of the trucker pit stop owner proudly exclaims that they play ‘both kinds of music – Country and Western’. Readers of my blog know I too tend to serve two kinds of articles. In think-pieces, I tend to pontificate on a topic that buzzes between my ears that week. In ‘hey, look at this code we wrote’, I share something we learned by doing. I am sure there is an audience for both kinds but if you are in the latter camp, this is your day.

I’m late, I’m really really late

The problem we had to solve recently was as follows: a React store contains a list of items. These items are rendered in a hierarchy of React components, with a ‘dummy’ React component rendering a card for each item. The item has a number of visual elements on it that can be easily rendered from the item’s properties, but it also contains a status indicator.


This status indicator was killing us: when we fetch the array of items via the XHR API call, we can get them fairly quickly. However, each item represents an entity that can be working or not working (and therefore have a green or red status indicator). Computing this status takes a while, needs to be done separately for each item, and cannot hold the main XHR call.

We solved this problem with the help of the module (at this point I need to remind you we use Node.js for our server side). You can apply the solution even if you are not using it, nor using Node at all – just use whatever Web Sockets library comes with your stack of choice.

The devil is in the pudding

OK, on to the details. What we do is as follows:

  • Whenever we want to update the status of the store, we issue an XHR.
  • The XHR fetches the items on the server via a downstream API call.
  • Before sending the response, the server endpoint initiates an array of async parallel calls to obtain status for each of the items. It sends the response to the client without waiting.
  • The client renders the cards, with the status rendered as a gray circle.
  •  When the status API calls return, the result is sent back via Web Socket channel as an event.
  • On the client, this event is converted into a Flux action, which in turn updates the Store, which updates the views.


Produce the code

OK, enough architecture, produce the code. In the interest of space conservation, I am going to assume you have rudimentary knowledge of both React and Flux. If not, Google away until React components, dispatcher, actions and stores are all concepts that make sense to you.

Our action creator is capable of dispatching two actions – fetching items and initiation service status update:

var AppDispatcher = require("./dispatcher");

var ServiceConstants = require("./action-constants");

var ServiceActions = {
	fetchItems: function(data) {
			actionType: ServiceConstants.FETCH_ITEMS,
			data: data
	serviceStatusUpdate: function(data) {
			actionType: ServiceConstants.SERVICE_STATUS_UPDATE,
			data: data

module.exports = ServiceActions;

OK, on to the ItemStore. It is a fairly standard affair. When asked to fetch items, it makes an XHR request to the endpoint in the Node.js server that returns server items. When status update arrives, it merges the provided status with the items and fires a change event again. This will drive the view to re-render:

// Modules

var Constants = require("./action-constants");
var EventEmitter = require("events").EventEmitter;

// Globals

var AppDispatcher = require("./dispatcher");
var XhrUtils = require("./xhr-util");

var EVENT_CHANGE = "items";
var items;
var emitter = new EventEmitter();
var listeningForSocket = false;

// Public Methods ------------------------------------------------------------->

var ItemStore = {

	getItems: function(type) {
		return items;

	emitChange: function(data) {
		emitter.emit(EVENT_CHANGE, data);

	addChangeListener: function(callback) {
		emitter.on(EVENT_CHANGE, callback);

	removeChangeListener: function(callback) {
		emitter.removeListener(EVENT_CHANGE, callback);


module.exports = ItemStore;

// Private Methods ------------------------------------------------------------>

function fetchItems(url) {
	if (!url) {
		console.warn("Cannot fetch items - no URL provided");

	// Fetch content
	XhrUtils.doXhr({url: url, json: true}, [200], function(err, result) {
		if (err) {
			console.warn("Error fetching assets from url: " + url);

		items = result;

function serviceStatusUpdate(data) {
	for (var i in data) {
		var s = data[i];

function _updateStatus(s) {
	for (var i in items) {
		var item = items[i];
		if ( {
			item.status = s.status;

// Dispatcher ------------------------------------------------------------>

// Register dispatcher callback
AppDispatcher.register(function(payload) {
	var action = payload.action;

	// Define what to do for certain actions
	switch (action.actionType) {
		case Constants.FETCH_ITEMS:
			return true;

	return true;


Finally, the view. It fires the action that fetches items upon mounting, and proceeds to render the initial state with the loading indicator. Once the items are ready, it re-renders itself with the server items in place. Finally, when the status update arrives, it just re-renders and lets React diff the DOM for changed properties. Which is why we love React in the first place:

var React = require('react');
var Router = require('react-router');
var ItemStore = require('../flex/item-store');
var ActionCreator = require('../flex/action-creator');
var ActionConstants = require('../flex/action-constants');

module.exports = React.createClass({
  getInitialState: function() {
    return { loading: true, items: [] };

  componentDidMount: function() {

  componentWillUnmount: function() {

  _handleAssetsChanged: function(type) {
    if(type === ActionConstants.STATE_ERROR) {
        error: "Error while loading servers"
    } else {
        loading: false,
        error: null,
        items: ItemStore.getItems() || []

  render: function render() {
    var loading;
    var items;
    var error;

    if (this.state.loading) {
      loading = (
         <div className="items-loading">
            <div className="LoadingSpinner-dark"></div>
    else {
      items = (
          { {
            var statusClass = "status-indicator";
            if (item.status==="active")
              statusClass+= " status-active";
            else if (item.status==="error")
              statusClass+= " status-error";
              statusClass+= " status-loading";
            return (
               <li className="service-card" key={}>
                  <div className="service-card-status">
                  <div className={statusClass}/></div>
                  <div className="service-card-name">{item.title}</div>
                  <div className="service-card-type">{item.dist}</div>
    if (this.state.error) {
      error = (
         <div className="error-box">{this.state.error}</div>
    return (
       <div id='list'>
             This list shows currently available servers and their status (
             <span className="text-loading">gray</span> - loading,&nbsp; 
             <span className="text-active">green</span> - active,&nbsp; 
             <span className="text-error">red</span> - error)

The status update portion involves module. When the status is being computed, it is being emitted via the Web Socket. Of course, for this example we are faking the delay, but you can imagine actual status API request being involved:

var io = require('')(server);

function emitStatus(items) {
  var status = [];
  for (var i in items) {
    var item = items[i];
    var s = {};
    s.status = (Math.random()<.5)?"active":"error"; =;
    setTimeout(function(status) {
      io.emit("status", [status]);
    }, (Math.floor((Math.random() * 900) + 100)), s);

This status even is being picked up on the client, producing the status update action that is dispatched, picked up by the store and eventually causing the view to update:

  var socket = io.connect(location.origin);
  socket.on('status', function (data) {

Conclusion and commentary

One of the many realization we have come to using React in production is that one can get very far with views alone. A top-level view can make XHR requests, then pass the data down to child components as props. Child components can capture low-level events, and translate them into events of higher level of abstraction via callbacks passed in as props.

We realized many simple pages can be entirely done this way. What we have noticed though is that when Web Sockets and server-side events are involved, Flux architecture really comes to its own and shows its potential. Mixing server side events with events produced by users interacting with the UI is much better done using Flux.

The problem we tried to solve is rendering the initial page. If the user stays on the page for a prolonged period of time, status changes of the items can continue, but the code above can handle it without any changes. Of course, if others can create or remove server items, new events and actions are needed to model that.

As usual, the entire example is available as a Node.js app on Github, and I have deployed it in Bluemix so that you can see the app running here: Just keep refreshing the browser to see the three-step rendering (page, then items, and finally status).

© Dejan Glozic, 2016

How to be a Good Software Architect

Wikimedia Commons
Wikimedia Commons

Ah, architecture – the broccoli of software industry. You know it is good for you, but you would much rather feed it to the dog under the table and move on to the ice cream. Ever since Joel Spolsky coined the phrase architecture astronauts, the role of architect has simultaneously been questioned by developers and line of business.

Developers suspect and occasionally despise software architects because they don’t like to be told what to do (they are wrong) and because architecture bestowed on them from the Ivory Tower is often confusing and verges on unimplementable (they are right).

Meanwhile, business suspects architecture because they don’t see how it solves its problems. Until you can ship iArchitecture 6s to the customers, business owners lack the direct link between architecture and bottom line. “I see what you are saying, Dejan, but in the last project a team of enthusiastic developers built a product on time without any architecture and it was just fine”. The fact that this product was an unmanageable steaming pile that nobody wants to maintain now didn’t help them draw the connection.

So what are we to do? Clearly, architecture is good for software the way vegetables are good for us. On the other hand, architecture created without any connection to the imperfect reality is useless. Selling architecture to executives is a fool’s errand. Is there a way out of this conundrum?

Of course there is, and is practiced as we speak in hundreds of companies. You just need to sneak the architecture in. Moms around the world know they cannot sell the broccoli to children by winning a rational argument involving the importance of nutrition. Instead, they either negotiate (first broccoli, then ice cream) or just sneak the veggies in. I don’t recommend you negotiate with business – like car salesmen, they are much more skilled in it than you are. Instead, just sneak the architecture in as part of normal development.

In order to be successful, you need to win over the two groups looking at you as if you are about to steal their lunch money.

The pragmatic architect

In order to win the developers over, you need to earn their trust. The easiest way to earn their trust is to become one of them. Architect of some kind is typically a role on the technical career path, on the way to Principal Engineer or Distinguished Engineer or Fellow. Notice the word “engineer” in the title. You have selected this path because you want to lead by example, to inspire, to make people follow you because they want to, not because of the org chart.

In order for developers to follow you enthusiastically, you need to get your hands dirty. Never stop coding! Coding is your anchor, your stabilizing force that prevents your architecture from morphing into unimplementable gibberish. The role of architecture is to give the code structure, a skeleton that prevents it from turning into an unmanageable goo. The role of practical architecture is to prevent the overgrowth of the said skeleton, before it turns a juggernaut into a WW I tank, with all of its speed and maneuverability.

It you are senior enough, it will be hard to code day in and day out. If you have the time for that, why are they paying you architect salary anyway? Spending too much time in the trenches may indicate that you are under pressure because people under pressure tend to slide back into a role they know best, and for you that is likely one of the coder. No, your role is more complex – you need to travel up and down the layers of abstraction constantly. You need to ascend to see the big picture that can be lost in the details, then descend to the details where the devil is, to make sure you kept all your pragmatic marbles.

What I like to do is go about two months ahead of the team. I anticipate the technical problems we are going to face, look for the possible solutions, write a prototype and blog about it (on this blog, in fact). By the time the team hits the problem in development, I have enough hands-on experience with it to speak about the problem without the imposter syndrome, a prototype, and a blog with code snippets (Blog as a Service). Works great, pays back tenfold.

Architecture in motion

On to the business. I already explained that architecture is not something you can directly sell to the business owners. People who love technology often derive pleasure just working with it. You learn a new language or a new framework and you are ecstatic. Executives look at you as you explain it and their eyes glaze over. But explain that the framework will cut operating costs in half due to the lower resource demands, or that it will cut time to market, and the room is yours.

Things are not very different in other walks of life. Actual architects may have a problem explaining the fact that load-bearing walls ensure building’s structural integrity. I bet they will have no problem explaining a pile of concrete, loss of revenue and a torrent of criminal negligence lawsuits. Architecture has to drive business impact, otherwise it is an intellectual exercise you should do on your own time (another Flux library, anyone?).

The problem with software is that it is hard to paint such a clear picture while it is being built. So don’t. Practice architecture in motion – infuse it into code while it is being built, don’t make it a separate phase. Blog about it, make diagrams for the teams so that everybody has a clear idea what you are building, write docs, define solid APIs, explain how everything fits together. Just don’t call it out as its own thing, because line of business does not care and cannot separate it from overall development cost.

Just say “we build good software”, and when the solidly architected product or service hums in production and does not implode from its own weight, say “because we are a great team”. Architecture is implied.

Lead it, own it

In order for all this to work, you cannot sit on the side occasionally bestowing architectural nuggets on the teams working on the project. Developers will politely listen to you but most of the leadership will come from the actual tech leads in the mud with the solders. Meanwhile, business will occasionally ask for your opinion but they will go to the tech leads for status. Don’t be the architect on the sidelines, be the technical leader using architecture as your secret weapon.

Then when you are in charge of the delivery, you can sneak the architecture in small doses into every feature, like parts of the wall Andy Dufresne sneaked into the yard in The Shawshank Redemption. The warden will never know.

© Dejan Glozic, 2016