Vive la Révolution App

Source: Wikimedia Commons

This post is a based on a presentation I made on a dare – something a former colleague proposed with only a title and a description, and it was up to me as the replacement to provide the actual content. It sort of reminds me of a debate club, where you are told that the topic is ‘App Revolution’, and you have 20 minutes to argue the ‘Pro’ position. What follows is my attempt to do it justice. Have fun (and mercy).

When we are confronted with the topic of revolutions, most of my North American friends immediately conjure up the sound of Yankee Doodle and the picture of George Washington crossing the Delaware River (I saw it last year in The Met – boy, is that painting big!). Being of European descent, my thoughts give preference to the French Revolution. It has essentially given us the modern European society, with milestone documents such as ‘Declaration of the Rights of Men and Citizen’ shown above. It has also given us the guillotine, which is sad but as Jacques Mallet du Pen famously quipped, all revolutions devour their children. What can you do – it’s a revolution, so16,594 people are bound to lose their heads, give or take.

One of the indispensable aspects of revolutions are slogans, something you can easily chant at the large group gatherings. Something catchy, such as ‘Freedom, Equality and Fraternity’ in the case of the French Revolution. Or as Blackadder interpreted it ‘Freedom, Equality and fewer fat bastards eating all the pie’.

As you correctly noticed, these slogans often call for three things. If that is true, and we are indeed witnessing an App Revolution, what would our slogan be? What three things would we want from our oppressors?

We are fighting for the freedom and abundance of data, infrastructure and architecture.

– Oppressed developers everywhere.

Note that when I say ‘freedom’, I don’t necessarily mean ‘completely free’. We know we will need to pay for some of it, hence the word ‘abundance’. While food in Western society is not exactly free, it is definitely abundant. You can go into any supermarket and leave with a whole rotisserie chicken for a few dollars. During the French Revolution, only the aforementioned fat bastards could afford it. That’s progress.

Hence, let me try to explain why we are fighting for these three things.

Freedom of Data

You have probably heard the phrase that we are living in an age of ‘API Economy’. What does that actually mean? In the past, data was a by-product of people using your application. Over time, your app’s database would fill up with data. The thinking was that the app is the product, and data is just internal by-product, a consequence of app usage. More recently, data started to take off as something that can be as important, or in some cases the only product you provide.

While in the past tacking on an API to your app would be an afterthought, something you may consider for partner or customer integrations, most modern systems are now built by first building the API for the data, then building up various clients that consume it. Your own clients are just ‘reference implementation’ of hopefully many other consumers of your APIs that will follow.

Source: IBM

Even music is going API these days. Sound engineers are now expected to provide stems of mastered music (drums, bass, guitars, keyboards, vox) so that remixers can easily provide derivative value without the hassle of sampling fully mixed songs (the audio equivalent of screen-scraping). What are stems but audio APIs?

Why is this important to us? Because when you open up your APIs, you become a platform, and platforms foster app eco-systems, with apps creating new value in many unexpected ways. Today, the most coveted place for any company is not to create a consumer product, but to create a platform that offers data and API, and creates a flourishing eco-system of apps built to take advantage of it. API discovery is now in the vogue, catalogs are sprouting, and all you need is to subscribe, obtain the authentication key and start building your innovative abstraction on top of it, or by combining multiple data sources in an innovative way. You can be data mining, providing innovative interfaces, analytics, or integrations with other systems.

If you are building a mobile app, all you need is a laptop and a phone to test your app. However, if you need anything in the back end you need to build a companion server-side app, which leads us to…

Freedom of Infrastructure

When I was a child, my parents bought me a Meccano kit. In those days, giving a child a box full of tiny sharp metal objects was consider totally cool. I quickly built all the possible toys based on the accompanied booklet, but sneaky bastards from Meccano also put a picture of a crane on the box that would require something like 10 sets to build. Since then, I developed this realization that I need to find a discipline in which I will not be limited by a box of finite number of parts.

Source: Meccano Beam Engine, Liskeard Museum

That’s why I chose software engineering – it is rare you will run out of files, or classes, or functions or variables the way you can run out of Meccano panels or tiny nuts and bolts.

However, once you venture into Web development, you hit the infrastructure version of Meccano. Your database, your server, your front end proxy all need to be hosted on physical boxes, and Mordac The Preventer from Information Services can make your life miserable in a hurry.

This is why Cloud is so important for our revolution. Regardless of where you fall on your ‘as a Service’ comfort level, you can use an IaaS or PaaS or SaaS to stand up your apps in minutes. Assuming you have found free or abundant source of data, your app can now be running and stay running without the need to worry about the messy sysadmin details or melted boards.

It does not end with just seeing your app running, either – you can jump into the third freedom that is the final cornerstone of our revolution.

Freedom of Architecture

In the dark ages of IT, it used to be that architecture was for the rich, and The Big Ball of Mud was for the rest of us. While you instinctively know that you should not be cashing those objects in memory, who is going to stand up, maintain and cluster Redis for it. You know that a message broker would be a real answer for your particular problem, but don’t have the stomach to stand up and administer RabbitMQ, or any of the popular alternatives. There is no accident that the famous Martin Fowler’s book from 2002 is called Patterns of Enterprise Application Architecture. At that time, only an enterprise could afford to provision and maintain all the boxes that such an architecture requires.

Source: Dejan Glozic

That same Martin Fowler not talks about Polyglot Persistence – the approach where apps in a distributed system choose different types of databases that perfectly suit their diverse needs, instead of underpowered MySql for everything. And he is not using the word ‘ enterprise’ this time, fully aware that a nerd hacking away on his Mac in Starbucks can provision such a system in minutes. App revolution indeed.

All together now

When we put our three demands together, great things can happen. To illustrate how far we have come, consider the system that I made the attendees of an IBM Interconnect 2015 lab build over the course of 2 hours:

Source: Dejan Glozic

This system is just a toy, designed to teach modern micro-service architecture, and yet it would require that we stand up several servers, install and configure a ton of software, and build our own user management system:

  1. It uses Facebook for delegated authentication and to tap into Facebook’s data. No need to stand up anything, just register as a Facebook developer, obtain your client ID and secret and off you go.
  2. It deploys complex infrastructure (two Node.js app servers, a proxy, a data cache) to Bluemix PaaS within a matter of minutes, all using just a Web browser. In a pinch you could do it on a bus using your iPad, while also debating someone totally wrong on the Internet.
  3. It uses serious architecture (OAuth2 provider, Nginx proxy, Node.js micro-services, session sharing via Redis store) that was unheard of for non-institutional developers in the past.

Platforms everywhere

Of course, the notion of a platform is not limited to the Web. In fact, some of you may have initially thought the article is about mobile apps. Phones are huge app ecosystems, and so are the upcoming wearable platforms, of which iWatch is just the latest example.

Venturing further away from the classic Web apps, cars are now becoming rife with platforms unleashing the app revolution of sorts. Consider Apple’s CarPlay that Scott Rich wrote about in O’Reilly Radar – a platform for apps in your car, tapping at the latent and closed data world and opening it up as a new app eco system. It is a different context but the model seems to be the same: create a platform, open up the data through APIs, and unleash the inventions of app revolutionaries hunched over their laptops around the world.

Means of production

In the past, the control of data, infrastructure and architecture were limiting factors for the masses of developers around the world. Creativity and ideas are dispersed far more equitably than the control over resources would make you believe. At least in the area of software development, the true app revolution is in removing these control points and allowing platforms and eco systems to let the best ideas bubble up.

Whether you are a guy at a reclaimed wood desk overlooking San Francisco’s Mission district, or a girl in Africa at a reclaimed computer in a school built by a humanitarian mission, we are approaching the time when we will only be limited by our creativity, and by our ability to dream and build great apps. And that, my fellow developers, is worth fighting for.

© Dejan Glozic, 2015

Meanwhile, in Nodeland…

Hieronymus Bosch, "The Garden of Earthly Delights", between 1480 and 1505.
Hieronymus Bosch, “The Garden of Earthly Delights”, between 1480 and 1505.

In India, we don’t call it “Indian food’. We just call it “food”.

Rajesh Koothrappali, “The Big Bang Theory”

It was always hard for me to make choices. In the archetypical classification of ‘satisficers’ and ‘maximizers’, I definitely fall into the latter camp. I research ad nauseum, read reviews, measure carefully, take everything into account, and then finally make a move, mentally exhausted. Then I burst into cold sweat of buyer’s remorse, or read a less than glowing review of the product I just purchased, and my happiness is dimished. It sucks to be me.

It appears it can be passed on as well. My 14-year-old daughter recently went to the mall to buy clothes (alone, a rite of passage of sorts). She returned empty-handed – ‘there was nothing nice’. I think we confirmed paternity right then and there.

Of course, this affliction carries over into my professional life. Readers who follow me for a while can attest my hemming and hawing about Node.js – should I stay or should I go, what if it is not ready, what if it turns out to be just a fad, what if I read a bad review after we port everything.

At least in my professional choices I appear to be more decisive than when looking at Banana Republic sweaters. After we jumped into Node.js waters in January this year, there was no turning back. It is almost April now and my mid-term report is due.

One of the beneficial things of ending the agony and giving Node a chance is that you stop worrying about HOW things are done and focus on GETTING them done. And once you do it, like Rajesh above, Node.js recedes into the background and the actual problems you are trying to solve become your focus (you are not solving problems in Node, you are just, you know, solving problems). Xenofobic people who get to know ‘the other’ are often surprised that under the surface of difference there is a sea of sameness – people around the world have similar worries, hopes, fears and joys. In Node, you still have to worry about security, i18n, performance, code structure, user experience, and Node is not the only player involved in this – just one tool in your toolbox.

One of the most important roles in converting an organization of any size to Node.js is that of influencers. People like myself who push for it, look for opportunities to get ‘camel’s nose under the tent’ (to borrow from Bill Scott), find a crevice where Node can squeeze and then expand like ice in the road potholes (I should know, I keep avoiding them this cold March in Toronto all the time). It is a startup of sorts, an inverted pyramid that can easily fall sideways if the founder flinches or loses hope and vision.

On the other hand, it can remind you of the locked potential of the people. There is a dark lake of passion in all of us, often trapped under a layer of dealing with day to day stress and ‘things that need to be done’. It is heart-warming to see that when you succeed in winning people over to the Node.js side, the passion bursts out and they jump in with enthusiasm and vigor that is great to experience.

Example: a member of the JazzHub team heard my pitch for Node.js and started experimenting with tutorials written in markdown. Like everything else, these tutorials are currently served using JSPs and client-side JavaScript. He wrote a small Node app that takes the tutorials, converts them to HTML using a nice little module and renders them using Dust.js. He did it in one afternoon and proudly demoed it the next time we met – it is one page of JavaScript – you don’t even need to scroll to see the entire code. Needless to say, he was hooked.

One of the most important jobs of an influencer is to prevent failure on stupid technicalities. People ready to yell ‘I told you Node was not ready’ are everywhere. Node is awesome,  but we all know that a single uncaught exception can crash the process. A JEE container can throw exceptions like a leaky barge, filling up the logs, but it will stay up. In Node, it is your responsibility to prop Node up, a Weekend at Bernies’s of sorts. Judging by the frequency of questions about this behaviour, it is hard to stomach by people moving over from the other platforms.

JEE is in fact not that different. When an uncaught exception is thrown in a simple Java program with ‘main’, it exits. When the same happens in a JEE container, the runnable running in a thread exits. The thread is reset, returned to the thread pool, and the next request is passed on to another thread. In a production system with Node.js, it is your responsibility to replicate this behaviour.

It is somewhat easier to end up with an uncaught exception in Node.js than it is in Java. Due to the asynchronous nature of Node, trying to surround a piece of code in a try/catch block is often futile. Exceptions can and will escape. This is such a tricky problem that it warranted the high priests of Node to huddle recently and release a detailed and super useful document that anybody trying to write production quality Node.js system should read and internalize.

It goes without saying then that you will never run ‘node myApp.js’ except during development. In production, you want to spread out to cover all the cores of the machine or VM you are running on. You want to restart the process when it exits (and it will, repeatedly). And finally, it would help to monitor CPU, memory and throughput of each running process. We are currently looking at PM2 for this, but as always with Node, there are other ways of addressing this need.

There are other things we discovered while getting serious with Node. For example:

  1. WebSockets – one of the first things you will try to do after getting the CRUD down is server push – after all, Data Intensive Real Time (DIRT) is what initially catapulted Node into our consciousness. After you get the code working on your machine, inspect the entire path to the browser on your system – proxies are surprisingly cranky when it comes to support for Web Sockets. NGINX didn’t have it until version 1.3. If you are using Apache, note that it will not let the Web Socket connections through unless you install and configure mod_proxy_wstunnel. Of course, this is not Node-specific, but chances are you are most likely to hit it with Node because of its affinity to server push.
  2. DevOps – make them your friend. If you thought Continous Integration and Continous Deployment was a neat idea, you will find it absolutely indispensable once you start getting real with Node. A well established pattern here is a system composed of micro-services, and it is not uncommon to juggle dozens upon dozens of separate Node apps performing a role in the system. You will go mad trying to manage micro-services by hand. Of course, once you get your DevOps going, the ability to deliver new features into a micro-service with surgical precision will bring no end of joy, particularly if you are replacing a JEE pig that takes minutes to restart. Zero downtime is a reachable goal in a micro-service based system comprised of Node processes. In IBM Rational, we use Urban Code, which is natural to us since we bought them recently. Do what you need to do to go automatic.
  3. Storage – Node will make you re-evaluate the rest of your system. When I started this blog, I was characteristically wishy-washy about databases. With Node.js, we noticed that we naturally lean towards JSON-based NoSQL databases. Many people use MongoDB with success due to the smooth transition from SQL, but if you don’t like its scalability characteristics or your lawyers don’t like its AGPL license, CouchDB is an easy choice. IBM recently bought Clodant (hosted CouchDB fork with Apache Lucene tossed in) so it is hard for us to say ‘no’ to a free, Cloud-based and professionally managed NoSQL JSON DB. Again, YMMV – use what works for you, but give JSON-based NoSQL databases a try – going JavaScript all the way down is liberating.
  4. Authentication and Single Sign-On – if there is anything that will remind you of the Ford Model T (you could order it in any colour as long as it was black), it will be authentication and SSO in your system. Nothing will uncover unnecessary complexity of the authentication schemes used in the system like trying to move it to something that is not JEE-based. The complexity is normally hidden in the JEE filters, and is uncovered as soon as you try to replicate it in Node.js. It is technology agnostic as long as you use Java. The good thing is that with a sympathetic team on the other end, this serves as a catalyst to finally simplify and clean up the protocol or switch to something saner.
  5. Legal – if you are in an enterprise of any size, you will not be allowed to just use whatever you find on the Internet (even startups find software promiscuity to have side-effects as soon as they grow enough to attract legal attention). Our legal department probably hates us – the amount of Node.js modules they need to vet is staggering, but the mountain of requests is tapering down. For example, the Markdown app I mentioned earlier in the post needed to vet only the markdown module – all the other modules were already encountered by the apps before it. At this rate, the flow will slow down to a trickle soon.

So there you go – three months after our foray into Nodeland, we are lining up several new Node micro-services, the sky didn’t fall, we are learning new things every day and worry more about what we are building than how we are building it. Every once in a while we are forced to solve a problem for the first time. We agree on the best practice together, write it down for others coming after us, and move on. Where we can, we rely on the efforts of people before us, allowing suites such as Kraken.js to let us focus on what is really unique about our code, rather than the boilerplate. Our lawyers hate us, but if ‘tried and true’ was the only criterium when choosing our tools and languages, we would be still writing everything in Cobol.

If only I was this decisive choosing my last Home Theatre receiver.

© Dejan Glozic, 2014