Building a platform for serverless data-driven applications with Node.js and microservices

I would imagine that it’s a dream of every developer — to be able to write pure logic and to be able push it live with a touch of a button. No worries about frameworks, deployment procedures, provisioning, etc — just your business logic. Kind of like magic.

Building a platform for serverless data-driven applications with Node.js and microservices

Of course, I had dreamed about something like this as well.

Even though I do enjoy tinkering with docker, checking out new shiny libraries and frameworks (hey, I’m primarily writing javascript after all!) — sometimes I just want to quickly build a solution for a very specific problem.

Most likely the problem itself is not even that interesting (or hard) to solve. I just need to solve it within a given time.

That’s where you’ll start thinking about scaffolding the project. Let’s have a look at how you create a modern project, shall we?

For the sake of simplicity, I’ll stick to full stack node.js application here. And I’ll assume we use the latest stable node.js.

Once again, for the sake of simplicity we won’t touch on topics like server-side rendering and all that more advanced stuff. It’s gonna be a simple REST API plus a standalone web UI for it.


We’ll start with a server-side and a basic REST API. Let’s see what questions you’ll have to answer before you create your project:

  • Do I want to use babel or plain ES5 is good enough? Maybe typescript ? Or just flow with babel?
  • Do I want to use all-in-one solution or minimal framework?
  • If I want an all-in-one, which one? MEAN.js ? Meteor ? New shiny Nodal ?
  • If it’s framework — which one do I want to take? Express ? Hapi ? Koa ? Something else?
  • Which database (and library for it) I want to use? Mongo ? Redis ? PostgreSQL ?
  • Do I need auth? Do I roll my own or use library? Passport.js ? Another one?

That’s far from all the questions you’ll have to answer, but it covers the very basics.Now let’s imagine you have picked your stack and it looks something like this (selection is arbitrary, things were picked mostly because that’s what I tend to use):

  • We’ll use babel.js with ES2015 and stage 2 features — rest/spread and async/await are awesome! If you don’t know how to setup babel — have a look at this guide , it gives a pretty good explanation and overview of the amount of work it takes. Of course, you can always use something like generator-babel to ease the pain.
  • Let’s say we’re building very simple REST API, so we don’t need a full-blown solution like Meteor and we’ll go with Express.js . If you don’t know how to setup Express.js project, head to their official docs — they are pretty good. Except that’s not all we need to do.
    Since we’re building REST API, we’d need to handle POST, PUT, DELETE and others. To simplify those, you’ll need to plug in body-parser and method-override . And then there’s logging , CORS , websockets — you get the idea.
  • For database we’ll pick MongoDB for no reason. And we’ll use mongoose as a library for interacting with it. Installing Mongo is not too hard if you are running modern system. Same goes for plugging in mongoose into your app. This part is somewhat the easiest one (if you exclude defining the schema you need).
  • Finally, we’ll need to plug in passport.js authentication. Luckily for us, this is very easy too — most of the work will be tying your DB and schema to passport.

Finally! We’re done setting up a project for our REST API. Now we just need to write some business logic. 

Oh, and we still need our web UI!

Web UI

Let’s imagine you’ve spent a couple of days and finished your basic REST API. It’s nice, fully tested (you’ve also picked the testing framework yourself!), has fancy logger and does everything you ever wanted! Now it’s time to build a new fancy web UI for it.

Once again, let’s start by looking at what questions you’ll have to answer before you create your UI project:

And once again — this list is far from all the questions you’ll have to answer, but it covers the very basics.Now let’s imagine you have picked your stack and it looks something like this (once again, those are arbitrary and were selected purely because that’s what I tend to use):

  • Let’s say we have a customers with up-to-date browsers, so we only have to support IE 11 and later. Which is cool — less headaches related to compatibility. And flexbox support! (well, almost )
  • Once again, we’ll go with babel since we already know how it works and how to set it up.
  • Now — module bundlers. Let’s not spend too much time picking one and stop on webpack since it provides some nifty things others don’t (e.g. css modules ). If you don’t know how to set it up —have a look at this amazing guide by Pete Hunt. And be ready — it’s gonna take some time to figure everything out. But it’s totally worth it, just look at all those loaders !
  • Okay, time to pick a framework. In case you are not too familiar with the area, here’s a decent overview of the current major players. We’ll just stop on React.js because I like it (totally valid reason here, yup). Good thing is — setting up basic React app is pretty straightforward . Most likely you’ll also need routing . And maybe hot reload , if you want to be slightly more productive.
  • Here comes the fun part — architecture and underlying libraries, yay! (nay!) I’d actually encourage you to go ahead and try all of the available options if you have time — there’s a ton to learn from all of those. But, once again, for sake of simplicity let’s pick “classic” (can you call it classic already?) Flux and one of many existing implementations . Say Alt — easy to setup, pretty straightforward in use, has some nice features.
  • As for CSS framework — let’s just stop on Bootstrap. Everybody likes Bootstrap and it plays nicely with React. Adding it to the project will only require including a .css file, which is pretty awesome.

Alright then! We’re done setting up a project for our web UI. All we need now some business logic — easy!

So, yeah..

I can easily see why some people arenot exactly happy with the javascript ecosystem at the moment. Some even call it “javascript fatigue”. A counterargument can be made that for developers constant learning is something everyone should be doing. While this is definitely true, unfortunately, real world does not always gives you enough space to roll at your own pace. More often than not, you need to deliver a working product within weeks if not days. So, what are possible solutions for this problem?

Probably, the most common approach is to learn one stack and use scaffolders and bootstrap projects to kickstart the development process. Specializing in one stack generally allows developers to have in-depth knowledge of all the parts and work at relatively fast pace. If the developers are familiar with CI/CD approaches, they might even set up auto-testing and auto-deployment. Problem is — that still takes a whole lot of time to do, even with help from scaffolders.

Another approach, which has started to gain traction in past few years, is to use existing Platform-as-a-Service solutions. While they do not eliminate the need to decide and learn a new stack, they generally do ease the development and deployment process quite a lot. Just look at all those shiny libraries for AWS! Still, most of the time you end up doing some parts (e.g. UI) of the system the old way — building and deploying them yourself.

I tried doing both. And they both work just fine, but obviously have different downsides. At some point, while building yet another platform for another data-driven application demo, I started thinking — shouldn’t there be a better way? Maybe somewhat opinionated, but faster and easier way? Can we build a generic platform that can help to solve those similar use cases? So we can use it later to build apps within hours, not days?

Core idea

With these thoughts we set out to build a platform that will make building data-driven apps a breeze (well, at least we hope so).If you look at most of the applications, you can easily split them into three simple parts:

  1. Source — something that gets or produces the incoming data.
  2. Processor or several processors — something that reacts to that incoming data and either changes it in some way or just does something (like sending an email). Might not be present in some cases.
  3. Renderer — simple UI that shows you nice things.

If you doubt that you can describe most apps like this, here are few examples.

Example 1.Product sentiment analysis using tweets

Input is a function that gets data from Twitter Firehose. Processor is sentiment analyzer for incoming tweets. Renderer is basic table renderer that show nice summary of sentiments.

Example 2.Slack bot

Input is a function that gets messages from Slack. Processor is a function that, for example, searches Google using incoming message. Another processor is a function that sends response back to Slack. And optional renderer is a dashboard with all requests and responses.

Example 3.Chat app

Input is a function that gets messages from users. Processor is not even needed in a simplest case. Renderer is a UI that shows input box and messages from the input.

And there’s many more . You get the idea. 

If I would have to formally describe this logic, it’d look like this:

Simplified representation of majority of data-driven applications

The cool thing here is that all of these things — source, processors and render — are simple functions. This means we can create a platform that allow us to easily reuse them. Not so cool thing is — functions are synchronous. What do we do if we want to work with asynchronous things? (like HTTP requests)

Here’s where RxJS comes to the rescue. It essentially allows us to write pretty much the same code, but adds an ability to do asynchronous things. Here’s how the logic would look using RxJS:

Data-driven application logic with async capabilities using RxJS

I have cheated a bit, because we can’t really use exactly same functions as in the previous snippet. Source function must utilize Observer as said in Rx.Observable.create docs. 

And processor must return Rx.Observable for flatMap instead of a plain value. But adding these tiny changes allows us to work asynchronously within a function — which is pretty huge. 

And, I had also imposed an additional constraint on the rendering function — instead of just returning HTML, it should return the React component. This will allow to easily integrate it in a final UI, plus we’ll be able to reuse a ton of existing React components within its render function without any additional hacks.

Approach benefits

Doesn’t all those things sound like an overcomplication? Yes and no.On the one hand, adding another library or layer of abstraction always increases a cognitive burden for developers. But in this case I think that the benefits outweigh that (somewhat minor) burden, primarily because we no longer need to think about all those questions mentioned above. Let’s take a closer look at all of those benefits.

I have already mentioned first two in the previous section — asynchronicity for processors and easy integration for UI components. Let’s start with asynchronicity.

Benefit 1.Asynchronous sources and processors with zero effort.

Being able to create functions that can handle data in an asynchronous manner out of the box without thinking about it too much a huge deal. If we’re talking about sources, we’d frequently want async sources that keep dispatching data as it comes in. For example — source of messages to Slack bot. Doing that with promises is near to impossible (because promise can only resolve once). Doing that with callbacks means it’ll be really hard (if not impossible) to use composition. Doing it with RxJS is trivial, here’s an example of Slack messages source:

Simple source for Slack direct mentions

This piece of code will keep dispatching Slack direction mentions until you stop it. That’s just 9 (nine!) lines of code.

Same goes for processors — we just return an RxJS observable first, then do the work and dispatch the values using the observable. And RxJS does all the work of delivering those values for us. You can find examples of processors in the tutorials in Exynize wiki .

The cool thing here is that we don’t lose synchronicity — RxJS observables are synchronous by default. How awesome is that?

Benefit 2.Distributed computing

Now, here’s the killer benefit of going with a functional approach — we can take all those functions, turn them into microservice and distribute them over several nodes. This effectively turns the code we just wrote into a full-fledged scalable microservice-based application. Which means that, if we need to, we can scale to handle some crazy amounts of data. If we have hardware.

I even wrote microwork.js — a library for simple creation of distributed scalable microservices in node.js based on RabbitMQ — to support this feature.

And we can still write our functions thinking (more or less) about them in a synchronous way (well, in RxJS observable way). We can still test the final application synchronously. We can even export the resulting application to a simple single javascript file if we want to.

Exynize platform — bringing ideas together

Using these ideas we built Exynize — a platform that allows you to easily create data-driven applications from small, reusable, single-purpose building blocks — functions.

It is, as I already mentioned, a heavily opinionated take on development of data-driven apps — after all, you will have to use javascript in general, RxJS for async things and React.js for UIs. Other than that — you are free to build whatever you want. You can follow one of our tutorials to build sentiment analysis for Twitter , heat-map of world news sentiments or even a simple Slack bot — give it a try.

It’s still in an early alpha stage (things can break and the platform itself is not quite stable yet), but it’s live! You can try it yourself at alpha.exynize.com .

And if you are interested in looking under the hood (or want to roll your own copy of it) — it’s open source and you can find it on Github .

Try it out and let us know what you think!

Also, definitely do let us know if you break something! :)

转载本站任何文章请注明:转载至神刀安全网,谢谢神刀安全网 » Building a platform for serverless data-driven applications with Node.js and microservices

分享到:更多 ()

评论 抢沙发

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址