I would imagine that it’s a dream of every developer — to be able to write pure logic and to be able push it live with a touch of a button. No worries about frameworks, deployment procedures, provisioning, etc — just your business logic. Kind of like magic.
Of course, I had dreamed about something like this as well.
Most likely the problem itself is not even that interesting (or hard) to solve. I just need to solve it within a given time.
That’s where you’ll start thinking about scaffolding the project. Let’s have a look at how you create a modern project, shall we?
For the sake of simplicity, I’ll stick to full stack node.js application here. And I’ll assume we use the latest stable node.js.
Once again, for the sake of simplicity we won’t touch on topics like server-side rendering and all that more advanced stuff. It’s gonna be a simple REST API plus a standalone web UI for it.
We’ll start with a server-side and a basic REST API. Let’s see what questions you’ll have to answer before you create your project:
- Do I want to use babel or plain ES5 is good enough? Maybe typescript ? Or just flow with babel?
- Do I want to use all-in-one solution or minimal framework?
- If I want an all-in-one, which one? MEAN.js ? Meteor ? New shiny Nodal ?
- If it’s framework — which one do I want to take? Express ? Hapi ? Koa ? Something else?
- Which database (and library for it) I want to use? Mongo ? Redis ? PostgreSQL ?
- Do I need auth? Do I roll my own or use library? Passport.js ? Another one?
That’s far from all the questions you’ll have to answer, but it covers the very basics.Now let’s imagine you have picked your stack and it looks something like this (selection is arbitrary, things were picked mostly because that’s what I tend to use):
- We’ll use babel.js with ES2015 and stage 2 features — rest/spread and async/await are awesome! If you don’t know how to setup babel — have a look at this guide , it gives a pretty good explanation and overview of the amount of work it takes. Of course, you can always use something like generator-babel to ease the pain.
- Let’s say we’re building very simple REST API, so we don’t need a full-blown solution like Meteor and we’ll go with Express.js . If you don’t know how to setup Express.js project, head to their official docs — they are pretty good. Except that’s not all we need to do.
Since we’re building REST API, we’d need to handle POST, PUT, DELETE and others. To simplify those, you’ll need to plug in body-parser and method-override . And then there’s logging , CORS , websockets — you get the idea.
- For database we’ll pick MongoDB for no reason. And we’ll use mongoose as a library for interacting with it. Installing Mongo is not too hard if you are running modern system. Same goes for plugging in mongoose into your app. This part is somewhat the easiest one (if you exclude defining the schema you need).
- Finally, we’ll need to plug in passport.js authentication. Luckily for us, this is very easy too — most of the work will be tying your DB and schema to passport.
Finally! We’re done setting up a project for our REST API. Now we just need to write some business logic.
Oh, and we still need our web UI!
Let’s imagine you’ve spent a couple of days and finished your basic REST API. It’s nice, fully tested (you’ve also picked the testing framework yourself!), has fancy logger and does everything you ever wanted! Now it’s time to build a new fancy web UI for it.
Once again, let’s start by looking at what questions you’ll have to answer before you create your UI project:
- Which browsers do I target? Do I need to suffer and support IE8?
- Once again — do I want to use babel or plain ES5 is good enough? Or maybe typescript ? Or just flow with babel?
- Do I need module bundler? (you probably do) Which one? Webpack , jspm , rollup ?
- What framework or library for the UI? Angular ? React ? Polymer ? Cycle.js ?
- If it’s not all-in-one framework, what approach and libraries do I choose for the application architecture? Time-tested backbone ? Plain flux ? Redux ? Redux with sagas ? Relay ? DIY redux with RxJS ?
- Do I need CSS framework? Which one should I use? Bootstrap ? Foundation ? Material Design Lite ? Semantic UI ?
And once again — this list is far from all the questions you’ll have to answer, but it covers the very basics.Now let’s imagine you have picked your stack and it looks something like this (once again, those are arbitrary and were selected purely because that’s what I tend to use):
- Let’s say we have a customers with up-to-date browsers, so we only have to support IE 11 and later. Which is cool — less headaches related to compatibility. And flexbox support! (well, almost )
- Once again, we’ll go with babel since we already know how it works and how to set it up.
- Now — module bundlers. Let’s not spend too much time picking one and stop on webpack since it provides some nifty things others don’t (e.g. css modules ). If you don’t know how to set it up —have a look at this amazing guide by Pete Hunt. And be ready — it’s gonna take some time to figure everything out. But it’s totally worth it, just look at all those loaders !
- Okay, time to pick a framework. In case you are not too familiar with the area, here’s a decent overview of the current major players. We’ll just stop on React.js because I like it (totally valid reason here, yup). Good thing is — setting up basic React app is pretty straightforward . Most likely you’ll also need routing . And maybe hot reload , if you want to be slightly more productive.
- Here comes the fun part — architecture and underlying libraries, yay! (nay!) I’d actually encourage you to go ahead and try all of the available options if you have time — there’s a ton to learn from all of those. But, once again, for sake of simplicity let’s pick “classic” (can you call it classic already?) Flux and one of many existing implementations . Say Alt — easy to setup, pretty straightforward in use, has some nice features.
- As for CSS framework — let’s just stop on Bootstrap. Everybody likes Bootstrap and it plays nicely with React. Adding it to the project will only require including a .css file, which is pretty awesome.
Alright then! We’re done setting up a project for our web UI. All we need now some business logic — easy!
Probably, the most common approach is to learn one stack and use scaffolders and bootstrap projects to kickstart the development process. Specializing in one stack generally allows developers to have in-depth knowledge of all the parts and work at relatively fast pace. If the developers are familiar with CI/CD approaches, they might even set up auto-testing and auto-deployment. Problem is — that still takes a whole lot of time to do, even with help from scaffolders.
Another approach, which has started to gain traction in past few years, is to use existing Platform-as-a-Service solutions. While they do not eliminate the need to decide and learn a new stack, they generally do ease the development and deployment process quite a lot. Just look at all those shiny libraries for AWS! Still, most of the time you end up doing some parts (e.g. UI) of the system the old way — building and deploying them yourself.
I tried doing both. And they both work just fine, but obviously have different downsides. At some point, while building yet another platform for another data-driven application demo, I started thinking — shouldn’t there be a better way? Maybe somewhat opinionated, but faster and easier way? Can we build a generic platform that can help to solve those similar use cases? So we can use it later to build apps within hours, not days?
With these thoughts we set out to build a platform that will make building data-driven apps a breeze (well, at least we hope so).If you look at most of the applications, you can easily split them into three simple parts:
- Source — something that gets or produces the incoming data.
- Processor or several processors — something that reacts to that incoming data and either changes it in some way or just does something (like sending an email). Might not be present in some cases.
- Renderer — simple UI that shows you nice things.
If you doubt that you can describe most apps like this, here are few examples.
Example 1.Product sentiment analysis using tweets
Input is a function that gets data from Twitter Firehose. Processor is sentiment analyzer for incoming tweets. Renderer is basic table renderer that show nice summary of sentiments.
Example 2.Slack bot
Input is a function that gets messages from Slack. Processor is a function that, for example, searches Google using incoming message. Another processor is a function that sends response back to Slack. And optional renderer is a dashboard with all requests and responses.
Example 3.Chat app
Input is a function that gets messages from users. Processor is not even needed in a simplest case. Renderer is a UI that shows input box and messages from the input.
And there’s many more . You get the idea.
If I would have to formally describe this logic, it’d look like this:
The cool thing here is that all of these things — source, processors and render — are simple functions. This means we can create a platform that allow us to easily reuse them. Not so cool thing is — functions are synchronous. What do we do if we want to work with asynchronous things? (like HTTP requests)
Here’s where RxJS comes to the rescue. It essentially allows us to write pretty much the same code, but adds an ability to do asynchronous things. Here’s how the logic would look using RxJS:
I have cheated a bit, because we can’t really use exactly same functions as in the previous snippet. Source function must utilize Observer as said in Rx.Observable.create docs.
And processor must return Rx.Observable for flatMap instead of a plain value. But adding these tiny changes allows us to work asynchronously within a function — which is pretty huge.
And, I had also imposed an additional constraint on the rendering function — instead of just returning HTML, it should return the React component. This will allow to easily integrate it in a final UI, plus we’ll be able to reuse a ton of existing React components within its render function without any additional hacks.
Doesn’t all those things sound like an overcomplication? Yes and no.On the one hand, adding another library or layer of abstraction always increases a cognitive burden for developers. But in this case I think that the benefits outweigh that (somewhat minor) burden, primarily because we no longer need to think about all those questions mentioned above. Let’s take a closer look at all of those benefits.
I have already mentioned first two in the previous section — asynchronicity for processors and easy integration for UI components. Let’s start with asynchronicity.
Benefit 1.Asynchronous sources and processors with zero effort.
Being able to create functions that can handle data in an asynchronous manner out of the box without thinking about it too much a huge deal. If we’re talking about sources, we’d frequently want async sources that keep dispatching data as it comes in. For example — source of messages to Slack bot. Doing that with promises is near to impossible (because promise can only resolve once). Doing that with callbacks means it’ll be really hard (if not impossible) to use composition. Doing it with RxJS is trivial, here’s an example of Slack messages source:
This piece of code will keep dispatching Slack direction mentions until you stop it. That’s just 9 (nine!) lines of code.
Same goes for processors — we just return an RxJS observable first, then do the work and dispatch the values using the observable. And RxJS does all the work of delivering those values for us. You can find examples of processors in the tutorials in Exynize wiki .
The cool thing here is that we don’t lose synchronicity — RxJS observables are synchronous by default. How awesome is that?
Benefit 2.Distributed computing
Now, here’s the killer benefit of going with a functional approach — we can take all those functions, turn them into microservice and distribute them over several nodes. This effectively turns the code we just wrote into a full-fledged scalable microservice-based application. Which means that, if we need to, we can scale to handle some crazy amounts of data. If we have hardware.
I even wrote microwork.js — a library for simple creation of distributed scalable microservices in node.js based on RabbitMQ — to support this feature.
Exynize platform — bringing ideas together
Using these ideas we built Exynize — a platform that allows you to easily create data-driven applications from small, reusable, single-purpose building blocks — functions.
It’s still in an early alpha stage (things can break and the platform itself is not quite stable yet), but it’s live! You can try it yourself at alpha.exynize.com .
And if you are interested in looking under the hood (or want to roll your own copy of it) — it’s open source and you can find it on Github .
Try it out and let us know what you think!
Also, definitely do let us know if you break something!
转载本站任何文章请注明：转载至神刀安全网，谢谢神刀安全网 » Building a platform for serverless data-driven applications with Node.js and microservices