神刀安全网

Message Queues, or how you can make processes talk to each other

When Processes need to talk

Applications and services often need to be scalable. The user base might grow from 10 to 10,000, or the incoming number of requests might increase by some orders of magnitude. One approach to scaling is to use faster computers. The other one is to use more computers and distribute workload among them.

Another scenario: Sometimes you need to keep separate concerns completely separated. In other words, every distinct functionality shall run as a separate OS process: A database, a Web server, a process that implements your business logic, and so forth.

Message Queues help connecting these processes.

Message Queues in a nutshell

Message Queue systems (or short, MQ systems) provide a means to pass arbitrary messages between processes. Usually they provide some sorts of topologies, or communication patterns, each of which fits a particular communication purpose.

Some MQ systems also provide a brokerage service, which acts as a well-known contact point for finding a particular service, as well as for routing messages from sender to receiver. This can be convenient especially for large distributed systems. Small systems may do well without a broker and thus benefit from higher throughput, as brokerless MQ systems can be much faster .

nanomsg – the minimalist MQ framework

In order to look into some examples of inter-process communications, we will use [nanomsg][NMG], a lightweight messaging system that has a couple of benefits:

  • nanomsg does not need a broker, nor a server infrastructure. Lightweight as can be.
  • nanomsg is dead-easy to understand and to use. No bloated concepts and over-complicated implementations.
  • /nanomsg provides a reasonable set of common communication topologies (also called “scalability protocols” in nanomsg terminology) out of the box. No need to reinvent the wheel over and over again.
  • A pure Go client is available (see below).
  • There is a ton of nanomsg implementations for other languages available, too. Want to connect your Go process with some other process written in C++, Java, Python, Rust, Ocaml, Erlang,…?Here you go!

Mangos: nanomsg in pure Go

Mangos is a Go implementation of nanomsg. It features an easy and intuitive API, as we will discover later when going through a first example.

How does nanomsg define communication patterns?

It does so by providing so-called “Scalability Protocols”. Each Scalability Protocol defines a particular communication pattern.

These protocols are currently defined:

  • Pair

Message Queues, or how you can make processes talk to each other

Motto: Scale your application by breaking it in two pieces.

  • Request-Reply

Message Queues, or how you can make processes talk to each other

Motto: Distribute workload among multiple stateless workers.

  • Publisher-Subscriber

Message Queues, or how you can make processes talk to each other

Motto: Broadcast messages to multiple destinations. Receivers can subscribe to specific topics.

  • Pipeline

Message Queues, or how you can make processes talk to each other

Motto: Collect output from multiple nodes of one processing step and distribute it among the nodes of the next processing step.

  • Survey

Message Queues, or how you can make processes talk to each other

Motto: Broadcast a survey and gather the responses. Wait for the replies for a certain time only.

  • Bus

Message Queues, or how you can make processes talk to each other

Motto: Broadcast messages from any node to all other nodes.

The basic building block of a Scalability Protocol is a nanomsg Socket.

What is a socket?

Sockets in general are messaging endpoints, usually defined by

  • a transport mechanism,
  • an IP address, and
  • a port number.

Example:

tcp://192.168.0.42:45890 

Message Queues, or how you can make processes talk to each other

nanomsg provides a couple of transport mechanisms:

  • In-process
  • Inter-process (but still on the same machine)
  • TCP
  • WebSockets

A process can provide a socket to other processes, as well as connect to a remote socket of another process.

When a process provides a socket to others, it “listens” on the socket (in Mangos lingo). When it connects to a remote socket, it “dials” this socket. (You can see this distinction later in our expample.)

Sockets in nanomsg have another interesting feature: They impelement a particular Scalability Protocol. That means, a nanomsg socket takes care of managing all internals of the protocol so that your code can more or less focus on sending and receiving messages.

A first example: PAIR

Let’s dive straight into our first example: A simple PAIR communication.

The PAIR protocol lets two processes send messages to each other. None of the two nodes has a particular role. Each one can send and receive messages to and from the other one.

Typical use case: To split up a large application into two smaller parts.

Outline

So what are we going to implement? In short, we want to have two processes running. One of them listens on a socket, the other one dials that socket. Once they are connected, they exchange a couple of messages.

You can get the full source code at github .

Use go get -d to ensure that the binary does not get installed into your $GOPATH/bin directory. Rather, use go build to generate a local binary that you then can run as ./messaging .

Installing Mangos

Installing Mangos is as easy as entering

go get -u github.com/go-mangos/mangos 

on the command line. To ensure everything has been installed correctly, you might want to run the tests. For this, cd to your $GOPATH and enter:

go test github.com/go-mangos/mangos/test 

If everything is ok, we can move forward to creating a sample PAIR implementation.

Implementing a PAIR example

转载本站任何文章请注明:转载至神刀安全网,谢谢神刀安全网 » Message Queues, or how you can make processes talk to each other

分享到:更多 ()

评论 抢沙发

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址