Let’s build our global infrastructure
The EC2 system makes it simple to fire up a global presence, so I did just that. I launched 3 m4.xlarge instances, running Ubuntu 14.04 LTS — one in the USA (Oregon), one in Europe (Dublin), and, finally, one in Australia (Sydney).
I ended up with the following hosts:
|AWS Region||Public Hostname||Public IP||Private IP|
|Europe (Dublin)||None assigned||126.96.36.199||172.31.40.213|
For the purposes of this experiment, I made the Security Groups wide open, so there would be no problems with network access. Of course, you wouldn’t do this in a production environment.
Check the network
To get an idea of network latency, I ran some ping tests.
From Oregon to Dublin
$ ping 188.8.131.52 PING 184.108.40.206 (220.127.116.11) 56(84) bytesofdata. 64 bytesfrom 18.104.22.168: icmp_seq=1 ttl=48 time=135 ms
From Oregon to Sydney
$ ping 22.214.171.124 PING 126.96.36.199 (188.8.131.52) 56(84) bytesofdata. 64 bytesfrom 184.108.40.206: icmp_seq=1 ttl=51 time=161 ms
From Sydney to Dublin
$ ping 220.127.116.11 PING 18.104.22.168 (22.214.171.124) 56(84) bytesofdata. 64 bytesfrom 126.96.36.199: icmp_seq=1 ttl=47 time=305 ms
Next, after ssh’ing in to each node, let’s install rqlite v3.1.0 on each node like so:
$ bash < <(curlhttps://raw.githubusercontent.com/otoolep/global-rqlite/master/install.sh)
The source of this script is available here .
Start the leader
Let’s choose to fire up the leader in Oregon, using the EC2 metadata system to retrieve the IP addresses. So I ran the following commands on the EC2 instance in Oregon.
$ PUBLIC_IP=`curl -s http://169.254.169.254/latest/meta-$ data/public-ipv4` $ LOCAL_IP=`curl -s http://169.254.169.254/latest/meta-data/local-ipv4` $ ./rqlited -http $LOCAL_IP:4001 -httpadv $PUBLIC_IP:4001 -raft $LOCAL_IP:4002 -raftadv $PUBLIC_IP:4002 $HOME/node
We now have a single rqlite node, with a real SQLite database underneath it. v3.1.0 includes the important options httpadv and raftadv , which allows the node to advertise an IP address other than the address it is bound to — in this case it advertises the public IP address of the EC2 instance, allowing other nodes to contact it from outside the AWS US Oregon region.
This node will accept commands, so let’s create a table and insert a record, using the rqlite CLI .
$ ./rqlite -H ec2-52-36-0-40.us-west-2.compute.amazonaws.com ec2-52-36-0-40.us-west-2.compute.amazonaws.com:4001> CREATETABLEfoo (idINTEGER NOT NULL PRIMARYKEY, nameTEXT) 0 rowaffected (0.000336 sec) ec2-52-36-0-40.us-west-2.compute.amazonaws.com:4001> INSERTINTOfoo(name) VALUES("fiona") 1 rowaffected (0.000074 sec) ec2-52-36-0-40.us-west-2.compute.amazonaws.com:4001> SELECT * FROMfoo +----+-------+ | id | name | +----+-------+ | 1 | fiona | +----+-------+
The rqlite CLI provides a very similar experience to the SQLite CLI. But for full functionality (such as bulk inserts and transaction control), use the HTTP API directly.
Create a cluster
There isn’t much point to a single-node cluster, so let’s add two more nodes. To do that I run the next commands on each of the two remaining EC2 instances:
$ PUBLIC_IP=`curl -s http://169.254.169.254/latest/meta-data/public-ipv4` $ LOCAL_IP=`curl -s http://169.254.169.254/latest/meta-data/local-ipv4` $ ./rqlited -http $LOCAL_IP:4001 -httpadv $PUBLIC_IP:4001 -raft $LOCAL_IP:4002 -raftadv $PUBLIC_IP:4002 -joinhttp://ec2-52-36-0-40.us-west-2.compute.amazonaws.com:4001 $HOME/node
With these simple steps, we’ve now got a globally replicated relational database. The new table and row have been replicated to the two new nodes. And, of course, every future change made on the leader database will be replicated synchronously to the other 2 nodes, placed out of harms way, thousands of miles from the leader.
This 3-node cluster can tolerate the failure of a node, and if that node is the leader, a new leader will be elected within a couple of seconds. If the cluster was 5 nodes in size, it could tolerate the failure of two nodes.
rqlite exposes a HTTP API, which the CLI uses. This API will also redirect clients to the leader node, if the node contacted is not the leader.
Is rqlite practical? Probably not on a cluster with such high latency between the nodes. In this set up I managed to insert about 2 rows a second, though if transactions and bulk updates are used, the effective rate is much higher.
However, within, say, a single EC2 Region, the performance will be much greater. And by running an rqlite cluster within the same data-center, with each node on a different rack for reliability, one should see 100s of insertions a second. And all with very easy deployment and operation.
Absolutely. rqlite is a serious program, and it shows the remarkable power available to individual developers today. rqlite builds on a world-class database , a proven distributed consensus implementation , and a worldwide public cloud, to provide a globally replicated relational database, with SQLite at its core.