One of the major reasons for the website upgrade the Varnish Project has been going through this month, was in an effort to eat more of our own dogfood.
The principle of eating your own dogfood is important for software quality, that is how you experience what your users are dealing with and I am not the least ashamed to admit that several obvious improvements have already appeared on my TODO list as a result of this transition.
But it is also important to externalize what you learn doing so, and therefore I thought I would document here how the projects new “internal IT” works.
Yes, we use some kind of hardware, but to be honest I don’t know what it is.
Our primary site runs on a RootBSD ‘Omega’ virtual server somewhere near CDG/Paris.
And as backup/integration/testing server we can use any server, virtual or physical, as long as it has a internet connection and contemporary performance, because the entire install is scripted and under version control (more below)
So, dogfood: Obviously FreeBSD.
Apart from the obvious reason that I wrote a lot of FreeBSD and can get world-class support by bugging my buddies about it, there are two equally serious reasons for the Varnish Project to run on FreeBSD: Dogfood and jails.
Varnish Cache is not “software for Linux”, it is software for any competent UNIX-like operating system, and FreeBSD is our primary “keep us honest about this” platform.
You have probably heard about Docker and Containers, but FreeBSD have had jails since I wrote them in 1998 and they’re a wonderful way to keep your server installation sane.
We currently have three jails:
Script & Version Control All The Things
We have a git repos with shell scripts which create these jails from scratch and also a script to configure the host machine properly.
That means that the procedure to install a clone of the server is, unabridged:
# Install FreeBSD 10.3 (if not already done by hosting) # Configure networking (if not already done by hosting) # Set the clock service ntpdate forcestart # Get git env ASSUME_ALWAYS_YES=yes pkg install git # Clone the private git repo git clone ssh://example.com/root/Admin # Edit the machines IP numbers in /etc/pf.conf # Configure the host sh build_host.sh |& tee _.bh # Build the jails foreach i (Tools Hitch Varnish) (cd $i ; sh build* |& tee _.bj) end
From bare hardware to ready system in 15-30 minutes.
It goes without saying that this git repos contains stuff like ssh host keys, so it should not go on github.
Right now there is nothing we need to backup.
When I move the mailserver/mailman/mailing lists over, those will need to be backed up, but here the trick is to only backup the minimal set of files, and in a “exchange” format, so that future migrations and upgrades can slurp them in right away.
We have taken snapshots of some of the old webproperties, Trac, the Forum etc as static HTML copies.
It is a little bit tedious to get a setup like this going, whenever you tweak some config file, you need to remember to pull the change back out and put it in your Admin repos.
But that extra effort pays of so many times later.
You never have to wonder “who made that change and why” or even try to remember what changes were needed in the first place.
For us as a project, it means, that all our sysadmin people can build a clone of our infrastructure, if they have a copy of our “Admin” git repos and access to github.
And when FreeBSD 11 comes out, or a new version of sphinx or something else, mucking about with things until they work can be done at leisure without guess work.
For instance I just added the forum snapshot, by working out all the kinks on one of my test-machines.
Once it was as I wanted it, I pushed the changes the live machine and then:
varnishadm vcl.use backup # The 'backup' VCL does a "pass" of all trafic to my server cd Admin git pull cd Tools sh build_j_tools.sh |& tee _.bj varnishadm vcl.load foobar varnish-live.vcl varnishadm vcl.use foobar
For a few minutes our website was a bit slower (because of the extra Paris-Denmark hop), but there was never any interruption.
And by doing it this way, I know it will work next time also.
All that buzz about “reproducible builds” ? Yeah, not a new idea.