Feb 19, 2018

My personal wiki deployment journey

I’ve always maintained a knowledge base of some kind. If something worked well for me I would write it down and eventually this knowledge accumulated over time. Examples of this knowledge includes:

  • Steps for Linux installation. This was back in the 90s when Linux didn’t install itself and you had to do it by hand (tips on how to set up network, how much swap to allocate, etc.)
  • How to optimally configure the ePSXe emulator so that I could play Playstation games without having to empty my teenage allowance for a GPU
  • Long before RDS, instructions to myself on configuring my.cnf MySQL config file and finding optimal values for max_heap_table_size, tmp_table_size, query_cache_size, table_open_cache, key_buffer_size, etc.
  • How to configure VirtualHosts for an Apache web server, because I could never keep that directive shit in my head

Here’s an example of some of my knowledge base writing (to myself) in high school, from a 17-year-old me with too much time on my hands, no normal life, and a rather devious side hobby of ripping DVD movies to create high quality compressed video files:

Video compression 1

Video compression 2

All this knowledge needed a home a bit more permanent than text files on my desktop. So sometime in 2006, I stumbled upon an open source Python project called Trac that I found perfect for organizing my knowledge base. I hosted it on a private VPS and served it through Apache web server. For 10 years between 2006-2016, I only ever tweaked this deployment twice to stay “with the times.” The first time was to swap the Apache module I used to serve Python from mod_wsgi to mod_python. The second time was years later but more disruptive: I migrated my Trac wiki to Heroku, started using a real Python package manager, served the Wiki with tracd process, and used Postgres rather than the default Sqlite (a by-product of adopting the ephemeral filesystem practice that Heroku introduced to me). For the most part though, the process of updating my wiki was pretty much the same.

All the while, I started seeing some problems with my wiki setup:

  1. Trac is really slow. Maybe it’s psycopg’s fault? Or maybe it’s just Python? Oops I may have just insulted a data scientist or two.
  2. To stay within Heroku’s free tier, my dyno has to go to sleep.
  3. Trac is really old. Just look at the design of their website and tell me this doesn’t scream 2006 to you.
  4. All my wiki content is stored in database. Although Trac versions everything there, it’s far from the modern git standards I’ve adopted over that same decade.
  5. Although I have Trac edit functionality protected by HTTP auth, I don’t like the idea of my wiki being web editable anyway. I’d rather edit text files and have the capability of staging my changes, rather than have changes go straight to database per article update.
  6. Trac wiki syntax is an odd format. This stands in contrast to the more modern markdown standards I’ve adopted, for example GFM.

With the exception of problem #6, there were no Trac plugins that solved any of these problems. Speaking of plugins, might I add:

7) The Trac plugin ecosystem is getting stale, with few/far between updates and an exodus of their maintainers.

Then in 2016, I discovered a couple of amazing technologies that completely flipped my personal wiki’s world upside down: Gollum and Docker. From the description of their capabilities, I felt pretty strongly that I could use them to solve all the problems I had above and more!

Let’s start with Gollum. To start, the default interface is really clean and nice, and mobile-friendly out of the box:

Gollum interface

While these characteristics don’t speak directly to my problems per se, they are a strong proxy for how much more modern the Gollum community is than the Trac community.

The way Gollum works is very cool and solves many of my problems. It is actually intimately tied to the underlying git repository and uses the commit history to derive changes and browse the history of wiki articles. This is the best of both worlds: Gollum still provides a nice CRUD interface for wiki modification, but all changes made become git commit operations. This also means I have the flexibility to modify the raw markdown files myself and commiting via command line as in my typical project workflow. Speaking of markdown, the flavor of markdown can be configured but even the default is a modern markdown format. Finally, Gollum itself runs as a Sinatra app and while it’s not mind-blowingly fast, it definitely beats serving Trac via any of tracd, Apache mod_python, or mod_wsgi.

So I migrated my Trac wiki to Gollum, which only required a few simple global string replacements to convert the wiki formats for each article. Here is a partial sample of a conversion command:

$ cat SomeTracArticle | sed "s/'''/**/g" | sed "s/''/*/g" | sed "s/^== \(.*\) ==$/### \1/g" | sed "s/^=== \(.*\) ===$/#### \1/g" | sed "s/^==== \(.*\) ====$/##### \1/g" | sed "s/[.*]" > SomeTracArticle.md

Boom, I was done in like 5 minutes! I now had a Gollum wiki ready to go. But then I hit a wall: how do I deploy this thing?

I had a quick WTF moment when I read Gollum Issue #1013, where a contributor answers a question about why there are no docs about deployment: “That is probably because Gollum is mostly meant as a personal wiki. Most people seem to use it locally on their computers, they launch and stop Gollum when they need to.” This was pretty worrisome because I derive a lot of value from my knowledge base being highly available and accessible. After all, my knowledge base is way less useful if a condition of getting to it is to be at my terminal running gollum. Or on a foreign terminal having to git clone my wiki repo and then gem install gollum just to view it.

Since I had already been running Trac on Heroku, I thought I’d give Heroku a shot. So I whipped together my Gemfile and Procfile and sent the app on its way:

# Procfile
web: bundle exec gollum --config config-prod.rb

# Gemfile
source 'https://rubygems.org'
gem 'gollum'

$ heroku apps:create albert-gollum-deployment
$ git push heroku master

There was a problem with this though:

Gollum::InvalidGitRepositoryError at /
file: git_layer_grit.rb location: rescue in initialize line: 289

Here was a fundamental clash: Gollum depends on an underlying git repository to derive its content, but Heroku by default strips the deployment’s app filesystem of the underlying git repo. I totally get it - you generally don’t need a whole repo sitting inside an app’s build, so dropping it makes it lighter weight, and there are potential security risks with it being in the filesystem, etc. But in this case, Gollum actually requires it. As Heroku goes, it’s not possible to override that situation.

Bending to Heroku’s standards for deployments has been the status quo for a while, and here I finally hit a wall with those ways. While Heroku has been great for quickly getting stuff out there, I felt it was time to get situated towards the more flexible and portable future of app deployment.

Enter Docker

As luck would have it, the Gollum project has its own documentation for Docker deploys. With this simple Dockerfile:

FROM ruby
RUN apt-get -y update && apt-get -y install libicu-dev cmake && rm -rf /var/lib/apt/lists/*
RUN gem install github-linguist
RUN gem install gollum
RUN gem install org-ruby  # optional
WORKDIR /wiki
ENTRYPOINT ["gollum", "--port", "80"]
EXPOSE 80

I could spin up a container that would run Gollum on my local Gollum repository like this:

$ docker build -t gollum .
$ docker run -d -v `pwd`:/wiki -p 4567:80 gollum

This produces the exact same result as simply running:

$ gollum

but the difference is that with the Docker way, none of it is dependent on what I have installed on my host. It’s running entirely self-sufficiently on what I gave it in the Dockerfile to build its image. I could delete the gollum package and even ruby itself on my host and it won’t matter a single bit because this container takes care of its own system dependencies! This was super exciting to me, but I still need to get this container on the Internet!

Back to VPS?

Normally at this point, I would spin up a standard Ubuntu VPS on any of {Digital Ocean, Linode, AWS}, install Docker daemon on it, clone my wiki repo there, then spin up the container and bind it to the VPS’ port 80, and voila – I’d have a production deployment of my wiki! But now this seems like a lot of VPS management once again. Is there a simpler container deployment solution?

Well, it turns out the world has changed a bit and now nearly every cloud provider has container services of some kind, with varying levels of service.

At the most basic, there is simply the ability to manually install Docker daemon on a VPS - the host just needs to be a 64-bit installation, which is pretty much a table stakes offering from every cloud provider at this point. Linode is an example of a cloud provider that has no more than this level.

The next higher level of service offers a VPS that comes pre-installed with Docker. This is pretty much how I would describe Digital Ocean’s one-click app offering for Docker.

Up to this point, all we have is a regular VPS that has Docker installed on it (however it happens). You still need to do the legwork of deploying the app onto the machine somehow, running a docker build, and then running a detached docker run command. If you want the image pre-built before hitting the VPS, you’d either have to push your locally built image to Docker Hub and then pull it from within the VPS (exposing it publicly along the way) or create a private Docker repository on yet another separate VPS, or perhaps even the same one. It’s certainly possible to do all this, but it’s still a lot of setup to do and SSH commands to run.

An even higher level of service offers a private Docker registry to push images to along with a VPS that is pre-configured with permission to pull your images from that registry. The registry and VPS work hand-in-hand to make it a smoother process to deploy. Heroku Docker Deploys, Amazon Container Services, Google Cloud Platform, and IBM Cloud fit the bill on this level of service.

Finally, there is a level of service that manages clusters through a Kubernetes master service. Containers are deployed, re-deployed, and scaled quickly, easily, and in a very strategic fashion. They are pulled into service ad-hoc via a load balancer of some kind. Amazon Container Services, Google Cloud Platform, and IBM Cloud offer this level of service for your container deployment.

The Decision

The amount of services out there is pretty overwhelming. While there’s a lot of comparison to be done at the VPS resources level (CPU, memory, etc.), cost comes into play too.

I decided that the first two levels of service required too much VPS management on my end. In addition to avoiding setup work, I would ideally like it such that I don’t need to SSH into anything or maintain keys here and there just to make a deploy. While there is certainly a way you can design a containerized deployment on VPS that only requires a git push, even that setup involves local/host key exchange and maintenance that I don’t want to deal with.

I also decided that the last level of service was overkill. While there are some advantages to running Kubernetes node clusters over VPS, such as faster container startup time and node auto repair, I could not see the benefits of autoscaling, round robin updates, microservices, and load balancing for a simple wiki that pretty much only I would be referencing. Especially considering that it’s relatively costly. GCP’s f1-micro and g1-small instance sizes require more than one node and if you want to expose the app externally (which is the point), then you have to pay a minimum $18/month to operate the Load Balancer service. IBM Cloud has a free tier for a single-node cluster, but yet again they get you with the exposer service: you could use NodePort for free but it limits you to ports 30000+, so in practice you’d use a costly Load Balancer service again.

So I went for Google Cloud Platform’s free tier single f1-micro VM instance. This was the only service that could not only serve a simple low-traffic single-node container for free, but it could do so without my ever having to (a) run a single shell command, (b) exchange a single public key or set a single permission, (c) alter a single network setting.

GCP VM

The minimum requirements are to have pushed my image to Google Cloud Registry using Google Cloud SDK, which after setting up a project and installing the SDK is as simple as:

$ export PROJECT_ID="$(gcloud config get-value project -q)"
$ docker build -t gcr.io/${PROJECT_ID}/${PROJECT_ID}:v1 .
$ gcloud docker -- push gcr.io/${PROJECT_ID}/${PROJECT_ID}:v1

Then in that instance creation form, I simply supply the gcr.io container image location and set the firewall to allow HTTP traffic (port 80), and I’m done! Updates to my app after this are simply a matter of editing the instance and supplying a new container image (the next version).

Small Fix Needed

There is one important point I didn’t touch on, which has to do with the Dockerfile I displayed earlier. To recall, it was written like this:

FROM ruby
RUN apt-get -y update && apt-get -y install libicu-dev cmake && rm -rf /var/lib/apt/lists/*
RUN gem install github-linguist
RUN gem install gollum
RUN gem install org-ruby  # optional
WORKDIR /wiki
ENTRYPOINT ["gollum", "--port", "80"]
EXPOSE 80

and when running the container, you would have to run this in the project’s root directory:

$ docker build -t gollum .
$ docker run -d -v `pwd`:/wiki -p 4567:80 gollum

This strategy works well in development locally, but it breaks down when you build it as is and try to deploy. Why? Mainly because the image expects a volume with all of the project’s files to be mounted into the container from the host via the -v argument, but in this managed host those files don’t exist (unless you manually copy them over, but that’s extra work for a simple deployment!). To fix this, I’d re-write the Dockerfile to the following:

FROM ruby
RUN apt-get -y update && apt-get -y install libicu-dev cmake && rm -rf /var/lib/apt/lists/*
RUN gem install github-linguist
RUN gem install gollum
RUN gem install org-ruby  # optional
RUN git config --global user.name "Albert Ho" && git config --global user.email ah323@cornell.edu && git config --global core.editor "vi"
ENV WIKI /wiki
WORKDIR $WIKI
COPY . $WIKI
CMD ["gollum", "--port", "80", "--config", "config-prod.rb"]
EXPOSE 80

A couple changes to note here:

  • The COPY line is added to make sure the entire project directory is copied into the image at build time. This fixes the production issue. When running this locally for development, you can still mount the host project directory as a volume and it will override the copied files! That way changes in the host filesystem are immediately reflected when developing the app. Best of both worlds.
  • Notice that ENTRYPOINT is replaced with CMD. This is because we still want that default command to be run, but also want to leave the capability to override that command in a local development container. Why do this? Well, there is now a second argument --config config-prod.rb, which invites a config file that closes off editing capability in the container by default. To re-open editing capability during development, you can run the development container like this to override that CMD:
$ docker run -d -v `pwd`:/wiki -p 4567:80 gollum gollum --port 80

That last part gollum --port 80 basically overrides the CMD with almost the equivalent, just that it omits the production config file.