Automatic github pulling in a Docker environment

tl;dr: How to use our webhooks Docker image for continuous deployment in development

We're transitioning our infrastructure to the cloud, to reduce our dependence on having a server admin, and to increase our uptime and scalability.

We've decided to build our new infrastructure on top of Docker, for portability and scalability. It also strongly encourages a modular design, which is another goal of ours.

For our production environment we haven't yet decided how we should do deploys, although having our CI setup build and deploy new containers is a likely approach.

For development, however, we wanted to recreate a system we've used until now on our legacy server. That system was very specific and not very portable, so we wanted to build a new, more portable version on top of Docker.

Since our current version is written in Python, and it seems to be a good tool for the job, we chose Python for the new version as well.

The solution

What we ended up building was a docker image that pulls code into a volume that it shares with the web server, based on incoming webhooks from GitHub. When done, it pings the web server to do whatever it needs to do with the new code.

The docker image is even more flexible than that, as it can run any arbitrary terminal commands when it receives the webhook. It can also be configured to serve multiple repositories at once.

How to use

1. Setting up your web server

We use our custom rails dev server. It is important that this server serves its files from a volume. this can either be a volume that the webhook hooks into with the --volume-from argument of docker run, or a shared mounted volume from the host computer.

2. Write your repos.json configuration file.

This is a configuration file that the webhook listener uses to know which hooks to listen to and how to verify their integrity with a webhook secret. We opted to have it call a separate .sh file, as we needed to add a ssh key, which proved tricky to do through Pythons subprocess. If you know how, let me know in the comments or on twitter.

{
    "PAM-AS/pam-api/branch:develop": {
        "path":"/usr/local/application",
        "key":"<our-secret-webhook-key>",
        "action":[
                ["sh", "/config/pamapi/pull.sh"]
        ]
    }
}

You can read more about webhooks in GitHub's docs.

Our pull.sh looks like this:

ssh -o StrictHostKeyChecking=no git@github.com
eval "$(ssh-agent -s)" \
&& ssh-add /config/pamapi/key.ssh \
&& git config --global user.email "pixie@pam.as" \
&& git config --global user.name "Pixie" \
&& git clone -b develop git@github.com:PAM-AS/pam-api.git temp_git \
&& rm -rf .git \
&& mv -f temp_git/.git . \
&& rm -rf temp_git \
&& git reset --hard \
&& nc rails-server 2000

Note: I'm planning to replace the netcat (nc) poke to the web server with a RabbitMQ message at a later point.

3. Pull down the webhook docker image

Now you can pull and boot the github webhooks base image.

docker pull thomassnielsen/github-webhooks-base

4. Launch the image with your configuration of choice

To use the image, you must provide a volume and a path for the repos.json config file, as well as a volume where you would like the code to go:

-v /config/rails:/config
--volumes-from=my-rails-server
-e "REPOS_JSON_PATH=/config/repos.json"

You can optionally provide an environment variable for which port the image should listen on (the default is 41414):

-e "WEBHOOKS_PORT=41414"

The finished command should look something like this:

docker run -d --name my-webhook-container -v /config/rails:/config -e "REPOS_JSON_PATH=/config/repos.json" -e "WEBHOOKS_PORT=41414" --volumes-from=my-rails-server thomassnielsen/github-webhooks-base

At this point, you should be ready to receive webhooks. Set up your webhook and update your repository to test it.

If you have the volume shared on the docker host, you should be able to see the git files show up there. If not, you can either try to access the web server container's exposed port, or you can enter the container and look for the files there:

docker exec -it my-rails-server bash

Ideas for where to go next

I'd like to tie all this together with docker composer, so that every container that works together can be launched together. I'd also like to nail down the production version of this.

Another challenge will be to put up a branch system. We have this for our web client in our legacy setup, and I want to bring it over both for the web client and the API. The branch system works by serving each branch from git on a different subdomain, automatically. It calls for some fancy web server work, and I'm not quite sure how to set it up for rails yet.