Detecting Warden in a BOSH Release

9 Aug 2019

This morning, I was trying to wrangle our CI/CD pipeline for the Containers BOSH Release so that I could cut a 1.1.0 release and generally forget about the process of integration testing.

Our stock pipeline architecture for BOSH releases runs a deployment test by taking a manifest — either the example manifest or a CI-specific manifest — and deploying it to a BOSH director. If the deployment takes, the BOSH release is considered fit for a release.

To conserve space, we usually target a BOSH director running the Warden CPI, which just spins up Linux containers for each BOSH instance group. For most things, this is sufficient; BOSH releases rarely care about the infrastructure they are deployed on, after all.

But the Containers BOSH Release is a bit different. It runs Docker, which needs a whole slew of container-y things like cgroups to function. When I tried deploying it onto a containerized CPI, it blew up, as things on the cutting edge are wont to do.

Task 352 | 18:16:01 | Updating instance docker:
   docker/b3ab4f49-2f05-4865-b46c-eb248ebded5b (0) (canary) (00:02:26)
   L Error:
       'docker/b3ab4f49-2f05-4865-b46c-eb248ebded5b (0)' is not running
       after update. Review logs for failed jobs: docker, compose

Looking into the logs, I found that, at least under Warden, you don't have access to mount new things (including cgroup hierarchies) under /sys/fs/cgroup. It's just plain not allowed.

A simple workaround is to move the mountpoint elsewhere. Docker still finds the cgroups wherever they happen to be mounted. However, since this is a workaround specifically for Warden, I was hoping to find a way to limit the behavior to only occur there. Other IaaS deployments (vSphere, AWS, GCP, etc.) are perfectly happy to mount things where they belong.

Luckily, I found just what I was looking for in /var/vcap/bosh/etc/infrastructure. On Warden, this contains the text warden, making it dead easy to recognize a Warden CPI from a mile away:

if grep -q warden /var/vcap/bosh/etc/infrastructure ]]; then
  echo 'Hello, Warden!'
  # ... do warden-y things ...

else
  # ...

fi

In the past, I've resorted to such tricks as the im_in_warden_dont_do_X property:

properties:
  im_in_warden_dont_do_nfs: yes

or the just-ignore-the-failure trick, which is less than ideal.

Now, armed with /var/vcap/bosh/etc/infrastructure, I can make IaaS-specific decisions to tailor my BOSH releases to the situation at hand.

newer: The Art of Writing (Blogs)
older: Putting Docker Compose on top of BOSH

James works on the Internet, spends his weekends developing new and interesting bits of software and his nights trying to make sense of research papers.

Currently exploring just how much data you can shove though DuckDB before it explodes.