This morning, I was trying to wrangle our CI/CD pipeline for the Containers BOSH Release so that I could cut a 1.1.0 release and generally forget about the process of integration testing.
Our stock pipeline architecture for BOSH releases runs a deployment test by taking a manifest — either the example manifest or a CI-specific manifest — and deploying it to a BOSH director. If the deployment takes, the BOSH release is considered fit for a release.
To conserve space, we usually target a BOSH director running the Warden CPI, which just spins up Linux containers for each BOSH instance group. For most things, this is sufficient; BOSH releases rarely care about the infrastructure they are deployed on, after all.
But the Containers BOSH Release is a bit different. It runs Docker, which needs a whole slew of container-y things like cgroups to function. When I tried deploying it onto a containerized CPI, it blew up, as things on the cutting edge are wont to do.
Task 352 | 18:16:01 | Updating instance docker: docker/b3ab4f49-2f05-4865-b46c-eb248ebded5b (0) (canary) (00:02:26) L Error: 'docker/b3ab4f49-2f05-4865-b46c-eb248ebded5b (0)' is not running after update. Review logs for failed jobs: docker, compose
Looking into the logs, I found that, at least under Warden, you don’t have access to mount new things (including
cgroup hierarchies) under
/sys/fs/cgroup. It’s just plain not allowed.
A simple workaround is to move the mountpoint elsewhere. Docker still finds the cgroups wherever they happen to be mounted. However, since this is a workaround specifically for Warden, I was hoping to find a way to limit the behavior to only occur there. Other IaaS deployments (vSphere, AWS, GCP, etc.) are perfectly happy to mount things where they belong.
Luckily, I found just what I was looking for in
/var/vcap/bosh/etc/infrastructure. On Warden, this contains the text
warden, making it dead easy to recognize a Warden CPI from a mile away:
if grep -q warden /var/vcap/bosh/etc/infrastructure ]]; then echo 'Hello, Warden!' # ... do warden-y things ... else # ... fi
In the past, I’ve resorted to such tricks as the
properties: im_in_warden_dont_do_nfs: yes
or the just-ignore-the-failure trick, which is less than ideal.
Now, armed with
/var/vcap/bosh/etc/infrastructure, I can make IaaS-specific decisions to tailor my BOSH releases to the situation at hand.