This morning, I was trying to wrangle our CI/CD pipeline for the Containers BOSH Release so that I could cut a 1.1.0 release and generally forget about the process of integration testing.
Our stock pipeline architecture for BOSH releases runs a deployment test by taking a manifest — either the example manifest or a CI-specific manifest — and deploying it to a BOSH director. If the deployment takes, the BOSH release is considered fit for a release.
To conserve space, we usually target a BOSH director running the Warden CPI, which just spins up Linux containers for each BOSH instance group. For most things, this is sufficient; BOSH releases rarely care about the infrastructure they are deployed on, after all.
But the Containers BOSH Release is a bit different. It runs Docker, which needs a whole slew of container-y things like cgroups to function. When I tried deploying it onto a containerized CPI, it blew up, as things on the cutting edge are wont to do.
Task 352 | 18:16:01 | Updating instance docker:
docker/b3ab4f49-2f05-4865-b46c-eb248ebded5b (0) (canary) (00:02:26)
L Error:
'docker/b3ab4f49-2f05-4865-b46c-eb248ebded5b (0)' is not running
after update. Review logs for failed jobs: docker, compose
Looking into the logs, I found that, at least under Warden, you don't have access to mount new things (including cgroup hierarchies) under /sys/fs/cgroup. It's just plain not allowed.
A simple workaround is to move the mountpoint elsewhere. Docker still finds the cgroups wherever they happen to be mounted. However, since this is a workaround specifically for Warden, I was hoping to find a way to limit the behavior to only occur there. Other IaaS deployments (vSphere, AWS, GCP, etc.) are perfectly happy to mount things where they belong.
Luckily, I found just what I was looking for in /var/vcap/bosh/etc/infrastructure. On Warden, this contains the text warden, making it dead easy to recognize a Warden CPI from a mile away:
if grep -q warden /var/vcap/bosh/etc/infrastructure ]]; then
echo 'Hello, Warden!'
# ... do warden-y things ...
else
# ...
fi
In the past, I've resorted to such tricks as the iminwardendontdoX property_:
properties:
im_in_warden_dont_do_nfs: yes
or the just-ignore-the-failure trick, which is less than ideal.
Now, armed with /var/vcap/bosh/etc/infrastructure, I can make IaaS-specific decisions to tailor my BOSH releases to the situation at hand.
James (@iamjameshunt) works on the Internet, spends his weekends developing new and interesting bits of software and his nights trying to make sense of research papers.
Currently exploring Kubernetes, as both a floor wax and a dessert topping.