Putting Docker Compose on top of BOSH — or — I May Never Write Another BOSH Release Again

My my, that is a provocative title for this blog, isn't it?

I've just finished up work on my latest experiment with BOSH: the Containers BOSH Release. As more and more of my life slips into Docker containers (on its way to Kubernetes), I find myself wanting to spend less and less time re-packaging stuff for BOSH. Lots of software already exists in Docker form, since that's a pre-requisite for deploying things on Helm (and K8s), which is all the rage.

The other day, I said to myself, “James, what if we just ran the Docker images on top of BOSH?” It seemed crazy at first, but the more I through it through (and the more I played with the implementation) the more sane and normal it became.

The premise of the Containers BOSH Release is simple: start with a docker-compose recipe, and run it on one or more BOSH VMs.

Hello, Vault!

I like to start with easy examples, so we're going to spin up a single-node Vault instance to prove that this crazy Containers thing works.

We'll start with the simplest of manifest stubs:

---
name: vault
stemcells:
  - alias:   default
    os:      ubuntu-xenial
    version: latest

releases:
  - name:    containers
    version: latest

update:
  canaries:          1
  max_in_flight:     1
  serial:            true
  canary_watch_time: 1000-120000
  update_watch_time: 1000-120000

instance_groups:
  - name:      docker
    instances: 1 
    azs:       [z1]
    vm_type:   default
    stemcell:  default
    networks:  [{name: default}]

    jobs:
      - name: docker
        release: containers
        properties:
          recipe:
            # .... START HERE ...

Under the recipe property, we'll just insert a bit of docker-compose:

# ... continuing on ...

          recipe:
            version: '3'
            services:
              vault:
                image: vault
                ports: ['8200:8200']
                environment:
                  VAULT_API_ADDR: http://127.0.0.1:8200
                  VAULT_LOCAL_CONFIG: >-
                    {
                      "disable_mlock": 1,
                      "backend": {
                        "file": {
                          "path": "/vault/file"
                        }
                      },
                      "listener": {
                        "tcp": {
                          "address":     "0.0.0.0:8200",
                          "tls_disable": 1
                        },
                      },
                      "default_lease_ttl": "168h",
                      "max_lease_ttl": "720h"
                    }
                cap_add: [IPC_LOCK]
                command: [vault, server, -config, /vault/config/local.json]

That's it. Toss that at your favorite BOSH director, and when it's all deployed, you should be able to access the Vault on port 8200.

(If you want the full manifest, download it from here)

$ bosh deploy -n vault.yml
$ bosh vms
Instance         Process State  AZ  IPs             VM CID      VM Type  Active
docker/b046a21f  running        z1  10.128.16.143  vm-a47c3bd5  default  true

Let's target that with safe:

$ safe target http://10.128.16.143:8200 dockerized
Now targeting dockerized at http://10.128.16.143:8200

Since this is a new Vault, we're going to need to initialize it:

$ safe init
Unseal Key #1: df16cd701bde233c768cda6c20e214e640bc43cd1b81d977a983d5590dd2659a03
Unseal Key #2: fe8ea8ab8a22ef931d5338dd6f4f2f6932ffa22f6caa26fd30cd57e11ffe137260
Unseal Key #3: c6a872983488ae92e30bb0f74a1a2795978e247c502b4684e70189fe0ba2ad90c6
Unseal Key #4: 6c72063fcf8a9b82a7c72fa286d14f84c9e46c7e30d21d9040ebbfab7725740170
Unseal Key #5: 836f40ec1f7c34460fc86b7547caccc7ffa6b680b9a69a0205b1cddebcb33d2530
Initial Root Token: s.BoFveccftUE9y9j1p4WfyPpO

Vault initialized with 5 keys and a key threshold of 3. Please
securely distribute the above keys. When the Vault is re-sealed,
restarted, or stopped, you must provide at least 3 of these keys
to unseal it again.

Vault does not store the master key. Without at least 3 keys,
your Vault will remain permanently sealed.

safe has unsealed the Vault for you, and written a test value
at secret/handshake.

You have been automatically authenticated to the Vault with the
initial root token.  Be safe out there!

There you go, a Vault! And we didn't have to write our own BOSH release.

Taking SHIELD For A Spin

Let's try a bit more complicated of an example, shall we?

The SHIELD docker-compose recipe consists of five different containers that work together to provide all the moving parts necessary to evaluate SHIELD's effectiveness as a data protection solution:

  1. core - The SHIELD Core (i.e. most of SHIELD itself)
  2. vault - A private vault, for storing encryption parameters.
  3. agent - The SHIELD agent, for executing backup / restore
  4. webdav - Cheap, local cloud storage. operations.
  5. demo - You gotta have something to backup, right?

Despite all of this complexity, deploying to BOSH via Containers is just as straightforward — just drop the docker-compose.yml file contents (properly indented, of course) under the recipe: property of the docker job.

If you want, you can read the entire BOSH manifest.

Once that is deployed, I used bosh vms to get the IP address of the deployed VM, and then targeted that IP (on port 9009) with the SHIELD CLI:

$ bosh vms
Instance         Process State  AZ  IPs           VM CID       VM Type  Active
docker/d9b2c11d  running        z1 10.128.16.144  vm-add4d02e  default  true

$ shield api http://10.128.16.144:9009 docker-shield
docker-shield  (http://10.128.16.144:9009)  OK
SHIELD DOCKER

$ shield -c docker-shield login
SHIELD Username: admin
SHIELD Password:
logged in successfully

$ shield -c docker-shield status
SHIELD DOCKER v8.2.1
API Version 2

If you want, you can head on over to the SHIELD web UI (also on port 9009).

Playing with Anchore (on BOSH)

Anchore is a security scanning solution for Docker images. You give it an image URL and it will pull that image down, unpack it, and scan it for known CVEs and other vulnerabilities. As more of my F/OSS projects move to delivering OCI images as release assets, I wanted a solution that could scan (and re-scan) those images quickly and painlessly.

(by the way, according to their Slack org, Anchore rhymes with encore).

It's a neat system, and its canonical deployment is via a docker-compose recipe, making it a perfect fit for this new mental model of deploying to BOSH.

To deploy, I started with the upstream docker-compose recipe, and then tweaked it slightly (mostly by renaming the Docker containers). The final manifest is here. Go ahead and deploy it; we're going to segue briefly into some theory, but we'll come back to Anchore soon enough.

How Does It Work?

The Containers BOSH release does the following for you:

  1. Package up Docker 18.x and docker-compose
  2. Create a docker-compose.yml file on the BOSH VM, based on what you put in the recipe property.
  3. Call docker-compose up from a script that monit (BOSH's supervisor) babysits for you.

That's (virtually) all there is to it. Don't let the simplicity deceive you! This straightforward formula unlocks some serious potential in using BOSH to run your (already) dockerized workloads. More importantly, it frees us from needing to figure out how to package and execute upstream software.

All the stability of BOSH, all of the flexibility of Docker!

What About Air Gapping?

The first thing our docker-compose up is going to do is contact some registry somewhere and pull down the images it needs to run the project. Depending on your environment, this may either be acceptable (I hope you are pinning your image tags), or it may not. In "air gapped" environments, you cannot directly download anything from the public Internet and run it, for security reasons.

That pretty much rules out this new thing, right?

Wrong. I saw this edge case a mile away — I work with lots of environments that are either forced to use semi-broken HTTP(S) proxies, or are simply not allowed out to the Internet.

After the Docker daemon boots, but before we run docker-compose up, we scan the disk looking for other BOSH jobs that might be able to provide us with the raw material of OCI-compliant images: layered tarballs.

The code in question looks a little something like this:

for image in $(cat /var/vcap/jobs/*/docker-bosh-release-import/*.lst)
do
  docker load <$image
done

Any job that gets co-located on the instance group has the option of defining a list of paths to exported OCI image tarballs, that we will load into our local Docker daemon.

So yes, if you want to run in an air gapped environment, you do still have to write BOSH releases, but they are super simple and require almost no thought. I even wrote a proof-of-concept image release that packages up the Vault image we've been playing with.

What About docker exec?

One of my favorite features of Kubernetes is kubectl exec; being able to just bounce into a container and poke around to verify things is amazingly powerful.

You can imitate this powerful feature by binding the Docker daemon to a TCP port. Normally, Docker just binds a UNIX domain socket (like a port, but its a file). This provides some level of protection for Docker, since you have to be on the Docker host to see the socket file, and you have to have the appropriate group memberships to be allowed to write to it.

If you set the bind: property of the docker job to a TCP port number, the Docker daemon will also listen on that port, across all interfaces, for inbound control messages. You can combine this with the -H ... flag to the docker CLI utility, or, better yet, the $DOCKER_HOST environment variable.

If you look closely at the Anchore BOSH manifest, you'll notice that it binds port 5001, letting us do this:

$ bosh -d anchore vms
Instance         Process State  AZ  IPs           VM CID       VM Type  Active
docker/2e016a81  running        z1 10.128.16.142  vm-ffbfedd1  default  true

$ docker -H 10.128.16.142:5001 ps
CONTAINER ID  IMAGE       STATUS         PORTS             NAMES
47fb54c4b539  (anchore)   Up (healthy)   8228/tcp          running_simpleq_1
16addb64383e  (anchore)   Up (healthy)   8228/tcp          running_policy-engine_1
c56b06c1ce10  (anchore)   Up (healthy)   *:8228->8228/tcp  running_api_1
f0ab474b62fc  (anchore)   Up (healthy)   8228/tcp          running_analyzer_1
3c46e9c70e3c  (anchore)   Up (healthy)   8228/tcp          running_catalog_1
c96f8a5b9596  (postgres)  Up             5432/tcp          running_db_1

Since we have complete access to Docker, let's try an exec:

$ export DOCKER_HOST=10.128.16.142:5001
$ docker exec -it running_api_1 /bin/bash
[anchore@c56b06c1ce10 anchore-engine]$

Success! From here, you can use the embedded anchore-cli to interact with the scanning solution:

[anchore@c56b06c1ce10 anchore-engine]$ anchore-cli system feeds list
Feed              Group            LastSync                    RecordCount
vulnerabilities   alpine:3.3       2019-06-13T01:23:20.228251  457
vulnerabilities   alpine:3.4       2019-06-13T01:23:32.916373  681
vulnerabilities   alpine:3.5       2019-06-13T01:23:49.301987  875
vulnerabilities   alpine:3.6       2019-06-13T01:24:10.832360  1051
vulnerabilities   alpine:3.7       2019-06-13T01:24:38.684207  1125
vulnerabilities   alpine:3.8       2019-06-13T01:25:08.305060  1220
vulnerabilities   alpine:3.9       2019-06-13T01:25:39.714090  1284
vulnerabilities   amzn:2           2019-06-13T01:26:02.169712  178
vulnerabilities   centos:5         2019-06-13T01:27:20.146913  1323
vulnerabilities   centos:6         2019-06-13T01:28:43.593786  1333
vulnerabilities   centos:7         2019-06-13T01:29:58.219342  793
vulnerabilities   debian:10        2019-06-13T01:37:15.136873  20352
vulnerabilities   debian:7         2019-06-13T01:45:14.279082  20455
vulnerabilities   debian:8         2019-06-13T01:53:57.173538  21775
vulnerabilities   debian:9         2019-06-13T02:02:19.820286  20563
vulnerabilities   debian:unstable  2019-06-13T02:10:43.437588  21245
vulnerabilities   ol:5             2019-06-13T02:12:14.888089  1233
vulnerabilities   ol:6             2019-06-13T02:14:05.563999  1417
vulnerabilities   ol:7             2019-06-13T02:15:33.553661  915
vulnerabilities   ubuntu:12.04     2019-06-13T02:20:54.345817  14948
vulnerabilities   ubuntu:12.10     2019-06-13T02:22:54.519832  5652
vulnerabilities   ubuntu:13.04     2019-06-13T02:24:15.630364  4127
vulnerabilities   ubuntu:14.04     2019-06-13T02:30:58.521151  18693
vulnerabilities   ubuntu:14.10     2019-06-13T02:32:41.325661  4456
vulnerabilities   ubuntu:15.04     2019-06-13T02:34:46.806824  5789
vulnerabilities   ubuntu:15.10     2019-06-13T02:37:12.849533  6513
vulnerabilities   ubuntu:16.04     2019-06-13T02:43:25.005589  15795
vulnerabilities   ubuntu:16.10     2019-06-13T02:46:17.313668  8647
vulnerabilities   ubuntu:17.04     2019-06-13T02:49:16.106250  9157
vulnerabilities   ubuntu:17.10     2019-06-13T02:51:51.897617  7935
vulnerabilities   ubuntu:18.04     2019-06-13T02:55:06.965036  10047
vulnerabilities   ubuntu:18.10     2019-06-13T02:57:35.083799  8134
vulnerabilities   ubuntu:19.04     2019-06-13T02:59:23.443062  6586

etc.

What About Insecure Registries?

In a perfect world, we'd all use TLS everywhere, and everyone would have a verifiable chain of trust back to a CA root authority.

Ha!

Sometimes, you can't help it that your Docker Registry is protected by a self-signed certificate, or one whose CA isn't in the system roots. Occasionally, you have to go without transport security altogether and run on plain old HTTP.

For that, Docker supports the concept of an insecure registry, and the Containers BOSH release lets you supply a list of those ip:port endpoints that just won't pass muster on X.509 validation:

jobs:
  - name: docker
    release: containers
    properties:
      insecure-registries:
        - docker.corp.int:3000

That way, if you have to pull any images from those registries as part of your compose recipe, you're covered.

What About Private Registries?

That one we're still working on, but given that we can now trivially spin our own Docker Registries, using this new BOSH release and upstream Docker images, I expect that will get fixed soon enough. You might even be the one to PR that!

Where To From Here?

I hope my experiments here have piqued your interest. Go out, snag a copy of the latest release, and deploy your favorite Dockerized workload, now with the power of BOSH!

Oh, and if you run into any problems along the way, or find a way to improve the BOSH release, we hope to hear from you.

Happy Hacking!

James (@iamjameshunt) works on the Internet, spends his weekends developing new and interesting bits of software and his nights trying to make sense of research papers.

Currently working on Rook.

Liked That? Try These:

The Future of BOSH & Kubernetes

Docker, on macOS, DIY-Style

A Troubleshooting Yarn

Lisp On The Web

Variadic Functions & Their Representation By Source Code

Roll-Your-Own X.509 CA