Jails are a super-chroot feature of the FreeBSD operating system that predate LXC/cgroups (first appearing in FreeBSD 4.0 in March 2000).
Each jail has it's own view of system resources, including the process table, filesystem, hostname, network numbering space (IPv4 and/or IPv6). From the Jails Chapter of the FreeBSD handbook:
Jails build upon the chroot(2) concept, which is used to change the root directory of a set of processes. This creates a safe environment, separate from the rest of the system. Processes created in the chrooted environment can not access files or resources outside of it. For that reason, compromising a service running in a chrooted environment should not allow the attacker to compromise the entire system. However, a chroot has several limitations. It is suited to easy tasks which do not require much flexibility or complex, advanced features. Over time, many ways have been found to escape from a chrooted environment, making it a less than ideal solution for securing services.
To understand the power of FreeBSD jails, I started applying some of the ideas we cooked up for Homeports and seeing how they would fly in a BSD environment.
Before I could even start the hands-on phase of the research (which, let's face it, is where you learn the important bits), I had to get FreeBSD up and running. Vagrant to the rescue!
$ mkdir ~/local/freebsd
$ cd ~/local/freebsd
$ cat >Vagrantfile <<EOF
Vagrant.configure(2) do |config|
# see https://atlas.hashicorp.com/search
# and https://forums.freebsd.org/threads/52717
config.vm.guest = :freebsd
config.vm.synced_folder ".", "/vagrant", id: "vagrant-root", disabled: true
config.vm.box = "freebsd/FreeBSD-11.0-CURRENT"
config.ssh.shell = "sh"
config.vm.base_mac = "080027D14C66"
config.vm.provider :virtualbox do |vb|
vb.customize ["modifyvm", :id, "--memory", "1024"]
vb.customize ["modifyvm", :id, "--cpus", "1"]
vb.customize ["modifyvm", :id, "--hwvirtex", "on"]
vb.customize ["modifyvm", :id, "--audio", "none"]
vb.customize ["modifyvm", :id, "--nictype1", "virtio"]
vb.customize ["modifyvm", :id, "--nictype2", "virtio"]
end
end
EOF
$ vagrant up
$ vagrant ssh
That's just a touch more involved than your typical jaunt with Vagrant.
Once inside the FreeBSD environment, I found I needed to do some additional setup:
vagrant@:~ % su -
Password: vagrant
root@:~ # pkg update
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo
file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
meta.txz : 100% 944 B 0.9kB/s 00:01
packagesite.txz : 100% 6 MiB 1.5MB/s 00:04
Processing entries: 100%
FreeBSD repository update completed. 26100 packages processed.
All repositories are up to date.
root@:~ # pkg install curl git vim tmux
Whew!
Here's a note on install VirtualBox Guest Additions in FreeBSD. I haven't had to do anything like that yet, but if I ever need too... If that doesn't work, I found (but haven't read) this one too.
Incidentally, I did run into some regular panic / disconnect / shutdown
events. Running a tail -F /var/log/* as root in one terminal eventually
got me this panic right before termination:
Feb 27 05:50:54 kernel: ACPI Error: No installed handler for fixed event - PM_Timer (0), disabling (20160422/evevent-323)
Feb 27 05:50:54 kernel: ACPI Error: No installed handler for fixed event - RealTimeClock (4), disabling (20160422/evevent-323)
Feb 27 05:50:54 kernel: ACPI Error: No handler or method for GPE 01, disabling event (20160422/evgpe-834)
Feb 27 05:50:54 kernel: ACPI Error: No handler or method for GPE 03, disabling event (20160422/evgpe-834)
Feb 27 05:50:54 kernel: ACPI Error: No handler or method for GPE 04, disabling event (20160422/evgpe-834)
Feb 27 05:50:54 kernel: ACPI Error: No handler or method for GPE 05, disabling event (20160422/evgpe-834)
Feb 27 05:50:54 kernel: ACPI Error: No handler or method for GPE 06, disabling event (20160422/evgpe-834)
Feb 27 05:50:54 kernel: ACPI Error: No handler or method for GPE 07, disabling event (20160422/evgpe-834)
A quick google found Bug #209222 in r298793, which in turn
references r298838 as the fix (Apr 2016). I tried adding the
following line to /boot/loader.conf, which is purported to disable ACPI
entirely:
hint.acpi.0.disabled="1"
But that just ended up panicing the kernel on boot, with a message of
panic: running without device atpic requires a local APIC
Le sigh. Upgraded Vagrant to use freebsd/FreeBSD-11.0-STABLE instead of
-CURRENT -- we'll see how that shakes out.
One piece of software that I spend a great deal of time with these days is Hashicorp's Vault secure credentials storage solution. It's a self-contained piece of software, distributed as a binary, with minimal configuration. And they have a FreeBSD 64-bit (AMD64) build!
(Note: remaining commands will be executed from inside the FreeBSD box)
root@:~ # curl -sLO https://releases.hashicorp.com/vault/0.6.5/vault_0.6.5_freebsd_amd64.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 12.6M 100 12.6M 0 0 1279k 0 0:00:10 0:00:10 --:--:-- 1490k
root@:~ # unzip vault_0.6.5_freebsd_amd64.zip
Archive: vault_0.6.5_freebsd_amd64.zip
extracting: vault
root@:~ # mv vault /usr/local/sbin
root@:~ # vault -v
Vault v0.6.5 ('5d8d702f33b5fd965cbe8d6d0728295de813a196')
Now we can get to work on jailing Vault.
Jails come in two flavors: complete and service. Complete jails are the full kit - an entire userland filesystem with all the bells, whistles, an init system, etc. These closely resemble full-blown virtual machines, or almost every Docker container ever built -- Alpine is still a userland.
I'm mostly interested in service jails, which are just chroots on
steroids — put your binaries, their library dependencies, and maybe
some socket files for IPC on the jail fs, and nothing more. No shell. No
cron. No CUPS. No nothin'.
For a Vault service jail, all we need is the Vault binary itself (I think).
root@:~ # mkdir -p /root/jails/vault1
root@:~ # cp /usr/local/sbin/vault /root/jails/vault1
With the filesystem ready to go, let's create the jail:
root@:~ # jail -c persist \
name=vault1 \
path=/root/jails/vault1 \
host=new \
host.hostname=vault \
host.domainname=example.com \
host.hostuuid=49205cb3-8dc6-4b0e-b131-6f0c3e3ee6d8 \
osrelease=JamesBSD
root@:~ # jls
JID IP Address Hostname Path
1 vault /root/jails/vault1
That was easy.
Let's try running the Vault inside of the jail, with jexec:
root@~ # jexec 1 /vault
The first argument to jexec, 1, is our jid, a numerical ID that is
unique to our new jail. I got it off of the jls output. The second
argument is the full path to the command to run, just like with chroot.
It didn't work out so well:
panic: unable to seed random number generator: open /dev/urandom: no such file or directory
goroutine 1 [running]:
github.com/hashicorp/vault/vendor/github.com/gocql/gocql.init.1()
/gopath/src/github.com/hashicorp/vault/vendor/github.com/gocql/gocql/control.go:27+0x2a8
github.com/hashicorp/vault/vendor/github.com/gocql/gocql.init()
/gopath/src/github.com/hashicorp/vault/vendor/github.com/gocql/gocql/uuid.go:252+0x73c
github.com/hashicorp/vault/builtin/logical/cassandra.init()
/gopath/src/github.com/hashicorp/vault/builtin/logical/cassandra/util.go:96+0x57
github.com/hashicorp/vault/cli.init()
/gopath/src/github.com/hashicorp/vault/cli/main.go:52 +0x93
main.init()
/gopath/src/github.com/hashicorp/vault/main.go:12 +0x49
Looks like Vault needs /dev/urandom and we didn't provide it. I figured
it was time for mknod, so I tried this:
root@:~ # mkdir /root/jails/vault1/dev
root@:~ # mknod /root/jails/vault1/dev/urandom c 9 0
Yeah, that didn't work. Strangely, the device didn't have the right major
number (9) once it was created. Also, the mknod(8) manpage carries this
warning at the top:
The mknod utility is deprecated on modern FreeBSD systems.
:/
There's a mount.devfs option to jail, let's try that:
root@:~ # mkdir /root/jails/vault1/dev
root@:~ # jail -c persist \
name=vault2 \
path=/root/jails/vault1 \
host=new \
host.hostname=vault \
host.domainname=example.com \
host.hostuuid=49205cb3-8dc6-4b0e-b131-6f0c3e3ee6d8 \
osrelease=JamesBSD \
mount.devfs
root@:~ # jls
JID IP Address Hostname Path
1 vault /root/jails/vault1
2 vault /root/jails/vault1
(Note we changed the name= parameter as well)
Let's try that jexec again:
root@:~ # jexec 2 /vault
usage: vault [-version] [-help] <command> [args]
Common commands:
delete Delete operation on secrets in Vault
path-help Look up the help for a path
read Read data or secrets from Vault
... etc ...
Success!
Now we need a configuration. Let's pop that in the filesystem at
/root/jails/vault1/etc/vault.conf
root@:~ # mkdir /root/jails/vault1/etc
root@:~ # cat >/root/jails/vault1/etc/vault.conf <<EOF
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = 1
}
backend "file" {
path = "/data"
}
EOF
I'm also going to go ahead and create that /data directory, relative to
the jail root fs:
root@:~ # mkdir /root/jails/vault1/data
Now we can try again:
root@:~ # jexec 2 /vault server -config /etc/vault.conf
Error initializing core: Failed to lock memory: operation not permitted
This usually means that the mlock syscall is not available.
Vault uses mlock to prevent memory from being swapped to
disk. This requires root privileges as well as a machine
that supports mlock. Please enable mlock on your system or
disable Vault from using it. To disable Vault from using it,
set the `disable_mlock` configuration option in your configuration
file.
Drat. We can either disable mlock via config, or we can try to add that root capability into the jail. (Note that this freebsd-questions mailing list post post seems to indicate that you just can't do mlock inside of a jail...)
The mlock(2) man page mentions that the syscall sets errno to EPERM
(which that handy errno util told me was "operation not permitted") when:
security.bsd.unprivileged\_mlockis set to 0 and the caller is not the super-user.
But according to sysctl, it's already set to 1:
root@:~ # sysctl security.bsd.unprivileged_mlock
security.bsd.unprivileged_mlock: 1
So I guess we disable mlock in Vault for now.
root@:~ # cat >/root/jails/vault1/etc/vault.conf <<EOF
disable_mlock = true
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = 1
}
backend "file" {
path = "/data"
}
EOF
And try again:
root@:~ # jexec 2 /vault server -config /etc/vault.conf
Error initializing listener of type tcp: listen tcp4 0.0.0.0:8200: socket:
protocol not supported
Looks like we need to turn on IPv4. From jail(8):
ip4 Control the availability of IPv4 addresses. Possible values
are ``inherit'' to allow unrestricted access to all system
addresses, ``new'' to restrict addresses via ip4.addr, and
``disable'' to stop the jail from using IPv4 entirely.
Setting the ip4.addr parameter implies a value of ``new''.
So let's modify jail 2:
root@:~ # jail -m jid=2 ip4=inherit ip4.addr=10.0.2.15
root@:~ # jls
JID IP Address Hostname Path
1 vault /root/jails/vault1
2 10.0.2.15 vault /root/jails/vault1
root@:~ # jail -m jid=2 ip4=new ip4.addr=10.0.2.15
root@:~ # jls
JID IP Address Hostname Path
1 vault /root/jails/vault1
2 10.0.2.15 vault /root/jails/vault1
root@:~ # jexec 2 /vault server -config /etc/vault.conf
==> Vault server configuration:
Backend: file
Cgo: disabled
Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "",
tls: "disabled")
Log Level: info
Mlock: supported: true, enabled: false
Version: Vault v0.6.5
Version Sha: 5d8d702f33b5fd965cbe8d6d0728295de813a196
==> Vault server started! Log data will stream in below:
Woot!
Unfortunately, this takes over the controlling terminal, and Ctrl-C causes the process to terminate. We need a way to run in the background.
root@:~ # jail -qc persist \
name=vault3 \
path=/root/jails/vault1 \
host=new \
host.hostname=vault \
host.domainname=example.com \
host.hostuuid=49205cb3-8dc6-4b0e-b131-6f0c3e3ee6d8 \
osrelease=JamesBSD \
mount.devfs \
ip4=new \
ip4.addr=10.0.2.15 \
exec.consolelog=/dev/null \
command=/vault server -config /etc/vault.conf &
[1] 1039
root@:~ # curl http://10.0.2.15:8200/v1/sys-status
{"errors":["Vault is sealed"]}
Now we can try to use it.
root@:~ # setenv VAULT_ADDR http://10.0.2.15:8200
root@:~ # vault status
Error checking seal status: Error making API request.
URL: GET http://10.0.2.15:8200/v1/sys/seal-status
Code: 400. Errors:
* server is not yet initialized
root@:~ # vault init
Unseal Key 1: gw1w9cY6y3wDKEzyWQftPgVB/YnwftzIrRI78DidZnMB
Unseal Key 2: xQK/HA+AlFGDOUDXZE17dtbXq3a2tht1x8RyOcXdkIcC
Unseal Key 3: +MKDX+sfBVojXeGAYRLtlf5irVIO//nNwEicMJ3sbD0D
Unseal Key 4: TAOrSa3vLPsUZFEc5OinU1rhMHrNYrHZ51UbB2UCB9EE
Unseal Key 5: ccOXCklwvfC0APBL4bcxsHJUNl51K1Nh4Nn1Dj0z+2sF
Initial Root Token: 2eee2f93-ad7a-72c1-0d39-f5f3dbcdc43c
Vault initialized with 5 keys and a key threshold of 3. Please
securely distribute the above keys. When the Vault is re-sealed,
restarted, or stopped, you must provide at least 3 of these keys
to unseal it again.
Vault does not store the master key. Without at least 3 keys,
your Vault will remain permanently sealed.
Then we're back we we wanted to be:
root@:~ # vault write secret/handshake knock=knock
Success! Data written to: secret/handshake
root@:~ # vault read secret/handshake
Key Value
--- -----
refresh_interval 768h0m0s
knock knock
These were found during research, and may be useful in their own right.
If you need to perform specific tasks from inside the jail such as ping,
traceroute, sockstat and so forth, you need to set the
security.jail.allow_raw_sockets=1 sysctl.
You can install FreeBSD on Linode, apparently.