Warden

Warden manages isolated, ephemeral, and resource-controlled environments.

The primary goal of Warden is to provide a simple API for managing isolated environments. These isolated environments, or containers, can be limited in terms of CPU usage, memory usage, disk usage, and network access. The only currently supported OS is Linux.

Components

This repository contains the following components:

  • warden: server
  • warden-protocol: protocol definition, used by both the server and clients
  • warden-client: client (Ruby)
  • em-warden-client: client (Ruby’s EventMachine)

Getting Started

This short guide assumes Ruby 1.9 and Bundler are already available. Ensure that Ruby 1.9 has GNU readline library support through the ‘libreadline-dev’ package and zlib support through the 'zlib1g-dev’ package.

Install the right kernel

If you are running Ubuntu 10.04 (Lucid), make sure the backported Natty kernel is installed. After installing, reboot the system before continuing.

$ sudo apt-get install -y linux-image-generic-lts-backport-natty

Install dependencies

$ sudo apt-get install -y build-essential debootstrap

Setup Warden

Run the setup routine, which compiles the C code bundled with Warden and sets up the base file system for Linux containers.

$ sudo bundle exec rake setup[config/linux.yml]

Note: If sudo complains that bundle cannot be found, try sudo env PATH=$PATH to pass your current PATH to the sudo environment.

The setup routine sets up the file system for the containers at the directory path specified under the key server -> container_rootfs_path in the config file config/linux.yml.

Run Warden

$ sudo bundle exec rake warden:start[config/linux.yml]

Interact with Warden

$ bundle exec bin/warden

Implementation for Linux

Isolation is achieved by namespacing kernel resources that would otherwise be shared. The intended level of isolation is set such that multiple containers present on the same host should not be aware of each other’s presence. This means that these containers are given their own Process ID (PID) namespace, network namespace, and mount namespace.

Resource control is handled using Control Groups. Each container is placed in its own control group, where it is configured to use an equal slice of CPU compared to other containers, and the maximum amount of memory it may use is set.

The following sections give a brief summary of the techniques used to implement the Linux backend for Warden. You can find a more detailed description in the root/linux directory of this repository.

Networking

Every container is assigned a network interface which is one side of a virtual ethernet pair created on the host. The other side of the virtual ethernet pair is only visible on the host (from the root namespace). The pair is configured to use IPs in a small and static subnet. Traffic from and to the container can be forwarded using NATs. Additionally, you can filter and shape all traffic as needed, using readily available tools such as iptables.

Filesystem

Every container gets a private root filesystem. This filesystem is created by stacking a read-only filesytem and a read-write filesystem. This is implemented using aufs on Ubuntu versions from 10.04 to 11.10 and overlayfs on Ubuntu 12.04.

The read-only filesystem contains the minimal set of Ubuntu packages and Warden-specific modifications common to all containers. The read-write filesystem stores files overriding container-specific settings when necessary. Because all writes are applied to the read-write filesystem, containers can share the same read-only base filesystem.

The read-write filesystem is created by formatting a large sparse file. Because the size of this file is fixed, the filesystem that it contains cannot grow beyond this initial size.

Difference with LXC

The Linux Containers or LXC project has goals that are similar to those of Warden: isolation and resource control. They both use the same Linux kernel primitives to achieve their goals. In fact, early versions of Warden even used LXC.

The major difference between the two projects is that LXC is explicitly tied to Linux, while you can implement Warden backends for any operating system that implements some way of isolating environments. It is a daemon that manages containers and can be controlled via a simple API rather than a set of tools that are individually executed.

While the Linux backend for Warden was initially implemented with LXC, the current version no longer depends on it. Because Warden relies on a very small subset of the functionality that LXC offers, we decided to create a tool that only implements the functionality we need in under 1k LOC of C code. This tool executes preconfigured hooks at different stages of the container start process, such that required resources can be set up without worrying about concurrency issues. These hooks make the start process more transparent, allowing for easier debugging when parts of this process are not working as expected.

Container Lifecycle

Warden manages the entire lifecycle of containers. The API allows users to create, configure, use, and destroy containers. Additionally, Warden can automatically clean up unused containers when needed.

Create

Every container is identified by its handle, which Warden returns upon creating it. It is a hexadecimal representation of the IP address that is allocated for the container. Regardless of whether the backend providing the container functionality supports networking, Warden allocates an IP address to identify a container.

Once a container is created and its handle is returned to the caller, it is immediately ready for use. All resources will be allocated, the necessary processes will be started and all firewalling tables will have been updated.

If Warden is configured to clean up containers after activity, it will use the number of connections that have referenced the container as a metric to determine inactivity. If the number of connections referencing the container drops to zero, the container will automatically be destroyed after a preconfigured interval. If in the mean time the container is referenced again, this timer is cancelled.

Use

The container can be used by running arbitrary scripts, copying files in and out, modifying firewall rules and modifying resource limits. Refer to the “Interface” section for a complete list of operations.

Destroy

When a container is destroyed – either per user request, or automatically after being idle – Warden first kills all unprivileged processes running inside the container. These processes first receive a TERM signal, followed several seconds later by a KILL signal if they have not yet exited. When these processes have terminated, Warden sends a KILL to the root of the container process tree. Once all resources the container used have been released, its files are removed and it is considered destroyed.

Networking

Interface

Warden uses a line-based JSON protocol to communicate with its clients, which it does over a Unix socket located at /tmp/warden.sock by default. Each command invocation is formatted as a JSON array, where the first element is the command name and subsequent elements can be any JSON object. Warden responds to the following commands:

create [CONFIG]

Creates a new container.

Returns the container handle, which is used to identify the container.

The optional CONFIG parameter is a hash that specifies configuration options used during container creation. The supported options are:

bind_mounts

If supplied, this specifies a set of paths to be bind mounted inside the container. The value must be an array. The elements in this array specify the bind mounts to execute, and are executed in order. Every element must be of the form:

[
  # Path in the host filesystem
  "/host/path",

  # Path in the container
  "/path/in/container",

  # Optional hash with options. The `mode` key specifies whether the bind
  # mount should be remounted as `ro` (read-only) or `rw` (read-write).
  {
    "mode" => "ro|rw"
  }
]

grace_time

If specified, this setting overrides the default time period during which no client references a container before the container is destroyed. The value can either be the number of seconds as floating point number or integer, or the null value to completely disable the grace time.

disk_size_mb

If specified, this setting overrides the default size of the container scratch filesystem. The value is expected to be an integer number.

spawn HANDLE SCRIPT [OPTS]

Run the script SCRIPT in the container identified by HANDLE.

Returns a job identifier that can be used to reap the job exit status at some point in the future. Also, the connection that issued the command may disconnect and reconnect later, but still be able to reap the job.

The optional OPTS parameter is a hash that specifies options modifying the command being run. The supported options are:

privileged

If true, this specifies that the script should be run as root.

link HANDLE JOB_ID

Reap the script identified by JOB_ID, running in the container identified by HANDLE.

Returns a three-element tuple containing the integer exit status, a string containing its STDOUT and a string containing its STDERR. These elements may be null when they cannot be determined (e.g. the script could not be executed, was killed, etc.).

stream HANDLE JOB_ID

Stream STDOUT and STDERR of scripts identified by JOB_ID, running in the container identified by HANDLE.

Returns a two-element tuple containing the type of stream: STDOUT or STDERR as the first element, and a chunk of the stream as the second element. Returns an empty tuple when no more data is available in the stream.

limit HANDLE (mem) [VALUE]

Set or get resource limits for the container identified by HANDLE.

The following resources can be limited:

  • The memory limit is specified in number of bytes. It is enforced using the control group associated with the container. When a container exceeds this limit, the kernel kills one or more of its processes. Additionally, the Warden is notified that an OOM happened. The Warden subsequently tears down the container.

net HANDLE in

Forward a port on the external interface of the host to the container identified by HANDLE.

Returns the port number that is mapped to the container. This port number is the same on the inside of the container.

net HANDLE out ADDRESS[/MASK][:PORT]

Allow traffic from the container identified by HANDLE to the network address specified by ADDRESS. Additionally, the address may be masked to allow a network of addresses, and a port to only allow traffic to a specific port.

Returns ok.

copy HANDLE in SRC_PATH DST_PATH

Copy the contents at SRC_PATH on the host to DST_PATH in the container identified by HANDLE.

Returns ok.

File permissions and symbolic links are preserved, while hardlinks are materialized. If SRC_PATH contains a trailing /, only the contents of the directory are copied. Otherwise, the outermost directory, along with its contents, is copied. The unprivileged user will own the files in the container.

copy HANDLE out SRC_PATH DST_PATH [OWNER]

Copy the contents at SRC_PATH in the container identified by HANDLE to DST_PATH on the host.

Returns ok.

Its semantics are identical to copy HANDLE in except with respect to file ownership. By default, root owns the files on the host. If you supply the OWNER argument (in the form of USER:GROUP), files on the host will be chowned to this user/group after the copy has completed.

stop HANDLE

Stop processes running inside the container identified by HANDLE.

Returns ok.

Because this takes down all processes, unfinished scripts will likely terminate without an exit status being available.

destroy HANDLE

Stop processes and destroy all resources associated with the container identified HANDLE.

Returns ok.

Because everything related to the container is destroyed, artifacts from running an earlier script should be copied out before calling destroy.

Configuration

You can configure Warden by passing a configuration file when it starts. Refer to config/linux.yml in the repository for an example configuration.

System prerequisites

Warden runs on Ubuntu 10.04 and higher.

A backported kernel needs to be installed on 10.04. This kernel is available as linux-image-server-lts-backport-natty (substitute generic for server if you are running Warden on a desktop variant of Ubuntu 10.04).

Other dependencies are:

  • build-essential (for compiling the Warden C bits)
  • debootstrap (for bootstrapping the base filesystem of the container)
  • quota (for managing file system quotas)

Run rake setup for further Warden bootstrapping.

Hacking

The included tests create and destroy real containers, so they require system prerequisites to be in place. They need to be run as root if the backend to be tested requires it.

See root/<backend>/README.md for backend-specific information.