/home/jeevanullas

Up in the Cloud!

Docker Machine and Eucalyptus

Docker machine is a new project launched by Docker last year. The goal is to launch instances/servers with docker installed and configure the clients to talk them. Most of the documentation is available at:

https://docs.docker.com/machine/

In the last blog about AWS ECS service on-premises with Eucalyptus we highlighted the need for a container instance. In that example we used CoreOS and along with a config file were able to quickly setup an instance on Eucalyptus cloud with docker installed and ECS agent running.

In this blog we make it easy for folks to create instances on Eucalyptus with docker installed and configured to talk to a docker client. We use Ubuntu Trusty as our preferred choice of operating system on the instance.

Before we really got going we tried to find more about the machine.

  • Machine is written in Go lang.
  • It appears to be actively developed and might have lot of moving parts in coming few weeks/months.
  • Machine has the concept of drivers. It has drivers for all major cloud platforms like AWS, Openstack etc. as well as Virtualbox and other virtualization platforms.

Our focus was to work on the AWS driver and make it work against an AWS compatible Eucalyptus cloud. Here are some thoughts on the current driver:

In order to get going with Eucalyptus and Docker machine we had to do the following, note that the goal was to get a POC setup going with minimal features:

  • Write a new driver for Eucalyptus. Basically copying the AWS driver for the time being (we expect AWS driver to change a lot) and modifying the necessary bits. Ideally we had want the AWS driver to work with Eucalyptus clouds.

Following things were done in the Eucalyptus driver:

  • We removed support for EBS volume types
  • We removed support for block device mapping (it appears root device cannot be a partition like /dev/sda1 on Eucalyptus it has to be a raw disk)
  • We removed support for spot instances (not implemented)
  • We removed support for VPC (tech-preview)
  • We modified the code here, making sure that machine checks for the IP address only after it had make sure the instance is running. Without this change we had issues where machine picked up the private IP of the instance because the instance just got launched on the cloud and Eucalyptus takes some time to get a public IP on it. On AWS we did not see any problems.
  • We added new command line parameter to machine that accept the EC2 Endpoint so one can specify which Eucalyptus compute endpoint they would like to point Machine at.

The driver is available here https://github.com/jeevanullas/machine/tree/eucaprovider

In order to compile the binaries one can fork/clone the branch to a local account/system and run script/build from within the source directory. The build script pulls in necessary Go dependencies and generates the binary in the source directory

Copy the binaries to somewhere like /usr/local/bin/ so they are generally available.

Test run of Docker machine on Eucalyptus:

These were the parameters we passed to Docker machine:

$ docker-machine_linux-amd64 -D create \
--driver eucalyptus \
--eucalyptus-emi <your EMI ID> \
--eucalyptus-access-key <your cloud access key> \
--eucalyptus-secret-key <your cloud secret key> \
--eucalyptus-zone <your AZ name> \
--eucalyptus-region eucalyptus \
--eucalyptus-instance-type m1.large \
--eucalyptus-compute-endpoint "EC2_URL of your Eucalyptus"  \
docker-host-euca-jeevan  

Once this is done you could see the nice debug output (thanks to -D flag)

Launching instance...  
creating key pair: docker-host-euca-jeevan  
configuring security group  
creating security group (docker-machine)  
waiting for group (sg-a00aa116) to become available  
configuring security group authorization for 0.0.0.0/0  
authorizing group  with permissions: [{tcp 22 22 0.0.0.0/0} {tcp 2376 2376 0.0.0.0/0}]  
waiting for ip address to become available  
Got the IP Address, it's "192.168.1.2"  
created instance ID i-195071cd, IP address 192.168.1.2, Private IP address  
Settings tags for instance  
Getting to WaitForSSH function...  
generating server cert: /home/jeevanullas/.docker/machine/machines/docker-host-euca-jeevan/server.pem ca-key=/home/jeevanullas/.docker/machine/certs/ca.pem private-key=/home/jeevanullas/.docker/machine/certs/ca-key.pem org=docker-host-euca-jeevan  

As can be seen Machine created a security group call docker-machine on Eucalyptus and authorize port 22/TCP and 2376/TCP to that security group.

2376 is the port that docker daemon listens on. It also created a keypair with the name docker-host-euca-jeevan and copied the private key for this keypair at ~/.docker/machine/machines/docker-host-euca-jeevan/id_rsa if one fancy logging into the instance directly and seeing what is going on.

The machine also generates certificates that are used for TLS based communication between the docker client and the docker daemon.

We took a peak inside the instance and found the following:

root@docker-host-euca-jeevan:~# docker version  
Client version: 1.6.0  
Client API version: 1.18  
Go version (client): go1.4.2  
Git commit (client): 4749651  
OS/Arch (client): linux/amd64  
Server version: 1.6.0  
Server API version: 1.18  
Go version (server): go1.4.2  
Git commit (server): 4749651  
OS/Arch (server): linux/amd64

root@docker-host-euca-jeevan:~# ps aux | grep docker  
root     29796  0.0  1.9 257732  9552 ?        Ssl  11:25   0:01 /usr/bin/docker -d -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver aufs --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=eucalyptus  
root     30081  0.0  0.1  10460   920 pts/1    S+   13:21   0:00 grep --color=auto docker

root@docker-host-euca-jeevan:~# cat /etc/default/docker  
# Docker Upstart and SysVinit configuration file
# Customize location of Docker binary (especially for development testing).
#DOCKER="/usr/local/bin/docker"
# Use DOCKER_OPTS to modify the daemon startup options.
#DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"
# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"
# This is also a handy place to tweak where Docker's temporary files go.
#export TMPDIR="/mnt/bigdrive/docker-tmp"
DOCKER_OPTS='  
-H tcp://0.0.0.0:2376
-H unix:///var/run/docker.sock
--storage-driver aufs
--tlsverify
--tlscacert /etc/docker/ca.pem
--tlscert /etc/docker/server.pem
--tlskey /etc/docker/server-key.pem
--label provider=eucalyptus
'  
root@docker-host-euca-jeevan:~# cat /var/log/upstart/docker.log  
Waiting for /var/run/docker.sock  
INFO[0000] +job init_networkdriver()  
INFO[0000] +job serveapi(unix:///var/run/docker.sock)  
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)  
/var/run/docker.sock is up
INFO[0000] -job init_networkdriver() = OK (0)  
WARN[0000] Your kernel does not support cgroup swap limit.  
INFO[0001] Loading containers: start.  
INFO[0001] Loading containers: done.  
INFO[0001] docker daemon: 1.6.0 4749651; execdriver: native-0.2; graphdriver: aufs  
INFO[0001] +job acceptconnections()  
INFO[0001] -job acceptconnections() = OK (0)  
INFO[0001] Daemon has completed initialization  
INFO[0004] GET /v1.18/version  
INFO[0004] +job version()  
INFO[0004] -job version() = OK (0)  
INFO[0008] GET /v1.18/version  
INFO[0008] +job version()  
INFO[0008] -job version() = OK (0)  
INFO[0014] Received signal 'terminated', starting shutdown of docker...  
INFO[0014] -job serveapi(unix:///var/run/docker.sock) = OK (0)  
INFO[0000] +job init_networkdriver()  
INFO[0000] +job serveapi(tcp://0.0.0.0:2376, unix:///var/run/docker.sock)  
INFO[0000] Listening for HTTP on tcp (0.0.0.0:2376)  
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)  
INFO[0000] -job init_networkdriver() = OK (0)  
WARN[0000] Your kernel does not support cgroup swap limit.  
INFO[0000] Loading containers: start.  
INFO[0000] Loading containers: done.  
INFO[0000] docker daemon: 1.6.0 4749651; execdriver: native-0.2; graphdriver: aufs  
INFO[0000] +job acceptconnections()  
INFO[0000] -job acceptconnections() = OK (0)  
INFO[0000] Daemon has completed initialization  
INFO[6960] GET /v1.18/version  
INFO[6960] +job version()  
INFO[6960] -job version() = OK (0)  
INFO[6962] GET /v1.18/containers/json  
INFO[6962] +job containers()  
INFO[6962] -job containers() = OK (0)  
INFO[6965] GET /v1.18/version  
INFO[6965] +job version()  
INFO[6965] -job version() = OK (0)  

From the machine we could do all the commands that it comes with against the Eucalyptus provider like ls/rm/start/stop/restart/env/ssh/inspect etc.

$ docker-machine_linux-amd64 ls
NAME                      ACTIVE   DRIVER       STATE     URL                         SWARM  
docker-host-euca-jeevan   *        eucalyptus   Running   tcp://192.168.1.2:2376 

$ docker-machine_linux-amd64 env docker-host-euca-jeevan
export DOCKER_TLS_VERIFY="1"  
export DOCKER_HOST="tcp://192.168.1.2:2376"  
export DOCKER_CERT_PATH="/home/jeevanullas/.docker/machine/machines/docker-host-euca-jeevan"  
# Run this command to configure your shell: eval "$(docker-machine_linux-amd64 env docker-host-euca-jeevan)"

We encourage you to take it for a spin against your Eucalyptus cloud and let us know what you think about it.

As usual we keep an eye on exciting new projects coming out of Docker hence next we plan to do something cool with Docker Compose and Docker machine on an Eucalyptus cloud. Stay tuned.

PS: Special thanks to the Docker team for quickly helping us out with issues and pointing at the right direction. The community pretty strong and always available to help.