It’s been long since I posted on this blog. This time I have come up with this new post which takes you through on how to go about creating your own Amazon Machine Image (AMI) for Amazon EC2. Note that there are several publicly available AMI’s on Amazon which one can use for various purposes but sometimes we require to have a AMI of our own which has all the require software / configuration to meet our daily requirements. That is time we need to know how to create our own AMI.
I would like to thank Phil Chen for his excellent post here http://www.philchen.com/2009/02/14/how-to-create-an-amazon-elastic-compute-cloud-ec2-machine-image-ami which I followed and have mentioned below with some extra addition and some modification.
The steps I mention in this article are part of the manual process. The next part to this article will discuss a great tool call boxgrinder which can let you automate this process in a very easy way. I personally use boxgrinder on my machine to deploy brand new AMIs on Amazon EC2.
The first step of course to this tutorial is to have a account on Amazon EC2. It’s pretty easy to get one if you have a credit card 
. Luckily I was provided a already existing account by my Manager. After you have the account on Amazon EC2 you need to login to the EC2 web console and get few things for yourself like:
- S3 ACCESS KEY
- S3 SECRET KEY
- AWS ACCOUNT NUMBER
- EC2 PRIVATE KEY
- EC2 CERTIFICATE
In order to get these click on Account weblink inside the Amazon web console and then on Security Credentials. Once you have get the required data we need to install two packages provided by Amazon on our local system. The tools can work on Mac OSX and Linux, I am not sure about Windows.
I do have a local Linux machine but if I create the AMI on my local machine it will take lot of time to upload it to Amazon (due to slow internet connection). So instead of that I started a instance on EC2 for developing AMI. The major advantage is that it takes only few minutes to upload from their as the download/upload speed provided by Amazon is fantastic.
To start a instance you can use the web console provided by Amazon. I wanted to build a AMI for Centos 5 so I started the 32bit image for Centos 5.4 provided by RightScale free of cost available on EBS (Elastic Block Storage).
The instance will startup in few seconds and you will get a public DNS name for it. You can use one of your security keypair you associated when creating the instance to login as root to the instance.
Once you have logged in you need to set this instance as the development bed for creating AMIs. Download these 2 files:
Extract these 2 zip files inside your home directory for e.g /root or wherever you would like to extract them. Next you need to copy your private key and certificate for the EC2 account inside this running instance. I created a directory ~/.ec2/ and copied them there.
Next we need to setup the environment for our configuration. The following is what I wrote inside my ~/.bashrc
export PATH=$PATH:/root/ec2amitools/bin/:/root/ec2apitools/bin/
export EC2_HOME=/root/ec2apitools/
export EC2_AMITOOL_HOME=/root/ec2amitools/
export EC2_PRIVATE_KEY=~/.ec2/pk-yourprivatekey.pem
export EC2_CERT=~/.ec2/cert-yourcertificate.pem
Note that I extracted the AMI tools inside /root/ec2amitools and API tools inside /root/ec2apitools/ hence those paths in EC2_HOME and PATH.
In order to reflect the changes you need to source the bash profile.
source ~/.bashrc
Now it’s time to create a image file inside the running instance which will basically hold our operating system for the new AMI.
dd if=/dev/zero of=/mnt/ami-centos5.5-32bit-base.img bs=1M count=10240
Next we need to create a filesystem on this image.
/sbin/mke2fs -F -j /mnt/ami-centos5.5-32bit-base.img
We need to now mount this image on some directory. I created a directory /mnt/centos and mounted this image as loopback.
Next we need to create some basic directories inside /mnt/centos/ which are required.
mkdir /mnt/centos/proc
mkdir /mnt/centos/etc
mkdir /mnt/centos/dev
mkdir /mnt/centos/var
mkdir /mnt/centos/var/cache
mkdir /mnt/centos/var/log
mkdir /mnt/centos/var/lock
mkdir /mnt/centos/var/lock/rpm
Also we need to create 3 basic devices inside /mnt/centos/dev
/sbin/MAKEDEV -d /mnt/centos/dev -x console
/sbin/MAKEDEV -d /mnt/centos/dev -x null
/sbin/MAKEDEV -d /mnt/centos/dev -x zero
We also need to create /mnt/etc/fstab file for our image and populate it with the following information which is required for a 32bit image
/dev/sda1 / ext3 defaults 1 1
none /dev/pts devpts gid=5,mode=620 0 0
none /dev/shm tmpfs defaults 0 0
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0
/dev/sda2 /mnt ext3 defaults 0 0
/dev/sda3 swap swap defaults 0 0
Now we need to mount the proc filesystem for our new image.
mount -t proc none /mnt/centos/proc
We will be required to install the base operating system inside this image file. Yum configuration should be created and saved in a file. I saved it in /mnt/yum-ami.conf
[main]
cachedir=/var/cache/yum
keepcache=1
debuglevel=2
logfile=/var/log/yum.log
pkgpolicy=newest
distroverpkg=redhat-release
tolerant=1
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
metadata_expire=1800
[base]
name=CentOS-5 - Base
mirrorlist=http://mirrorlist.centos.org/?release=5&arch=i386&repo=os
#baseurl=http://mirror.centos.org/centos/5/os/i386/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
#released updates
[updates]
name=CentOS-5 - Updates
mirrorlist=http://mirrorlist.centos.org/?release=5&arch=i386&repo=updates
#baseurl=http://mirror.centos.org/centos/5/updates/i386/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
#packages used/produced in the build but not released
[addons]
name=CentOS-5 - Addons
mirrorlist=http://mirrorlist.centos.org/?release=5&arch=i386&repo=addons
#baseurl=http://mirror.centos.org/centos/5/addons/i386/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
#additional packages that may be useful
[extras]
name=CentOS-5 - Extras
mirrorlist=http://mirrorlist.centos.org/?release=5&arch=i386&repo=extras
#baseurl=http://mirror.centos.org/centos/5/extras/i386/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-5 - Plus
mirrorlist=http://mirrorlist.centos.org/?release=5&arch=i386&repo=centosplus
#baseurl=http://mirror.centos.org/centos/5/centosplus/i386/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
#contrib - packages by Centos Users
[contrib]
name=CentOS-5 - Contrib
mirrorlist=http://mirrorlist.centos.org/?release=5&arch=i386&repo=contrib
#baseurl=http://mirror.centos.org/centos/5/contrib/i386/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
Now it’s time to install the base:
yum -c /mnt/yum-ami.conf --installroot=/mnt/centos -y groupinstall Base
This will install around 333 packages. After this is done we will install openssh packages inside the image because using openssh server only we will accessing the instance remotely
yum -c /mnt/yum-ami.conf --installroot=/mnt/centos/ -y install *openssh*
Make sure you put sshd on chkconfig so that it starts automatically when the instance runs:
/usr/sbin/chroot /mnt/centos/ /sbin/chkconfig sshd --add
/usr/sbin/chroot /mnt/centos/ /sbin/chkconfig --level 12345 sshd on
After this is done we need to configure our image for Amazon. First thing is to put kernel modules for Amazon inside the image. This is required in order to make this image run on EC2.
cp -r /lib/modules/2.6.21.7-2.fc8xen/ /mnt/centos/lib/modules/
/usr/sbin/chroot /mnt/centos/ /sbin/depmod -ae 2.6.21.7-2.fc8xen
One of the service which is set on in a default base install is kudzu (the hardware detecting service). We need not start it at all on EC2
/usr/sbin/chroot /mnt/centos /sbin/chkconfig --del kudzu
Next we need to configure openssh server configuration for EC2
vi /mnt/centos/etc/ssh/sshd_config
The two lines which we should make changes are:
UseDNS no
PermitRootLogin without-password
Next we need to setup networking inside this image. The following needs to be done for the same:
vi /mnt/centos/etc/sysconfig/network
NETWORKING=yes
HOSTNAME=localhost.localdomain
vi /mnt/centos/etc/sysconfig/network-scripts/ifcfg-eth0
ONBOOT=yes
DEVICE=eth0
BOOTPROTO=dhcp
When a instance boots in EC2 it needs to get your SSH keys in order to allow sshkey based authentication. We will create a init script for the same in /mnt/centos/etc/init.d/ with the name getssh
#!/bin/bash
# chkconfig: 2345 95 20
# description: getssh
# processname: getssh
#
export PATH=:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin
# Source function library.
. /etc/rc.d/init.d/functions
# Source networking configuration.
[ -r /etc/sysconfig/network ] && . /etc/sysconfig/network
# Check that networking is up.
[ "${NETWORKING}" = "no" ] && exit 1
start() {
if [ ! -d /root/.ssh ] ; then
mkdir -p /root/.ssh
chmod 700 /root/.ssh
fi
# Fetch public key using HTTP
/usr/bin/curl -f \
http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key \
> /tmp/my-key
if [ $? -eq 0 ] ; then
cat /tmp/my-key >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
rm /tmp/my-key
fi
# or fetch public key using the file in the ephemeral store:
if [ -e /mnt/openssh_id.pub ] ; then
cat /mnt/openssh_id.pub >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
fi
}
stop() {
echo "Nothing to do here"
}
restart() {
stop
start
}
# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
*)
echo $"Usage: $0 {start|stop}"
exit 1
esac
exit $?
###END OF SCRIPT
We will add this script to chkconfig and make sure it is executed everytime our instance starts
/bin/chmod +x /mnt/centos/etc/init.d/getssh
/usr/sbin/chroot /mnt/centos/ /sbin/chkconfig --level 34 getssh on
This makes our image ready for EC2. We need to do cleanup now so the image size is reduced:
yum -c /mnt/yum-ami.conf --installroot=/mnt/centos clean all
sync
umount /mnt/centos/proc
umount /mnt/centos
Now its time to bundle the image for EC2.
ec2-bundle-image --image /mnt/ami-centos5.5-32bit-base.img --prefix \
ami-centos5.5-32bit-base --cert ~/.ec2/cert-yourcertificate.pem \
--privatekey ~/.ec2/pk-yourprivatekey.pem \
--user youraccountnumber --destination /mnt/centos/ --arch i386
After the bundle is over we need to upload it to S3 using the following command:
ec2-upload-bundle --manifest /mnt/centos/ami-centos5.5-32bit-base.manifest.xml \
--bucket centos55-cpr-base \
--access-key youraccesskey \
--secret-key yoursecretkey \
--location US
Finally we register this image with EC2 so we can use it to boot a instance
ec2-register -n centos55cpr \
centos55-cpr-base/ami-centos5.5-32bit-base.manifest.xml
Our image is ready to boot. We can now run the instance using this image via following command:
ec2-run-instances youramid \
-k yoursshkey \
--kernel aki-a71cf9ce \
--ramdisk ari-a51cf9cc \
--instance-type m1.small \
--region us-east-1
It takes sometime for the instance to startup (as the image is in S3). To find what public DNS name this instance got we can issue the following command
ec2-describe-instances
After the instance has booted we will login into it using the sshkey we specified while starting the instance. Note that this instance we will be stored in S3 and will act as base for other images which we might plan to create in future. For e.g we have a requirement to have a Apache webserver image based on Centos 5.5 32bit we can utilize this base image and create a EBS volume for the new Apache image. Let’s see how to do that after the base image has booted.
We need to create a EBS volume of size at least 10GB.
ec2-create-volume -s 12 -z us-east-1d
Now we need to wait for few minutes till the volume is created. As soon as the volume creation is finished we attach it to our already running instance (image is in S3).
ec2-attach-volume vol-ea92ef83 -i i-b6dd60dc -d /dev/sdb
The volume id vol-ea92ef83 you can find out from ec2-describe-volumes and the instance id i-b6dd60dc you can find out from ec2-describe-instances.
Now we need to login to our instance and copy the whole root filesystem which is in /dev/sda1 to /dev/sdb using dd command.
dd bs=65536 if=/dev/sda1 of=/dev/sdb
Filesystem check on the EBS volume should be done now:
fsck /dev/sdb
Now we can mount the EBS volume and make any necessary changes we would like to make. For e.g we would like to make a HTTPD server EBS volume we simply follow these steps:
mkdir /mnt/ebs
mount /dev/sdb /mnt/ebs
mount -t proc none /mnt/ebs/proc
yum -c /mnt/yum-ami.conf --installroot=/mnt/ebs -y install httpd
/usr/sbin/chroot /mnt/ebs chkconfig httpd --add
/usr/sbin/chroot /mnt/ebs chkconfig --level 234 httpd on
umount /mnt/ebs/proc
umount /mnt/ebs
Finally we will have to create a snapshot of the EBS volume:
ec2-create-snapshot -d centos55-apache vol-ea92ef83
Now we can build a AMI out of this snapshot which will be stored on EBS volume using the following command
ec2-register -n centos5.5-32bit-httpd \
--architecture i386 \
--block-device-mapping /dev/sda1=snap-a06dbfcb:12:false
You can find out the snapshot id snap-a06dbfcb from ec2-describe-snapshots. The command above will finally generate the AMI id which we can use later to boot our instance out of EBS volume.
ec2-run-instances -k yoursshkeypair \
ami-936289fa \
--region us-east-1 \
--instance-type m1.small
I hope this article will help all the EC2 users out there who plans to roll their own base AMI and then custom software stack specific EBS volumes. In the next article I plan to cover boxgrinder a great tool which helps you do the whole process mentioned above in 1 single command. So stay tuned!
