• Mar 3, 2010 Added call to script ec2-set-defaults that is normally called on ec2 init that sets the locale and apt sources for EC availability Zone

Introduction

Opscode has officially released 0.8.x of Chef. It is now even more fabulous. I’ve been using the pre-release version for the last couple of months and it is rock steady and very powerful. I’ll be having a post soon on how I used it to deploy a pretty complicated cloud stack with multiple Rails/Mysql/Nginx/Unicorn/Postfix apps for front-ends, and a back end made up of a mix of a Clojure/Swarmiji distributed processing swarm, HBase/Hadoop, Redis, RabbitMQ.

But first, I needed to upgrade my Amazon EC2 AMIs for the officially released Chef 0.8.x. I also wanted to try the EBS Boot image as a basis for the AMI.

This is an update to my earlier post, Creating an Amazon EC2 AMI for Opscode Chef 0.8, but now using the official Opscode 0.8.x Gems instead of building your own Gems. A lot of the content is the same, but you can consider this mostly superceding the older one except where mentioned otherwise. This version will use the EBS Boot AMIs as per Eric Hammond’s Tutorial Building EBS Boot AMIs Using Canonical’s Downloadable EC2 Images. Much of this is blog post is taken from Eric’s blog post but in the context of creating a Chef Client base AMI and a Chef Server. Note that Opscode now has their own AMIs, including ones for Chef 0.8.4, but as of this writing, they do not have AMIs for Amazon us-west.

Setup

Prerequisites

On your host development machine (ie your laptop or whatever machine you are developing from) you should have already installed:

• ec2-api-tools and ec2-ami-tools (these assume you have a modern Java run time setup)
• chef-0.8.4 or later chef client gem (which implies the entire ruby 1.8.x and rubygems toolchain)

Set some Shell variables on host machine

Just to make using these instructions as a cookbook, we’ll have some shell variables that you can set once and then all the instructions will use the variables so you can just cut and paste the instructions into your shell.

keypair=id_runa-staging-us-west
fullpath_keypair=~/.ssh/runa/id_runa-staging-us-west
availability_zone=us-west-1a
instance_type=m1.large
region=us-west-1

# Pick one of these two AMIs (Note that it will be different for different Amazon Regions)
# 32bit AMI
origin_ami=ami-fd5100b8
#64bit AMI
origin_ami=ami-ff5100ba

Start up an instance and capture the instanceid

instanceid=$(ec2-run-instances \
  --key $keypair \
  --availability-zone $availability_zone \
  --instance-type $instance_type \
  $origin_ami \
  --region $region  |
  egrep ^INSTANCE | cut -f2)
echo "instanceid=$instanceid"

Wait for the instance to move to the “running” state

while host=$(ec2-describe-instances --region $region "$instanceid" |
  egrep ^INSTANCE | cut -f4) && test -z $host; do echo -n .; sleep 1; done
echo host=$host

This should loop till you see something like:

$ echo host=$host
host=ec2-184-72-2-93.us-west-1.compute.amazonaws.com

Upload your certs

This assumes that your Amazon certs are in ~/.ec2

rsync                            \
 --rsh="ssh -i $fullpath_keypair" \
 --rsync-path="sudo rsync"      \
 ~/.ec2/{cert,pk}-*.pem         \
 ubuntu@$host:/mnt/

Connect to the instance

ssh -i $fullpath_keypair ubuntu@$host

Update the Amazon ec2 tools on the instance

export DEBIAN_FRONTEND=noninteractive
echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
  sudo tee /etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list &&
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873 &&
sudo apt-get update &&
sudo -E apt-get dist-upgrade -y &&
sudo -E apt-get install -y ec2-api-tools

Set some parameters on instance shell environment

Again this makes it easier to cut and paste the instructions.

codename=karmic
release=9.10
tag=server
region=us-west-1
availability_zone=us-west-1a
if [ $(uname -m) = 'x86_64' ]; then
  arch=x86_64
  arch2=amd64
  # You will need to set the aki and ari values base on the actual base AMI you used
  # It will be different for different regions.  These are set for us-west-1
  ebsopts="--kernel=aki-7f3c6d3a --ramdisk=ari-cf2e7f8a"
  ebsopts="$ebsopts --block-device-mapping /dev/sdb=ephemeral0"
else
  arch=i386
  arch2=i386
  # You will need to set the aki and ari values base on the actual base AMI you used
  # It will be different for different regions. These are set for us-west-1
  ebsopts="--kernel=aki-773c6d32 --ramdisk=ari-c12e7f84"
  ebsopts="$ebsopts --block-device-mapping /dev/sda2=ephemeral0"
fi

Download and unpack the latest released Ubuntu server image file

This contains the output of vmbuilder as run by Canonical.

imagesource=http://uec-images.ubuntu.com/releases/$codename/release/unpacked/ubuntu-$release-$tag-uec-$arch2.img.tar.gz
image=/mnt/$codename-$tag-uec-$arch2.img
imagedir=/mnt/$codename-$tag-uec-$arch2
wget -O- $imagesource |
  sudo tar xzf - -C /mnt
sudo mkdir -p $imagedir
sudo mount -o loop $image $imagedir

Bring the packages on the instance up to date

# Allow network access from chroot environment
sudo cp /etc/resolv.conf $imagedir/etc/

# Fix what I consider to be a bug in vmbuilder
sudo rm -f $imagedir/etc/hostname

# Add multiverse
sudo perl -pi -e 's%(universe)$%$1 multiverse%' \
$imagedir/etc/ec2-init/templates/sources.list.tmpl

# Add Alestic PPA for runurl package (handy in user-data scripts)
echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu karmic main" |
sudo tee $imagedir/etc/apt/sources.list.d/alestic-ppa.list
sudo chroot $imagedir \
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571

# Add ubuntu-on-ec2/ec2-tools PPA for updated ec2-ami-tools
echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
sudo tee $imagedir/etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list
sudo chroot $imagedir \
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873

# Upgrade the system and install packages
sudo chroot $imagedir mount -t proc none /proc
sudo chroot $imagedir mount -t devpts none /dev/pts

cat <<EOF > /tmp/policy-rc.d
#!/bin/sh
exit 101
EOF
sudo mv /tmp/policy-rc.d $imagedir/usr/sbin/policy-rc.d

chmod 755 $imagedir/usr/sbin/policy-rc.d
DEBIAN_FRONTEND=noninteractive

# It seems this has to be done to set up the Locale & apt sources
sudo -E chroot $imagedir /usr/bin/ec2-set-defaults

# Update the apt sources and packages
sudo chroot $imagedir apt-get update &&
sudo -E chroot $imagedir apt-get dist-upgrade -y &&
sudo -E chroot $imagedir apt-get install -y runurl ec2-ami-tools

Install Chef Client and other customizations

Install Ruby and needed packages

sudo -E chroot $imagedir apt-get -y install ruby ruby1.8-dev libopenssl-ruby1.8 rdoc ri irb \
build-essential wget ssl-cert git-core rake librspec-ruby libxml-ruby \
thin couchdb zlib1g-dev libxml2-dev emacs23-nox

Install Rubygems

Rubygems will be installed from source since debian/ubuntu try to control rubygems upgrades. If you don’t care you can install it via apt-get install rubygems

cd $imagedir/tmp
wget http://rubyforge.org/frs/download.php/69365/rubygems-1.3.6.tgz
tar zxf rubygems-1.3.6.tgz
cd rubygems-1.3.6
sudo -E chroot $imagedir ruby /tmp/rubygems-1.3.6/setup.rb
cd ..
sudo rm -rf rubygems-1.3.6
sudo -E chroot $imagedir ln -sfv /usr/bin/gem1.8 /usr/bin/gem
sudo -E chroot $imagedir gem sources -a http://gems.opscode.com
sudo -E chroot $imagedir gem sources -a http://gemcutter.org
sudo -E chroot $imagedir gem install chef

Use Opscode Chef Solo Bootstrap to configure the Chef Client

The following will set up all the default paths and directories as well as install and configure runit to start and monitor the chef-client. Originally I shied away from runit, but this time I’m going as Opscode Vanilla as possible and they like runit.

Create the solo.rb file

All of the following files should be done in $imagedir as we are going to have to run this as chroot to $imagedir

Create $imagedir/solo.rb with an editor and put in the following:

file_cache_path "/tmp/chef-solo"
cookbook_path "/tmp/chef-solo/cookbooks"
recipe_url "http://s3.amazonaws.com/chef-solo/bootstrap-latest.tar.gz"

Create the chef.json file

Create $imagedir/chef.json with the following. (set the server_fqdn to the chef server you are using):

{
  "bootstrap": {
    "chef": {
      "url_type": "http",
      "init_style": "runit",
      "path": "/srv/chef",
      "serve_path": "/srv/chef",
      "server_fqdn": "chef-server-staging.runa.com"
    }
  },
  "run_list": [ "recipe[bootstrap::client]" ]
}

Run the chef-solo command

sudo -E chroot $imagedir chef-solo -c solo.rb -j chef.json \
  -r http://s3.amazonaws.com/chef-solo/bootstrap-latest.tar.gz

I had to run it 3 times before it completed with no errors.
After it does work, clean up the chef-solo stuff:

sudo rm $imagedir/{solo.rb,chef.json}

Update the client config file

The Chef Solo Client bootstrap process creates an /etc/chef/client.rb that is not ideal for Amazon EC2. The following will replace that:

mkdir -p /etc/chef
chown root:root /etc/chef
chmod 755 /etc/chef

Put the following in /etc/chef/client.rb:

# Chef Client Config File
# Automatically grabs configuration from ohai ec2 metadata.

require 'ohai'
require 'json'

o = Ohai::System.new
o.all_plugins
chef_config = JSON.parse(o[:ec2][:userdata])
if chef_config.kind_of?(Array)
  chef_config = chef_config[o[:ec2][:ami_launch_index]]
end

log_level        :info
log_location     STDOUT
node_name        o[:ec2][:instance_id]
chef_server_url  chef_config["chef_server"]

unless File.exists?("/etc/chef/client.pem")
  File.open("/etc/chef/validation.pem", "w", 0600) do |f|
    f.print(chef_config["validation_key"])
  end
end

if chef_config.has_key?("attributes")
  File.open("/etc/chef/client-config.json", "w") do |f|
    f.print(JSON.pretty_generate(chef_config["attributes"]))
  end
  json_attribs "/etc/chef/client-config.json"
end

validation_key "/etc/chef/validation.pem"
validation_client_name chef_config["validation_client_name"]

Mixlib::Log::Formatter.show_time = true

Finish creating the new image

Clean up from the building of the image

sudo chroot $imagedir umount /proc
sudo chroot $imagedir umount /dev/pts
sudo rm -f $imagedir/usr/sbin/policy-rc.d

Copy the image files to a new EBS volume, snapshot and register the snapshot

size=15 # root disk in GB
now=$(date +%Y%m%d-%H%M)
prefix=runa-chef-0.8.4-ubuntu-$release-$codename-$tag-$arch-$now
description="Runa Chef 0.8.4 Ubuntu $release $codename $tag $arch $now"
export EC2_CERT=$(echo /mnt/cert-*.pem)
export EC2_PRIVATE_KEY=$(echo /mnt/pk-*.pem)

volumeid=$(ec2-create-volume --region $region --size $size \
  --availability-zone $availability_zone | cut -f2)

instanceid=$(wget -qO- http://instance-data/latest/meta-data/instance-id)

ec2-attach-volume --region $region --device /dev/sdi --instance "$instanceid" "$volumeid"

while [ ! -e /dev/sdi ]; do echo -n .; sleep 1; done

sudo mkfs.ext3 -F /dev/sdi
ebsimage=$imagedir-ebs
sudo mkdir $ebsimage
sudo mount /dev/sdi $ebsimage

sudo tar -cSf - -C $imagedir . | sudo tar xvf - -C $ebsimage
sudo umount $ebsimage

ec2-detach-volume --region $region "$volumeid"
snapshotid=$(ec2-create-snapshot --region $region "$volumeid" | cut -f2)

ec2-delete-volume --region $region "$volumeid"

# This takes a while
while ec2-describe-snapshots --region $region "$snapshotid" | grep -q pending
  do echo -n .; sleep 1; done

ec2-register \
  --region $region \
  --architecture $arch \
  --name "$prefix" \
  --description "$description" \
  $ebsopts \
  --snapshot "$snapshotid"

Afterward

That will get you an AMI that you can now use as a chef-client. You can use the directions from the section Creating a Chef Server from your new Image in the previous article: Creating an Amazon EC2 AMI for Opscode Chef 0.8.

There are other issues using this technique. To quote Flinn:

Achtung! Monkey patching with the above will expose your create method without using an actual post. Imho no big deal, others might be more cautious (CAPTCHAis always an option), ymmv.

See his original article for details on how to use it generally. 

The problem with the monkeypatch that he wrote up is that Rails 2.2.2 changed the function that was monkeypatched so the patch stopped working. Below is code that will work on both pre and post 2.2.2 Rails:

module ActionController
  class AbstractRequest

    def request_method

      if Rails::VERSION::STRING < "2.2.2"

        @request_method ||= begin

          method = (parameters[:_method].blank? ? @env['REQUEST_METHOD'] : parameters[:_method].to_s).downcase

          if ACCEPTED_HTTP_METHODS.include?(method)

            method.to_sym

          else

            raise UnknownHttpMethod, "#{method}, accepted HTTP methods are #{ACCEPTED_HTTP_METHODS.to_a.to_sentence}"

          end

        end

      else

        method = @env['REQUEST_METHOD']

        method = parameters[:_method] unless parameters[:_method].blank?

        HTTP_METHOD_LOOKUP[method] || raise(UnknownHttpMethod, "#{method}, accepted HTTP methods are #{HTTP_METHODS.to_sentence}")
      end

    end

  end
end