If I wanted to test if this value is set I couldn’t safely just say:

because in some cases

or

Up until recently a friend lent me a Virtual Machine in he Cloud for my Blog. I didn’t have to do anything to manage [...]]]> Updates

September 9, 2011

Included the latest Chef Knife ec2 server create argument that sets the EBS Volume to not be deleted on the termination of the EC2 Instance

Intro

Up until recently a friend lent me a Virtual Machine in he Cloud for my Blog. I didn’t have to do anything to manage it. But his company is no longer supporting those machines so I had to move my blog.
Right around that time Amazon announced their Micro Instances at a very low price. I also wanted to try out the new Opscode Chef knife commands that bootstrap an EC2 instance from scratch as well as their Chef Server SaaS. So this was a good reason to combine all these to create my new Blog Instance. And now Amazon even offers the ability to have a single micro instance free for a year! (You still have to pay for I/O charges but they are really cheap compared to the instance charges, unless you have a blog that is too popular, but then you’ll need a bigger server anyway)
Spoiler Alert: It was way too easy and no problem at all! (Though I did end up having to write a few support cookbooks like vsftpd, but now you don’t have to)

Some Assumptions for this post

• You are using a *nix platform for your local development (ie your laptop is a Mac, Linux, *BSD or equivalent) and that your target server you want to deploy to is a relatively recent Ubuntu Linux.
• You have or will install git client on your local development box
• You followed the directions or have done the equivalent of the instructions in the Opscode How to Get Started pages as noted below

Set up an Account on Amazon Web Services

If you don’t already have an Amazon EC2 Account, go to the Amazon Web Services page and click on the Sign Up Now button. Create all your user info and then Sign Up for Amazon EC2. You’ll need to put in  credit card info at this point since you’ll need to pay for the EC2 instance you’ll be using shortly. After you complete your signup, you’ll need to get your credentials at the AWS Security Credentials page.  Copy down your Access Key ID and click on Show under the Secret Access Key and get that as well. You will need these values to put into your knife.rb file that you will get to in the following steps.

Get an Opscode Platform Account

Its free and easy. Just go to the Opscode Platform Signup page. Fill in your information and submit. There is no cost for up to 5 client nodes. Once you set up and confirm your account you can go thru the How to Get Started pages which includes how to set up your client development machine (installing Chef Client, Knife and various dependencies) as well as downloading your private key, organization key and your Knife Configuration File. You should go thru all 5 steps of the Getting Started section. And please do follow their examples of using git. The rest of this post assumes you have git installed and will use it for your own repository even if you don’t push it to an upstream git repository.

Once you have completed that you will be ready to use the remaining steps of this blog post. The remaining steps will assume you put your chef-repo in the same location as the Opscode instructions suggested (~/chef-repo). If you put it somewhere else, just adjust your path to your chef-repo as appropriate.

It also assumes you got your private user key (your_user_name.pem) and organization validator key (your_organization-validator.pem) and knife.rb in Section 3 of How to Get Started: Setting Up a Chef Client. In that section you ran the command knife configure client ./client-config inside your ~/chef-repo/ directory. That will have created ~/chef-repo/.chef and put the keys and knife.rb in that directory.

For the use of this blog post, we will use the username: rberger_test and organization name: install_wordpress. So the private user key name for this example will be: rberger_test.pem and the organization validator key will be called install_wordpress-validator.pem. You should copy your keys someplace that you will not loose outside of ~/chef-repo. There are ways to create new ones, but its always easier not to have to. Bottom line, is its expected that your keys and the knife.rb will be in your ~/chef-repo/.chef directory at this point.

Set up your Development Environment

Your development environment is your home or work computer/laptop. Its the machine that is local to you. It is on this machine that you put together your Cookbooks. From here you push your cookbooks to the Opscode Chef Server, issue the commands to configure AWS and launch your AWS instances.

Tweak up your chef-repo

I like to keep the “standard” chef recipes that get downloaded from git or from cookbook.opscode.com in their own directory (called cookbooks) and all the cookbooks I create or highly modify in another directory (site-cookbooks). In Step 2 of the How to Get Started: Setting Up Your User Environment, they had you create a ~/chef-repo directory and populate it from git or from a tar ball. You should add the site-cookbooks directory to your ~/chef-repo. We’re also going to add an empty README.md to the site-cookbooks directory so when we create our own git repository that directory will be there (an empty directory will not be added to a git repository)

cd ~/chef-repo
mkdir site-cookbooks
echo "Directory for customized cookbooks" > site-cookbooks/README.md

You will probably also not want to include your .chef directory with all your keys in what gets uploaded to any outside chef repository. If you are just keeping things local, you can skip this step. Edit ~/chef-repo/.gitignore and add .chef to the file on its own line. You might also want to add client-config to .gitignore as well as any temporary or backup file suffixes you might have. For instance if you use Emacs, you would add ~* (emacs backup files suffix), the .DS_Store which is something left by the Mac filesystem,  .rake_test_cache which is left around by Rake and metadata.json which is a file generated by chef. My .gitignore looks like:

.chef
client-config
*~
.DS_Store
.rake_test_cache
metadata.json

If you created the ~chef-repo from the git clone of the Opscode repository, you’ll want to get rid of the git configuration and history from the cloning of the Opscode chef-repo and create your own git repository:

rm -rf .git
git init
git add -A
git commit -a -m "Created my own basic chef-repo"

The above commands will have removed the old git config that came when you did the git clone http://github.com/opscode/chef-repo.git command as part of the Opscode How to Get Started pages. The git init, add and commit will create a new local git repository for your own use not connected to the opscode repository. You can then add a remote repository if you want to be able to push your repository and future changes to another git repository such as github.com.

Updating your knife.rb file with Amazon Credentials

Add the following lines to the end of your ~/chef-repo/.chef/knife.rb file. You should have gotten your AWS Access Key and Secret Access key when you signed up to Amazon Web Services, but you can always go back and get it at AWS Security Credentials page. Your final knife.rb should look something like this, except the various items that are customized to your setup. In the example below rberger_test would be replaced by your Opscode User name and install_wordpress would be replaced by your Opscode Organization name that was used when you went thru the Section 3 of the Opscode How to Get Started: Setting Up a Chef Client.

current_dir = File.dirname(__FILE__)
log_level                :info
log_location             STDOUT
node_name                "rberger_test"
client_key               "#{current_dir}/rberger_test.pem"
validation_client_name   "install_wordpress-validator"
validation_key           "#{current_dir}/install_wordpress-validator.pem"
chef_server_url          "https://api.opscode.com/organizations/install_wordpress"
cache_type               'BasicFile'
cache_options( :path => "#{ENV['HOME']}/.chef/checksums" )
cookbook_path            ["#{current_dir}/../cookbooks", "#{current_dir}/../site-cookbooks"]
knife[:aws_access_key_id]     = "Your Access Key"
knife[:aws_secret_access_key] = "Your Secret Access Key"

You can test that your knife.rb is setup enough to access AWS by issuing the command

knife ec2 server list

And you should see something like this (just the heading and no instances unless you’ve launched some EC2 instances earlier:

Instance ID      Public IP        Private IP       Flavor        Image        Security Groups  State

Get the Appropriate Cookbooks

We’ll get cookbooks using the knife command and the cookbooks.opscode.com web service. We’ll be using the following cookbooks:

• chef
• apache2
• mysql
• openssl
• php
• postfix
• sudo
• users
• vsftpd
• wordpress

Use the knife command on your local development machine to pull down the cookbooks you need. The command we’re using (knife cookbook site vendor COOKBOOK) will automatically download the cookbooks and install them in the ~/chef-repo/cookbooks directory. It will also check them into your git repository as a vendor branch (Stay on the master branch at least until you have installed all the cookbooks).

cd ~/chef-repo
knife cookbook site vendor chef -d
knife cookbook site vendor apache2 -d
knife cookbook site vendor mysql -d
knife cookbook site vendor openssl -d
knife cookbook site vendor php -d
knife cookbook site vendor postfix -d
knife cookbook site vendor sudo -d
knife cookbook site vendor users -d
knife cookbook site vendor vsftpd -d
knife cookbook site vendor wordpress -d

Those commands will download all the cookbooks and any other cookbook dependencies they may have into your ~/chef-repo/cookbooks directory and check each one in as a git branch in your repo. If you do an ls on your ~/chef-repo/cookbooks directory you should see:

README.md       bluepill        couchdb         java            php             runit           users           xml
apache2         build-essential daemontools     mysql           postfix         sudo            vsftpd          zlib
apt             chef            erlang          openssl         rabbitmq_chef   ucspi-tcp       wordpress

And if you do a git branch you should see your master branch as the current and a chef-vendor- for each of the cookbooks you installed:

  chef-vendor-apache2
  chef-vendor-apt
  chef-vendor-bluepill
  chef-vendor-build-essential
  chef-vendor-chef
  chef-vendor-couchdb
  chef-vendor-daemontools
  chef-vendor-erlang
  chef-vendor-java
  chef-vendor-mysql
  chef-vendor-openssl
  chef-vendor-php
  chef-vendor-postfix
  chef-vendor-rabbitmq_chef
  chef-vendor-runit
  chef-vendor-sudo
  chef-vendor-ucspi-tcp
  chef-vendor-users
  chef-vendor-vsftpd
  chef-vendor-wordpress
  chef-vendor-xml
  chef-vendor-zlib
* master

If you ever want to update these standard cookbooks,  you can just redo the knife cookbook site vendor Cookbook command.

Create site-cookbooks to extend standard cookbooks

It is standard practice to put the official cookbooks in the ~chef-repo/cookbooks directory, as we just did in the previous step. Any cookbook overrides, extensions or custom cookbooks go into the ~chef-repo/site-cookbooks directory. If you create a cookbook directory in ~chef-repo/site-cookbooks with the same name as a cookbook in the ~chef-repo/cookbooks directory, the files, templates and/or recipes in the ~chef-repo/site-cookbook directory will override the matching files, templates and/or recipes in the cookbook of the same name in the ~chef-repo/cookbooks directory. We will now extend two of the cookbooks; users and wordpress.

Extend the Sudo cookbook so its suitable for EC2

The standard sudo cookbook creates a sudoers file that requires passwords to activate sudo. Most EC2 environments do not allow passwords for logins and require that you login only with ssh keys. So we need to modify the Sudo cookbook to create the sudoers file with the NOPASSWORD flag set for all the users we want to have sudo powers. We just need to override the template file used in the standard sudo cookbook.

First have to make a directory for the new template in your site-cookbooks directory:

mkdir -p site-cookbooks/sudo/templates/default

Copy the following into site-cookbooks/sudo/templates/default/sudoers.erb:

#
# /etc/sudoers
#
# Generated by Chef for
#

Defaults !lecture,tty_tickets,!fqdn

# User privilege specification
root  ALL=(ALL) ALL

 ALL=(ALL) NOPASSWD:ALL

# Members of the sysadmin group may gain root privileges
%sysadmin ALL=(ALL) NOPASSWD:ALL

# Members of the group '' may gain root privileges
% ALL=(ALL) NOPASSWD:ALL

Fix a bug in the latest version of the Standard Mysql Cookbook

As I was writing this post, Opscode came out with a new version of the Mysql Cookbook that seems to have a bug with the Chef Client version 0.9.12. It may be fixed by the time you read this. If you are running Chef 0.9.12, check for line 59 of cookbooks/mysql/recipes/client.rb. Change

if platform_version.to_f >= 5.0

to:

if node.platform_version.to_f >= 5.0

Extend the WordPress cookbook to do some custom actions

We need to do a few custom actions after we install wordpress. The main one being to change the onwnership of the wordpress directory and most of the files to the user blog.

We need to add a user named blog that has its home directory the same as the wordpress directory. We will use this blog user to do automatic updates to wordpress. It will use vsftpd for secure ftp and will have only access to the wordpress directory.

We also need to add a swap file to the server. We could create a new cookbook to hold this as its not really wordpress related, but because this is such a simple system, we will just add a new recipe to wordpress to handle these miscellaneous actions.

Create a recipe to add the blog user and change ownership of the wordpress directory

First make the directories in site-cookbooks for extending the wordpress cookbook:

mkdir -p site-cookbooks/wordpress/recipes
mkdir -p site-cookbooks/wordpress/attributes
mkdir -p site-cookbooks/wordpress/templates/default

Create and edit the file site-cookbooks/wordpress/attributes/wordpress.rb and put the following in it (note, this must have a different name than the one used in the standard wordpress cookbook templates directory):

default[:wordpress][:blog_updater][:username] = "blog"

::Chef::Node.send(:include, Opscode::OpenSSL::Password)

default[:wordpress][:blog_updater][:password] = secure_password
# hash set by recipe or manually using makepasswd
default[:wordpress][:blog_updater][:hash] = nil

# For creating the swap partition. Swap_size is in GB
default[:wordpress][:gb_swap_size] = 2
default[:wordpress][:swap_file] = "/swap_file"

This will set the [:wordpress][:blog_updater] to be the name “blog”. This is the default for the username that will have the ability to use vsftpd to update wordpress and its plugins. We actually override this in the wordpress.rb role file. But we put a default here as well for good practice (ie. the cookbook will work even if someone doesn’t override the value in a role).

The ::Chef::Node.send(:include, Opscode::OpenSSL::Password) line is there so we can use the Chef mechanism to create an auto-generated password (secure_password). We then use that mechanism to set the default password for the blog_updater.

Create and edit site-cookbooks/wordpress/recipes/blog_user.rb. put the following as the contents:

# Get the password cryptographic hash for node[:wordpress][:blog_updater][:password
package "makepasswd"
package "libshadow-ruby1.8"
if node[:wordpress][:blog_updater][:hash].nil? || node[:wordpress][:blog_updater][:hash].empty?
  cmd = "echo #{node[:wordpress][:blog_updater][:password]} | /usr/bin/makepasswd --clearfrom=- --crypt-md5 |awk '{ print $2 }'"
  ruby_block "create_blog_updater_pw" do
    block do
      node.set[:wordpress][:blog_updater][:hash] = `#{cmd}`.chomp
    end
    action :create
  end
end

# Create the blog_updater user with their home directory being the wordpress directory and the group as the same group as the Apache runtime group
user "#{node[:wordpress][:blog_updater][:username]}" do
  home "#{node[:wordpress][:dir]}"
  gid "#{node[:apache][:user]}"
  shell "/bin/bash"
  supports :manage_home => true
  unless node[:wordpress][:blog_updater][:hash].nil? || node[:wordpress][:blog_updater][:hash].empty?
    password "#{node[:wordpress][:blog_updater][:hash]}"
  end
end

# Change the ownership of the wordpress directory so that the blog user can update
execute "chown wordpress home for blog user" do
  cwd "#{node[:wordpress][:dir]}"
  user "root"
  command "chown -R #{node[:wordpress][:blog_updater][:username]}:#{node[:apache][:user]} #{node[:wordpress][:dir]}"
  not_if { node[:wordpress][:dir].nil? || node[:wordpress][:dir].empty? || (not File.exists?(node[:wordpress][:dir])) }
end

The above code will create the blog_user as a Linux user on the target system and set its home directory to be the wordpress directory. This is to make it work with vsftpd.

Create a template to override the default wordpress apache config

The standard WordPress cookbook sets the Apache Server Name the FQDN of the EC2 Public DNS and sets the Server Aliases to the EC2 FQDN Private DNS name. This is pretty useless. We would like to have the cookbook set the Server Alias to the FQDN’s based on our own DNS names. To do this without overriding the whole standard WordPress cookbook, we can override one template and name it: site-cookbooks/wordpress/templates/default/wordpress.conf.erb.

  ServerName
  ServerAlias
  DocumentRoot

  >
    Options FollowSymLinks
    AllowOverride FileInfo
    Order allow,deny
    Allow from all

    Options FollowSymLinks
    AllowOverride None

  LogLevel info
  ErrorLog /-error.log
  CustomLog /-access.log combined

  RewriteEngine On
  RewriteLog /-rewrite.log
  RewriteLogLevel 0

The key changes are the ServerAlias line where we now add the @node[:wordpress][:server_aliases] will add any aliases specified by this attribute which we set in the wordpress.rb role file. We also change the AllowOverride to FileInfo for the docroot

Create a recipe to add a swap file to the server

The t1.micro instance only has 612MB of RAM. You can easily run out of that with a WordPress blog. So we have a recipe to add a swap file system utilizing some space the EBS  disk Volume. This recipe creates a 2GB file called /swap_file  using dd and then uses the mkswap and swapon commands to make that file into a swap partition. The recipe also updates the /etc/fstab file so that the swap file will be mounted again if the instance reboots.

Create and edit the file site-cookbooks/wordpress/recipes/add_swap.rb with the following content:

mb_block_size = 100
count = (node[:wordpress][:gb_swap_size] * 1024) / mb_block_size
bash "add_swap" do
  user "root"
  code < "#{node[:wordpress][:swap_file]}"
  )
end

Create and edit the file site-cookbooks/wordpress/templates/default/fstab.erb and put the following content:

# /etc/fstab: static file system information.
#
proc                   /proc           proc   nodev,noexec,nosuid     0       0
      none            swap   sw                      0       0
/dev/sda1              /               ext3   defaults                0       0
/dev/sda2              /mnt            auto   defaults,nobootwait,comment=cloudconfig 0       0

Create WordPress Role

This example will use a single role named wordpress. Use your favorite editor to create a file in your repo with the path roles/wordpress.rb with the following contents (Substitute your domain for ibd.com and change the hostnames such as test and wordpress-test to names appropriate for your blog. Replace rberger_test with the userid you want to use to log into your server via ssh):

name "wordpress"
description "Blog using wordpress"
recipes "apt", "build-essential", "chef::client_service", "users::sysadmins",
        "sudo", "postfix", "mysql::server", "wordpress", "wordpress::blog_user",
        "wordpress::add_swap", "vsftpd"

override_attributes(
  "postfix" => {"myhostname" => "test.ibd.com", "mydomain" => "ibd.com"},
  "authorization" => {
    "sudo" => {
      "groups" => [],
      "users" => ["rberger_test", "ubuntu"]
    }
  },
  "wordpress" => {
     "server_aliases" => %w(test.ibd.com wordpress-test.ibd.com),
     "version" => "3.0.4",
     "checksum" => "c68588ca831b76ac8342d783b7e3128c9f4f75aad39c43a7f2b33351634b74de",
     "blog_updater" => {
       "username" => "blog",
       "password" => "big-secret"
     }
   },
   "vsftpd" => {"chroot_users" => %w(blog)}
)

The recipes line will be used to determine which cookbook/recipes (order is important) should be loaded by Chef when the chef-client is run on your new server.

• apt: Configures various APT components on Debian-like systems.
• build-essential: Installs C compiler / build tools
• chef::client_service: Sets up a Chef client daemon to run periodically
• users::sysadmins: Creates users with ssh authorized keys. Requires a databag to be configured with users info
• sudo: Installs sudo and configures the /etc/sudoers file
• postfix: Installs and configures postfix for outgoing email
• mysql::server: Installs & configures packages required for mysql servers
• wordpress: Installs and configures WordPress according to the instructions at http://codex.wordpress.org/Installing_WordPress
• wordpress::blog_user: Custom add-on recipe to add a user named “blog” to use with vsftpd for automatic wordpress and plugin updates
• wordpress::add_swap: Custom add-on recipe to add a swap partition to the instance
• vsftpd: Very Basic installation and configuration of vsftpd to support Secure (SSL) SFTP

The override_attributes are used to configure various cookbooks.

• postfix – Parameters for the postfix cookbook. Mainly sets the host and domain name to be meaningful
• authorization – Configures the sudo cookbook. Tells which users and groups should have sudo capability
• wordpress Some of these override values in the base cookbook and others for the site-cookbook version
  • server-aliases – Sets aliases for the blog name. Will be used as serveralias names in the apache config.
  • version – The version of wordpress to download.
  • checksum – The checksum of the tar image of the wordpress download.
  • blog_updater- Info needed to create a user that will do auto updates to wordpress via vsftps
    • username – The username of the user
    • password – The password to create for the user
• vsftpd – Sets what user should be allowed to access via ftp and have their home directory chroot’d (should be the same as wordpress-blog_updater).

Upload the cookbooks and roles to Opscode Chef Platform

Run the following commands while you are in ~/chef-repo. This will upload the wordpress role and all the cookbooks in your chef-repo to your account on the Opscode Chef Platform:

knife role from file roles/wordpress.rb
knife cookbook upload -a

Create the Users databag

The users cookbook will take info from Opscode Chef Server Data Bag names users. There can be an item for each user that you want to create a login for. The standard Opscode users cookbook expects the users set up in the data bags to be in the group sysadmin and have the ability to sudo and gain root powers.

We’ll need to create an item for each user you would like to have on your system. I suggest you make at least one for yourself. Here is the data bag I used for my setup. I don’t show the ssh key. You’ll have to substitute your own public ssh key for <your public ssh key> that you will use to ssh to the server. Its a requirement that you have an ssh key as described in the next section on the sudo cookbook.

Here is the JSON representation of my user data bag item. Create a directory users in ~/chef-repo/data_bags/users and put the following JSON in the file ~/chef-repo/users/.json (where is the username you want to have on the target system. The id will be the name of the item in the data bag and what will become your username  (in this case rberger_test) You will also need to include the public ssh key you want associated with this user. You will need to have created a ssh keypair (private and public) locally using something like ssh-keygen. You don’t really need the openid. You should be able to set that to an empty string (“”):

{
  "id": "rberger_test",
  "comment": "Robert J. Berger",
  "uid": 2001,
  "groups": "sysadmin",
  "shell": "/bin/bash",
  "openid": "rberger_test.myopenid.com",
  "ssh_keys": ""
}

You will need to create the users databag and then upload your version of the user JSON (rberger_test.json in the example) to the Chef server with the following commands:

knife data bag create users
knife data bag from file users data_bags/users/rberger_test.json

With Amazon EC2 instances its best to only allow access without passwords using ssh keys. Since the login is protected by ssh keys, and you don’t have passwords associated with the users, you need to make sure sudo is set up to allow invoking sudo for specific users (sysadmins) without a password. The users cookbook creates such a user based on the users data bag. But the sudo cookbook does not set up sudoers to support not having password. We will modify the sudoers.erb template later. Make sure you don’t deploy without this modification as the default sudo cookbook will make it impossible to sudo on an EC2 instance after its run.

Configure AWS

You can do most of the following by using the a GUI web app such as Amazon’s AWS console, the Firefox plugin ElasticFox other such GUI  tools or the command line ec2-api-tools. For now, we’ll show how to do this with the Amazon AWS Console.

Set up Security Group

Add a WordPress group, that enables  ssh, http and https. You should open at least http and https to all IP addresses (represented by Source IP: 0.0.0.0/0) You can decide to open up ssh to every IP or just to your own development network or host. In this example we’ll open it up to the world. Note: by default ping (ICMP) is not enabled so you can not ping your instance. You can enable ping by adding a line where it doesn’t matter what is in Connection Method, Protocol is ICMP, From Port and To Port is set to -1 and Source IP is 0.0.0.0/0.

Navigate to Security Groups and Click on Create Security Group

Enter the name and description of the Security Group

Set the Ports that are to be enabled (Select the Connection Method, enter the Source IP, and click Save)

Generate an SSH Key Pair for accessing your instance[s]

You need to use the Amazon Key Pair generator to generate a key that will be used to make initial ssh connections to your new instances after they are created. You can also do this on the AWS Management Console’s EC2 Key Pairs page:

Navigate to the Key Pairs page and click on Create Key Pair

You can name the key pair anything, but you may want to use this key pair to access this and future instances, so you might want to name it something general like aws-east. Here we’re going to name it something more specific: aws-wordpress just for this example.

Enter the name for the key

After the key pair is created, make sure to save the private key that is downloaded automatically

At this point a file named asw-wordpress.pem will have been downloaded by your browser. Make sure not to loose it! Put it into your ~/.ssh directory and chmod it to 0600:

chmod 0600 ~/.ssh/aws-wordpress.pem

The final Key Pairs page on the AWS Management Console should look something like:

Final Key Pair Page

Create the Instance and Bootstrap Chef on the Instance

The Chef Knife command has the ability to launch EC2 (and other cloud) instances. This process automatically installs chef and all its dependencies after the instance is created. If all goes well, it then loads and executes your roles and cookbooks on the instance creating your server.

You can see what options are available to this command:

# knife ec2 server create --help
knife ec2 server create (options)
    -Z, --availability-zone ZONE     The Availability Zone
    -A, --aws-access-key-id KEY      Your AWS Access Key ID
    -K SECRET,                       Your AWS API Secret Access Key
        --aws-secret-access-key
        --user-data USER_DATA_FILE   The EC2 User Data file to provision the instance with
        --bootstrap-version VERSION  The version of Chef to install
    -N, --node-name NAME             The Chef node name for your new node
        --server-url URL             Chef Server URL
    -k, --key KEY                    API Client Key
        --color                      Use colored output
    -c, --config CONFIG              The configuration file to use
        --defaults                   Accept default values for all questions
    -d, --distro DISTRO              Bootstrap a distro using a template
        --ebs-no-delete-on-term      Do not delete EBS volumn on instance termination
        --ebs-size SIZE              The size of the EBS volume in GB, for EBS-backed instances
    -e, --editor EDITOR              Set the editor to use for interactive commands
    -E, --environment ENVIRONMENT    Set the Chef environment
    -f, --flavor FLAVOR              The flavor of server (m1.small, m1.medium, etc)
    -F, --format FORMAT              Which format to use for output
    -i IDENTITY_FILE,                The SSH identity file used for authentication
        --identity-file
    -I, --image IMAGE                The AMI for the server
        --no-color                   Don't use colors in the output
    -n, --no-editor                  Do not open EDITOR, just accept the data as is
        --no-host-key-verify         Disable host key verification
    -u, --user USER                  API Client Username
        --prerelease                 Install the pre-release chef gems
        --print-after                Show the data after a destructive operation
        --region REGION              Your AWS region
    -r, --run-list RUN_LIST          Comma separated list of roles/recipes to apply
    -G, --groups X,Y,Z               The security groups for this server
    -S, --ssh-key KEY                The AWS SSH key id
    -P, --ssh-password PASSWORD      The ssh password
    -x, --ssh-user USERNAME          The ssh username
    -s, --subnet SUBNET-ID           create node in this Virtual Private Cloud Subnet ID (implies VPC mode)
        --template-file TEMPLATE     Full path to location of template to use
    -V, --verbose                    More verbose output. Use twice for max verbosity
    -v, --version                    Show chef version
    -y, --yes                        Say yes to all prompts for confirmation
    -h, --help                       Show this message

The actual command we’ll use is:

knife ec2 server create --run-list 'role[wordpress]' --node-name test-wordpress --flavor t1.micro \
--identity-file ~/.ssh/aws-wordpress.pem --image ami-a2f405cb --groups wordpress \
--ssh-key aws-wordpress --ssh-user ubuntu --ebs-no-delete-on-term

Details of knife command to launch instance

role[wordpress]: The role[s] given to this instance. More than one can be specified by an orderd space separated list of strings: ‘role[role0]‘ ‘role[role1]‘ …

–node-name test-wordpress: The name of the instance. Used by Chef to name the Node and Client

–flavor t1.micro: The EC2 Instance Type. Here we are using the smallest type. This is the only one that is “free”

–identity-file ~/.ssh/aws-wordpress.pem: The path to the ssh private key that was downloaded earlier from the AWS Management Console. You could potentially not include this if you added the key to your ssh-agent.

–image ami-a2f405cb: The Amazon Machine Image assigned to this instance. It is the image of the root file system for the instance and thus determines what OS and software is booted when the instance is started. In this case it is the Canonical Ubuntu 10.4 32 bit AMI. You can find the latest Ubuntu AMIs for each region at the top of the home page of Eric Hammond’s super helpful site.

–groups wordpress: The Security Group[s] to be assigned to this instance. In this case its “wordpress” Multiple Groups can be assigned as a comma separated list

–ssh-key aws-wordpress: The name of the SSH Key Pair that was downloaded from the AWS Management Console

–ssh-user ubuntu: The user name for ssh access. This AMI uses “ubuntu”. The AMI’s usually are configured to allow only a single user to ssh by default. Different AMI’s use different names such as root or ec2-user.

–ebs-no-delete-on-term: By default, the EBS Volume is deleted when the EC2 instance is terminated. By adding this flag it will instead make it so the EBS volume will continue to exist after the EC2 instance has been terminated. You want this for your final deployed site so that if something goes wrong with the EC2 instance you will still have your EBS volume and can use it to create a new EC2 instance without loosing your data. (That is the topic of another tutorial though!)

Successful launch results

After you fire off the knife ec2 server create command, you’ll see something like:

[WARN] Fog::AWS::EC2#new is deprecated, use Fog::AWS::Compute#new instead (/Library/Ruby/Gems/1.8/gems/chef-0.9.12/lib/chef/knife/ec2_server_create.rb:145:in `run')
Instance ID: i-d10ae5bd
Flavor: t1.micro
Image: ami-a2f405cb
Availability Zone: us-east-1b
Security Groups: wordpress
SSH Key: aws-wordpress

Waiting for server..............
Public DNS Name: ec2-184-73-44-17.compute-1.amazonaws.com
Public IP Address: 184.73.44.17
Private DNS Name: domU-12-31-39-10-60-17.compute-1.internal
Private IP Address: 10.198.99.229

Waiting for sshd...done
INFO: Bootstrapping Chef on ec2-184-73-44-17.compute-1.amazonaws.com

That will be followed by loads of debugging info as the knife command bootstraps chef and its related packages and gems. This can go on for 10 to 20 minutes. Eventually you’ll see something along the lines of:

Instance ID: i-d10ae5bd
Flavor: t1.micro
Image: ami-a2f405cb
Availability Zone: us-east-1b
Security Groups: wordpress
SSH Key: aws-wordpress
Public DNS Name: ec2-184-73-44-17.compute-1.amazonaws.com
Public IP Address: 184.73.44.17
Private DNS Name: domU-12-31-39-10-60-17.compute-1.internal
Private IP Address: 10.198.99.229
Run List: role[wordpress]

You can look just before this block and see if chef finished the running of the wordpress related cookbooks ok. If within a page above the last block you don’t see any errors then all is ok. The last few lines should be something like:

[Mon, 03 Jan 2011 07:23:34 +0000] INFO: Chef Run complete in 10.945359 seconds
[Mon, 03 Jan 2011 07:23:34 +0000] INFO: cleaning the checksum cache
[Mon, 03 Jan 2011 07:23:34 +0000] INFO: Running report handlers
[Mon, 03 Jan 2011 07:23:34 +0000] INFO: Report handlers complete

If there are errors, you’ll have to debug your cookbooks which is beyond the scope of this post.

Now you should be able to log into your instance ether as the ubuntu default user or the user you created in the wordpress role and the Users Databag (rberger_test in this example):

# Using the ubuntu user and an explicit ssh key
ssh -i ~/.ssh/aws-wordpress.pem ubuntu@ec2-184-73-44-17.compute-1.amazonaws.com

# Using the user created by the cookbook and a key that is already on you ssh-agent
ssh rberger_test@ec2-184-73-44-17.compute-1.amazonaws.com

Configure DNS to have preferred FQDNs point to your instance

You can access your site using the Amazon Public DNS name, but that would not be good in general. You probably want to access it via a URL like http://www.myydomain.com. To do this you must configure your DNS to add a CNAME to map your FQDN to the Amazon Public DNS name. How this is done is very specific to your DNS service provider. Bottom line is that you want to do a CNAME not an A record.  (I.E. an alias of your FQDN for the Amazon Public DNS name, not an A record that uses the Amazon IP address). There are some issues of using an A record with Amazon. You probably won’t see them for a simple situation such as hosting a single instance. But once you have many instances that need to talk to each other, using the CNAME will make life easier.

Installing your WordPress Blog

At this point you should be able to access your new instance via http. The initial screen will be the WordPress setup dialog. You should be able to access it via http using the Amazon Public DNS name or any CNAME aliases  you created and also added in the wordpress.rb role file override attribute for (wordpress =>server_aliases. You should see something like:

Wordpress startup installation page

It is possible to move an existing WordPress Blog to this new instance but that is beyond the scope of this post.

Happily Ever After

By default, the chef client runs every 1/2 hour on the instance. If you change any of the cookbooks and push them up to the Opscode Chef Server, those changes will be propogated to the instance the next time the chef-client runs again on the instance.

This is the way to maintain the server. By updating or adding cookbooks, you define the state of the server and the server will converge to that state when the chef-client runs. The inverse is true. If you change something on the server directly and the service you changed is managed by Chef, your direct changes could be reverted by the chef-client the next time it runs.

You shouldn’t need to but you can disable the chef-client from running automatically by running the following command while ssh’d to the instance:

sudo /etc/init.d/chef-client stop

That will be reset (ie automatic chef-client runs will be re-enabled) if you reboot. You can permanently disable the automatic running of chef-client by running the following commands while ssh’d into the instance:

cd /etc/init.d
sudo update-rc.d -f chef-client remove

Using the WordPress Automatic Upgrade Mechanism

At this point you should be able to use your wordpress blog as normal. You should be able to use the automatic update feature of WordPress to update WordPress itself and the plugins. When you are asked to supply the Connection Information, put in:

• Hostname: The Public FQDN of the host (ether the EC2 Public DNS Name or one of the DNS CNAMEs you set up)
• FTP Username: “blog” (or whatever you set node[:wordpress][:blog_updater][:username] in the wordpress.rb role file)
• FTP Password: “big-secret” (or whatever you SHOULD have set node[:wordpress][:blog_updater][:password] in the wordpress.rb role
• Connection Type: FTPS (SSL)

For instance for the Plugin Update Page:

Upgrade Plugins Connection Information

That should work and be secure using the vsftpd server that we installed automatically.

Hopefully all will work well for you. I will try to answer questions but can’t guarantee quick response here. A great resource is the Opscode Chef IRC channel irc.freenode.net #chef. And of course the Opscode Chef Wiki and the Opscode Support Site.

Source Code at Github

You can get all the source for this at https://github.com/rberger/ibd-wordpress-repo

So I thought it was time [...]]]> I wanted to do some experimenting with various tools for doing Hadoop and HBase activities and didn’t want to have to bother making it work with our Cluster in the Cloud. I just wanted a simple experimental environment on my Macbook Pro running Snow Leopard Mac OS X.

So I thought it was time to revisit installing Hadoop and HBase on the Mac using the latest versions of everything. This will be deployed as Psuedo-Distributed mode native to Mac OS X. Some folks actually create a set of Linux VMs with a full Hadoop/HBase stack and run that on the Mac, but that is a bit of overkill for now.

These instructions mainly follow the standard instructions for Apache Hadoop and Apache HBase

Prerequisits

Mac OS X Xcode developer tools which includes Java 1.6.x. You can get this for free from the Apple Mac Dev Center. You have to become a member but there is a free membership available.

Download and Unpack Latest Distros

You can get a link to a mirror for Hadoop via the Hadoop Apache Mirror link and for Hbase at the HBase Apache Mirror link. Each of those links will bring you to a suggested link to a mirror for Hadoop or HBase. Once you click on the suggest link, it will bring you to a mirror with the recent releases. You can click on the stable link which will then bring you to a directory that has the latest stable Hadoop (as of this writing: hadoop-0.20.2.tar.gz) or HBase (as of this writing: hbase-0.20.3.tar.gz ). Click on those tar.gz files to download them.

I am going to keep the distros in ~/work/pkgs. I usually create a directory ~/work/pkgs and unpack the tar files there as numbered versions and then create symbolic links to them in ~/work. But you can do this all in any directory that you can control.:

cd ~/work
mkdir -p pkgs
cd pkgs
tar xvzf hadoop-0.20.2.tar.gz
tar xvzf hbase-0.20.3.tar.gz
cd ..
ln -s pkgs/hadoop-0.20.2 hadoop
ln -s pkgs/hbase-020.3 hbase
mkdir -p hadoop/logs
mkdir -p hbase/logs

Now you can have your tools all access ~/work/hadoop or ~/work/hbase and not care what version it is. You can update to later version just by downloading, untarring the distro and then just change the symbolic links.

Configure Hadoop

All the configuration files mentioned here will be in ~/work/hadoop/conf. In this example we are assuming that the Hadoop servers will only be accessed from this localhost. If you need to make it accessable from other hosts or VMs on your lan that support Bonjour, you could use the bonjour name (ie. the name of your mac followed by .local such as mymac.local) instead of localhost in the following Hadoop and HBase configuraitons

hadoop-env.sh

Mainly need to tell Hadoop where your JAVA_HOME is.

Add the following line below the commented out JAVA_HOME line is in hadoop-env.sh

export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home

core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
  </property>
</configuration>

Make sure you can ssh without a password to the hostname used in the configs

The Hadoop and Hbase start/stop scripts use ssh to access the various servers. In this case of doing a Pseudo-Distributed mode, everything is running on the localhost, but we still need to allow the scripts to ssh to the localhost.

Check that you can ssh to the localhost (or whatever hostname you used in the above configs)

We’re assuming that we’ll be running the Hadoop/HBase servers as the same user as our login. You can set things up to run as the hadoop user, but its kind of complicated on Mac OS X. See the section File System Layout in an earlier post Hadoop, HDFS and Hbase on Ubuntu & Macintosh Leopard. That section and a few other points thru that post describe how to create and use a hadoop user to run the Hadoop and HBase servers.

Back to just doing this as our own user. Test that you can ssh to the localhost without a password:

ssh localhost

If you see something like the following paragraph that ends up with a password prompt, then you need to add a key to your ssh setup that does not need a password (you may need to say yes if you are asked if you want to continue connecting).

The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is 3c:5d:6a:39:64:78:02:9d:a3:c9:69:68:50:23:71:eb.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Password:

To create a passwordless key and add it to your set of authorized keys that can access your host, do the following (as yourself, not as root. The id_dsa file name can be arbitrary):

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa_for_hadoop
cat ~/.ssh/id_dsa_for_hadoop.pub >> ~/.ssh/authorized_keys

If you have strong alternative opinions on how to set up your own keys to accomplish the same thing please do it your own way. This is just the basic way of doing a passwordless ssh. You may want to use a key you already have lying around or some other mechanism.

Start Hadoop

One time format of  Hadoop File System

Only once, before the first time you use Hadoop, you have to create a formated Hadoop File System. Don’t do this again once you have data in your Hadoop file system as it will erase anything you might have saved there. You may have to do this command again if somehow you screw up your file system. But its not something to do lightly the second time.

~/work/hadoop/bin/hadoop namenode -format

If all goes well, you should see something like:

10/05/02 18:45:04 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = Psion.local/192.168.50.16
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
10/05/02 18:45:04 INFO namenode.FSNamesystem: fsOwner=rberger,rberger,admin,com.apple.access_screensharing,_developer,_lpoperator,_lpadmin,_appserveradm,_appserverusr,localaccounts,everyone,com.apple.sharepoint.group.2,com.apple.sharepoint.group.3,dev,com.apple.sharepoint.group.1,workgroup
10/05/02 18:45:04 INFO namenode.FSNamesystem: supergroup=supergroup
10/05/02 18:45:04 INFO namenode.FSNamesystem: isPermissionEnabled=true
10/05/02 18:45:04 INFO common.Storage: Image file of size 97 saved in 0 seconds.
10/05/02 18:45:04 INFO common.Storage: Storage directory /tmp/hadoop-rberger/dfs/name has been successfully formatted.
10/05/02 18:45:04 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Psion.local/192.168.50.16
************************************************************/

Starting and stopping Hadoop

Now you can start Hadoop. You will use this command to start Hadoop in general:

~/work/hadoop/bin/start-all.sh

You can stop Hadoop with the command

~/work/hadoop/bin/stop-all.sh

But remember if you are running HBase, stop that first, then stop Hadoop.

Making sure Hadoop is working

You can see the Hadoop logs in ~/work/hadoop/logs

You should be able to see the Hadoop Namenode web interface at http://localhost:50070/ and the JobTracker Web Interface at http://localhost:50030/. If not, check that you have 5 java processes running where each of those java processes have one of the following as their last command line (as seen from a ps ax | grep hadoop command) :

org.apache.hadoop.mapred.JobTracker
org.apache.hadoop.hdfs.server.namenode.NameNode
org.apache.hadoop.mapred.TaskTracker
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
org.apache.hadoop.hdfs.server.datanode.DataNode

If you do not see these 5 processes, check the logs in ~work/hadoop/logs/*.{out,log} for messages that might give you a hint as to what went wrong.

Run some example map/reduce jobs

The Hadoop distro comes with some example / test map / reduce jobs. Here we’ll run them and make sure things are working end to end.

cd ~/work/hadoop
# Copy the input files into the distributed filesystem
# (there will be no output visible from the command):
bin/hadoop fs -put conf input
# Run some of the examples provided:
# (there will be a large amount of INFO statements as output)
bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'
# Examine the output files:
bin/hadoop fs -cat output/part-00000

The resulting output should be something like:

3	dfs.class
2	dfs.period
1	dfs.file
1	dfs.replication
1	dfs.servers
1	dfsadmin
1	dfsmetrics.log

Configuring HBase

The following config files all reside in ~/work/hbase/conf. As mentioned earlier, use a FQDN or a Bonjour name instead of localhost if you need remote clients to access HBase. But if you don’t use localhost here, make sure you do the same in the Hadoop config.

hbase-env.sh

Add the following line below the commented out JAVA_HOME line is in hbase-env.sh

export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home

Add the following line below the commented out HBASE_CLASSPATH= line

export HBASE_CLASSPATH=${HOME}/work/hadoop/conf

hbase-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://localhost:9000/hbase</value>
    <description>The directory shared by region servers.
    </description>
  </property>
</configuration>

Making Sure HBase is Working

If you do a ps ax | grep hbase you should see two java processes. One should end with:
org.apache.hadoop.hbase.zookeeper.HQuorumPeer start
And the other should end with:
org.apache.hadoop.hbase.master.HMaster start
Since we are running in the Pseudo-Distributed mode, there will not be any explicit regionservers running. If you have problems, check the logs in ~/work/hbase/logs/*.{out,log}

Testing HBase using the HBase Shell

From the unix prompt give the following command:

~/work/hbase/bin/hbase shell

Here is some example commands from the Apache HBase Installation Instructions:

base> # Type "help" to see shell help screen
hbase> help
hbase> # To create a table named "mylittletable" with a column family of "mylittlecolumnfamily", type
hbase> create "mylittletable", "mylittlecolumnfamily"
hbase> # To see the schema for you just created "mylittletable" table and its single "mylittlecolumnfamily", type
hbase> describe "mylittletable"
hbase> # To add a row whose id is "myrow", to the column "mylittlecolumnfamily:x" with a value of 'v', do
hbase> put "mylittletable", "myrow", "mylittlecolumnfamily:x", "v"
hbase> # To get the cell just added, do
hbase> get "mylittletable", "myrow"
hbase> # To scan you new table, do
hbase> scan "mylittletable"

You can stop hbase with the command:

~/work/hbase/bin/stop-hbase.sh

Once that has stopped you can stop hadoop:

~/work/hadoop/bin/stop-all.sh

Conclusion

You should now have a fully working Pseudo-Distributed Hadoop / HBase setup on your Mac. This is not suitable for any kind of large data or production project. In fact it will probably fail if you try to do anything with lots of data or high volumes of I/O. HBase seems to not like to work well until you get 4 – 5 regionservers.

But this Pseudo-Distributed version should be fine for doing experiments with tools and small data sets.

Now I can get on with playing with Cascading-Clojure and Cascalog!

Right now, the most productive way [...]]]> There are a lot of good example Opscode cookbooks out there. Unfortunately they can be hard to find. People are not submitting them to the Opscode Cookbook repository. Its still hard to untangle your own cookbooks into something that can be put in a sharable format I guess.

Right now, the most productive way to find cookbooks seems to be to search github. I always do a search before I write my own. Google searches are tough since “chef”, “cookbooks” are overloaded from the “real cooking” domain. And if you put in some package name, you tend to get announcements about the package and a mention about Opscode, but rarely about an Opscode Cookbook for that package.

Today I discovered that I needed a cookbook to install jruby. So after a useless Google Search. I did a search “jruby cookbook” on github and soon found Theo Cushion‘s cookbook clone with a jruby addition.

Its better to not have to write it yourself! My thanks to Theo and the many others who share their code.

One of the cool features of using an Amazon EC2 instance that boots from [...]]]> Introduction

Well there is already an update to Chef’s Ohai library. At first I thought, “Oh no, I have to generate another EC2 image”. But then I remember reading that you can update and clone a running EBS boot image.

One of the cool features of using an Amazon EC2 instance that boots from an EBS Snapshot is that its easy to create new boot images from an existing running EC2 instance, assuming that you are running an EC2 instance that is itself bootable from an EBS Image.

Prerequisites

The following expects that you have a recent copy of the Amazon ec2-api-tools on the instance and that you have recent version of the ec2-api-tools on your host development system.

Start up an instance, make changes

Start up an instance you can use as a base, for instance the one we created in Using the Official Opscode 0.8.x Gems to build EC2 AMI Chef Client and Server

Get the name of the instance

First you will need the instance name of your instance you want to copy. You can use Elasticfox or other tool. Or run the following command on the instance:

wget -qO- http://instance-data/latest/meta-data/instance-id

On another host

The rest of the instructions will be run on your host development system (not the system you are copying). This makes it so you don’t have to put your Amazon Certs onto the machine you are cloning (you don’t want those keys to end up on the cloned image)

Create some shell defines

To make the instructions easier make some defines we’ll use in commands. Tune them for your environment.

# This will be the instance id of the running instance you want to clone
instanceid=i-07202042

# Some info for creating the name and description
codename=karmic
release=9.10
tag=server
region=us-west-1
availability_zone=us-west-1a

# Make sure you set this as appropriate
# 64bit
arch=x86_64
arch2=amd64
#32bit
arch=i386
arch2=i386
now=$(date +%Y%m%d-%H%M)

# Make this specific to what you are making
prefix=runa-chef-0.8.4-ubuntu-$release-$codename-$tag-$arch-$now
description="Runa Chef 0.8.4 Ubuntu $release $codename $tag $arch $now"

Get the info about your running instance

Use Elasticfox or your favorite tool or the following command to get the volume id of the instance

ec2-describe-instances --region $region "$instanceid" > /tmp/instance_info
volumeid=$(egrep ^BLOCKDEVICE /tmp/instance_info | cut -f3); echo $volumeid
kernel=$(egrep ^INSTANCE /tmp/instance_info | cut -f13); echo $kernel
ramdisk=$(egrep ^INSTANCE /tmp/instance_info | cut -f14) ;echo $ramdisk

Shutdown  the instance

Its not clear if you really need to do this. But when I first tried doing it without shuting down the instance, the snapshots took forever.

Create a new snapshot

snapshotid=$(ec2-create-snapshot -region $region -d "$description" $volumeid | cut -f2)

Register the new image

ec2reg --region $region -s $snapshotid -a $arch --kernel $kernel --ramdisk $ramdisk -d "$description" -n "$prefix"

The result of this command will be the ami image name. After this completes, the image and snapshot can be used to create new instances.

To do this I had to get rid of the “validator” and [...]]]> In upgrading from my own custom hacked pre-0.8.x Chef server/clients to the official new and shiny 0.8.2 release, I wanted to make everything vanilla. One issue was somewhere along the line I set the validation_client_name to “validator”. The vanilla setting is “chef-validator”.

To do this I had to get rid of the “validator” and “chef-validator” authentication client entries I had on the chef-server. It turns out you can’t just delete them with knife or the web-ui. You have to edit the couchdb to delete the entries.

The fantastic Chef IRC channel came to the rescue in the usual personage of Josh Timberman (jtimberman) who paused from I’m sure one of his most hectic days of his life (they were still cleaning up all the loose ends of todays release of 0.8.2) to help me. The steps are:

• Create an ssh tunnel from your local machine to the chef-server:

ssh -L 5984:localhost:5984 fqdn-of-chef-server

• Then with a browser on your local machine access:

http://localhost:5984/_utils

That will connect you to futon, a web interface to couchdb running on the chef-server.

• Click on “chef”

• Select the View to be Client->All

• Select the “chef-validator”

Then delete the “chef-validator”

Once you have removed the client authentication from the couchdb, you need to remove the validation.{pem,key,crt} from /etc/chef on the chef server (there may just be validation.pem). Then restart the chef server (/etc/init.d/chef-server restart).

You should now have a fresh clean valid validation.pem in /etc/chef on the chef-server. You can then copy that to the /etc/chef on your chef client[s]. Remeber to also remove the client.pem in /etc/chef on the client. If client.pem is there, the chef-client will not try to re-validate with the new valdation.pem.

Once I had the proper validation.pem I  used the Chef upgrade bootstrapping process to update all my server and clients and use the new validation.pem to create new client authentication on the server. Everything was clean and fresh after that.

Note that this is an unusual situation that requires clearing out the old validator client.  You should not have this problem in any normal situation. If you do have this problem, make backup copies of any validation.pem or client.pem until you make sure everything is cool.

is an event-based replacement for the /sbin/init daemon which handles starting of tasks and services duringboot, stopping them during shutdown and supervising them while the system is running.

One would think that the best place to find the current documentation on Upstart would be its home page and/or the [...]]]> To quote the Upstart home page:

is an event-based replacement for the /sbin/init daemon which handles starting of tasks and services duringboot, stopping them during shutdown and supervising them while the system is running.

One would think that the best place to find the current documentation on Upstart would be its home page and/or the Upstart Wiki. But you would be wrong. The homepage has almost no documentation and the Wiki has ancient documentation (mostly from version 0.3.9 and one page of useless info from 0.5.1while the current version is 0.6.5) with no mention on how to find current documentation.

If you look in /usr/share/doc/upstart* you will find nothing of use.

The secret is man 5 init on an Ubuntu system that has upstart installed (like Ubuntu 9.10). This has some quite good and complete reference info on Upstart.

I have been using forms of Unix for over 30 years (yikes!) and “init” is not what I think of when I think of Upstart. Yes I know its a replacement for init now. But I still don’t think of it the same. If at least the Upstart pages would say “See man 5 init” for details, I might remember…

Anyway, I updated the wiki page to remind me and others that is where to get the latest info…

A good article that is a bit dated, but has more info than I found anywhere else is:

Upstart 0.5.0/trunk job definitions: The missing manual!