Press "Enter" to skip to content

Building Opscode Chef 0.8.x from HEAD of the git repo

Update:

I am having problems using the chef dev tools/client from the HEAD of the chef git repository with the Opscode Alpha Server service. I’m not sure if its me or if the latest versions of the chef client from HEAD is compatible with the Alpha Server Service. So the following is still useful for understanding how to build from HEAD, but it will not work with the Opscode Alpha SaaS server. It will work with the server you build from HEAD. See the next article Creating an Amazon EC2 AMI for Opscode Chef 0.8 for info on creating a Chef client and server on EC2.

Introduction

Opscode is introducing a pretty major set of changes in Chef in the 0.8 release. Its a major step forward and has some major changes as to how one interacts with Chef. (as well as some major bug fixes that alone make it worth the move). The Opscode Alpha Program introduces a new service where Opscode runs the actual Chef Server as a service.

This post will describe setting up your User/Dev environment by building your own Chef Client / Dev Gems from the latest HEAD of the Chef repo from Github. It assumes that you did sign up for the Alpha program and have access to the Opscode Alpha Server. Though much of it would be the same if you were running your own chef server also built from the latest source from github. This post does not show how to actually use Chef and the chef-client on a target node. Hope to have a post on that in the next few days.

The documentation on how to move to and use Chef 0.8 is still very sparse, so I figured I would jot down some of the things we are learning as we apply this to our infrastructure at Runa. If any of you OpsChefs out there see something wrong or something I left out, let me know in the comments or via email.

The Opscode Chef Alpha Environment

If you are in the Opscode Alpha program, you would have been given login[s] and some pem keys. I won’t go into the details of this since they do have pretty good docs on setting this up (if you have an alpha login you can get them at http://opscode.zendesk.com/forums/58858/entries/49336). Its probably a good idea to follow these and start with their 0.8.0 gem to make sure you are talking with the Alpha Server before trying to use the Chef Git Repository to build your own gems.

The Alpha instructions use a Chef gem that is frozen at 0.8.0. But the Chef folks have already progressed much further than the Oct 29h release of 0.8.0 in the Chef Git Repository.

The HEAD of the Git Repository has many changes since 0.8.0. Some big ones include:

  • The Knife sub commands are completely different
  • There is now a Chef Shell (A REPL like irb but for the chef client)
  • Lots of Bug Fixes

And if we’re going to be on the bleeding edge, we might as well go all the way! So the rest of this blog will be about using the Chef HEAD branch from the Chef git repository. We’ll still use the Alpha Chef Server at least to start with.

Configuring your Dev Environment

Prerequisites

I’m using Mac OS X 10.6 (snow leopard). Our target environments are Ubuntu Linux on Amazon EC2. But assuming you have *nix, Ruby and Ruby Gems set up on your environment it should generally be the same (don’t know about people stuck in the Legacy Windows environment though).

So you will need to have installed and know how to use:

  • Ruby
  • RubyGems
  • Git

And the following Ruby Gems should be installed (I think this is the minimum you need, these will include their own dependencies:

  • rake
  • rspec
  • cucumber
  • uuidtools
  • nanite
  • gemcutter
  • jeweler

You will need http://gems.opscode.com as a gem source for the following. You can use the command:

sudo gem sources -a http://gems.opscode.com
  • mixlib-authentication

Getting and building the code/GEMs for the Dev Environment

The instructions that are in the README.doc of the Chef Git Repository are out of date as of now (Dec 20, 2009). The instructions on the wiki, Installing Chef from HEAD are more accurate. Even though it seems like one can use the mixlib gems as the repository and the gems have the same version number, I found that I needed to install the mixlib libraries from source.

Getting and building Ohai & Mixlib Gems from Github

We won’t be making any changes in these, so we’ll just git clone and build it:

cd to where you want to keep your local repositories
git clone git://github.com/opscode/ohai.git
cd ohai
sudo rake install
cd ..
git clone git://github.com/opscode/mixlib-config.git
sudo rake install
cd ..
git clone git://github.com/opscode/mixlib-log.git
sudo rake install
cd ..
git clone git://github.com/opscode/mixlib-cli.git
sudo rake install
cd ..

Getting the Chef code from github

You can get the Chef repository from github. The readme there has most of the info you need for

If you plan to submit any patches or other changes back to Opscode, or you would like to have your own repository of this, you can fork the Opscode repository into your own Github account. This is what I did and will demonstrate below. If you don’t want any hardcore forking action, you can just git clone the opscode repository as shown here (assuming your current working directory is where you want the local directory repository placed. It will be named using the default “chef”):

git clone git://github.com/opscode/chef.git

If you have forked into your own github account (mine is rberger), you would git clone using the “Your Clone URL”:

git clone git@github.com:rberger/chef.git rberger-chef

This assumes you want your local directory name for the repository to be rberger-chef, just so you can distinguish it from the official opscode one. (I will refer to the top of the local repository as rberger-chef from now on).

What’s in the Chef Git Repository

Change directory into the local repository and do an ls. You’ll see that there are several components here.


$ cd rberger-chef
$ ls
CHANGELOG         README.rdoc       chef-server       chef-solr         scripts
LICENSE           Rakefile          chef-server-api   cucumber.yml
NOTICE            chef              chef-server-webui features

There are 2 main trees:

  • chef: chef-client and dev gem
  • chef-server: Chef Server gem. Used only if you build your own server
    • chef-server-api: Implements the REST interface sub-system as part of the full Chef Server
    • chef-server-webui: Implements the WebUI as part of the full Chef Server
    • chef-solar: Implements the Solar Search sub-system as part of the full Chef Server
    • features: Not 100% sure all its used for, definately for the cucumber tests. But is part of the Server as far as I can tell

For now we are only interested in the chef tree. That will be used to set up the local dev environment. We’re not going to follow the outdated instructions that are in the README.doc in the root of the chef repository which assumes you are setting up the whole stack on the Dev machine. We’re going to just install the chef client and tools from the chef sub-tree on the dev machine.

This post will not describe how to build /use the chef-server, though you can pretty much build everything by running

sudo rake install

from the top of the distro. There are more gem dependencies that need to be installed before you can build the chef-server trees.

Building and Installing the Chef Client / Dev tools

Change directory to the chef subdirectory so you should be in rberger-chef/chef (or if you have a direct clone of the opscode chef repository: chef/chef)

cd chef

Some minor tweaks to the Source

(shef is now included in the executables in the latest repository and setting my own sub-version number was lame)

I have done a few mods to the source. Mainly to set the version number to something that will not conflict with the official numbering now or when new releases come out and to have shef be installed by the gem.

  1. Changed line 30 in the Rakefile to s.executables = %w( chef-client chef-solo knife shef ) so the install puts shef in /usr/bin
  2. Changed line 7 in the Rakefile to CHEF_VERSION = "0.8.0.1"
  3. Change line 30 in lib/chef.rb to VERSION = '0.8.0.1'
Build and install
rake install

Its going to eventually ask for your sudo password as it needs to use sudo to do the gem install. The run should look something like:

(in /Users/rberger/work/Chef/rberger-chef/chef)
mkdir -p pkg
WARNING:  no rubyforge_project specified
WARNING:  description and summary are identical
  Successfully built RubyGem
  Name: chef
  Version: 0.8.0.1
  File: chef-0.8.0.1.gem
mv chef-0.8.0.1.gem pkg/chef-0.8.0.1.gem
sudo gem install pkg/chef-0.8.0.1 --no-rdoc --no-ri
Password:
Building native extensions.  This could take a while...
Successfully installed eventmachine-0.12.10
Successfully installed amqp-0.6.5
Successfully installed thor-0.12.0
Successfully installed deep_merge-0.1.0
Successfully installed moneta-0.6.0
Successfully installed chef-0.8.0.1
6 gems installed

Using Chef with the Opscode Alpha SaaS Server

This just touches on some of the things that are described in The Official Guide to Getting Started With Opscode

Setting up your Dev Environment

Its not clear if you really have to do everything as described in the document if you are building the latest release from the chef repository and using the ~/.chef/knife.rb config described below. For instance I didn’t have to set the environment variables for OPSCODE_USER and OPSCODE_KEY since they are now set in the knife.rb nor did I have to create /etc/chef/client.rb. And even without the global Chef config, I was able to use most of the knife commands. But not some like the ec2 instances data seemed to need the organization validation key to be in /etc/chef/validation.pem

Copy your assigned validation key to /etc/chef

When you got your Opscode Alpha welcome stuff, you should have gotten your user keys and a key for your organization. Copy your organization (in our case runa.pem) to /etc/chef/validation.pem. You will probably have to create /etc/chef directory first.

The User Chef/Knife config

You must configure a knife config file in your home directory under ~/.chef/knife.rb and have your key that you got from Opscode somewhere pointed to by a line in ~/.chef/knife.rb. The configuration parameters are described on the Knife Wiki Page. For instance my config file:

log_level        :info
log_location     STDOUT
node_name        'rberger'
client_key       '/Users/rberger/.chef/rberger.pem'
chef_server_url  "https://api.opscode.com/organizations/runa"
cache_type       'BasicFile'
cache_options( :path => '/Users/rberger/.chef/checksums' )

Once you have this set up you can now use knife and the chef rake commands. You can test things out by saying something like:

knife client list

Which should return and empty list assuming you haven’t set up any clients on this server yet.

The first real useful command you want to do is to upload your cookbooks to the Opscode Server:

cd to where your chef cookbook repository is
rake upload_cookbooks

You can also do it with just knife:

knife cookbook upload -a

This may take a while as it will upload all the cookbooks in cookbooks and site-cookbooks in your current repository.

After that you can upload single cookbooks

knife cookbook upload

Just remember the knife documentation on the Alpha site no longer applies to the knife that you get from building from the HEAD of the chef git repository. Strangely enough, the knife documentation on the wiki is accurate.

Conclusion

Once you’ve been thru it, its all quite simple. I hope to post some more on using 0.8.0+ soon. See a more recent blog post for building your own Chef Server Creating an Amazon EC2 AMI for Opscode Chef 0.8