So I thought it was time to revisit installing [...]]]> I wanted to do some experimenting with various tools for doing Hadoop and HBase activities and didn’t want to have to bother making it work with our Cluster in the Cloud. I just wanted a simple experimental environment on my Macbook Pro running Snow Leopard Mac OS X.

So I thought it was time to revisit installing Hadoop and HBase on the Mac using the latest versions of everything. This will be deployed as Psuedo-Distributed mode native to Mac OS X. Some folks actually create a set of Linux VMs with a full Hadoop/HBase stack and run that on the Mac, but that is a bit of overkill for now.

These instructions mainly follow the standard instructions for Apache Hadoop and Apache HBase

Prerequisits

Mac OS X Xcode developer tools which includes Java 1.6.x. You can get this for free from the Apple Mac Dev Center. You have to become a member but there is a free membership available.

Download and Unpack Latest Distros

You can get a link to a mirror for Hadoop via the Hadoop Apache Mirror link and for Hbase at the HBase Apache Mirror link. Each of those links will bring you to a suggested link to a mirror for Hadoop or HBase. Once you click on the suggest link, it will bring you to a mirror with the recent releases. You can click on the stable link which will then bring you to a directory that has the latest stable Hadoop (as of this writing: hadoop-0.20.2.tar.gz) or HBase (as of this writing: hbase-0.20.3.tar.gz ). Click on those tar.gz files to download them.

I am going to keep the distros in ~/work/pkgs. I usually create a directory ~/work/pkgs and unpack the tar files there as numbered versions and then create symbolic links to them in ~/work. But you can do this all in any directory that you can control.:

cd ~/work
mkdir -p pkgs
cd pkgs
tar xvzf hadoop-0.20.2.tar.gz
tar xvzf hbase-0.20.3.tar.gz
cd ..
ln -s pkgs/hadoop-0.20.2 hadoop
ln -s pkgs/hbase-020.3 hbase
mkdir -p hadoop/logs
mkdir -p hbase/logs

Now you can have your tools all access ~/work/hadoop or ~/work/hbase and not care what version it is. You can update to later version just by downloading, untarring the distro and then just change the symbolic links.

Configure Hadoop

All the configuration files mentioned here will be in ~/work/hadoop/conf. In this example we are assuming that the Hadoop servers will only be accessed from this localhost. If you need to make it accessable from other hosts or VMs on your lan that support Bonjour, you could use the bonjour name (ie. the name of your mac followed by .local such as mymac.local) instead of localhost in the following Hadoop and HBase configuraitons

hadoop-env.sh

Mainly need to tell Hadoop where your JAVA_HOME is.

Add the following line below the commented out JAVA_HOME line is in hadoop-env.sh

export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home

core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
  </property>
</configuration>

Make sure you can ssh without a password to the hostname used in the configs

The Hadoop and Hbase start/stop scripts use ssh to access the various servers. In this case of doing a Pseudo-Distributed mode, everything is running on the localhost, but we still need to allow the scripts to ssh to the localhost.

Check that you can ssh to the localhost (or whatever hostname you used in the above configs)

We’re assuming that we’ll be running the Hadoop/HBase servers as the same user as our login. You can set things up to run as the hadoop user, but its kind of complicated on Mac OS X. See the section File System Layout in an earlier post Hadoop, HDFS and Hbase on Ubuntu & Macintosh Leopard. That section and a few other points thru that post describe how to create and use a hadoop user to run the Hadoop and HBase servers.

Back to just doing this as our own user. Test that you can ssh to the localhost without a password:

ssh localhost

If you see something like the following paragraph that ends up with a password prompt, then you need to add a key to your ssh setup that does not need a password (you may need to say yes if you are asked if you want to continue connecting).

The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is 3c:5d:6a:39:64:78:02:9d:a3:c9:69:68:50:23:71:eb.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Password:

To create a passwordless key and add it to your set of authorized keys that can access your host, do the following (as yourself, not as root. The id_dsa file name can be arbitrary):

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa_for_hadoop
cat ~/.ssh/id_dsa_for_hadoop.pub >> ~/.ssh/authorized_keys

If you have strong alternative opinions on how to set up your own keys to accomplish the same thing please do it your own way. This is just the basic way of doing a passwordless ssh. You may want to use a key you already have lying around or some other mechanism.

Start Hadoop

One time format of  Hadoop File System

Only once, before the first time you use Hadoop, you have to create a formated Hadoop File System. Don’t do this again once you have data in your Hadoop file system as it will erase anything you might have saved there. You may have to do this command again if somehow you screw up your file system. But its not something to do lightly the second time.

~/work/hadoop/bin/hadoop namenode -format

If all goes well, you should see something like:

10/05/02 18:45:04 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = Psion.local/192.168.50.16
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
10/05/02 18:45:04 INFO namenode.FSNamesystem: fsOwner=rberger,rberger,admin,com.apple.access_screensharing,_developer,_lpoperator,_lpadmin,_appserveradm,_appserverusr,localaccounts,everyone,com.apple.sharepoint.group.2,com.apple.sharepoint.group.3,dev,com.apple.sharepoint.group.1,workgroup
10/05/02 18:45:04 INFO namenode.FSNamesystem: supergroup=supergroup
10/05/02 18:45:04 INFO namenode.FSNamesystem: isPermissionEnabled=true
10/05/02 18:45:04 INFO common.Storage: Image file of size 97 saved in 0 seconds.
10/05/02 18:45:04 INFO common.Storage: Storage directory /tmp/hadoop-rberger/dfs/name has been successfully formatted.
10/05/02 18:45:04 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Psion.local/192.168.50.16
************************************************************/

Starting and stopping Hadoop

Now you can start Hadoop. You will use this command to start Hadoop in general:

~/work/hadoop/bin/start-all.sh

You can stop Hadoop with the command

~/work/hadoop/bin/stop-all.sh

But remember if you are running HBase, stop that first, then stop Hadoop.

Making sure Hadoop is working

You can see the Hadoop logs in ~/work/hadoop/logs

You should be able to see the Hadoop Namenode web interface at http://localhost:50070/ and the JobTracker Web Interface at http://localhost:50030/. If not, check that you have 5 java processes running where each of those java processes have one of the following as their last command line (as seen from a ps ax | grep hadoop command) :

org.apache.hadoop.mapred.JobTracker
org.apache.hadoop.hdfs.server.namenode.NameNode
org.apache.hadoop.mapred.TaskTracker
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
org.apache.hadoop.hdfs.server.datanode.DataNode

If you do not see these 5 processes, check the logs in ~work/hadoop/logs/*.{out,log} for messages that might give you a hint as to what went wrong.

Run some example map/reduce jobs

The Hadoop distro comes with some example / test map / reduce jobs. Here we’ll run them and make sure things are working end to end.

cd ~/work/hadoop
# Copy the input files into the distributed filesystem
# (there will be no output visible from the command):
bin/hadoop fs -put conf input
# Run some of the examples provided:
# (there will be a large amount of INFO statements as output)
bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'
# Examine the output files:
bin/hadoop fs -cat output/part-00000

The resulting output should be something like:

3	dfs.class
2	dfs.period
1	dfs.file
1	dfs.replication
1	dfs.servers
1	dfsadmin
1	dfsmetrics.log

Configuring HBase

The following config files all reside in ~/work/hbase/conf. As mentioned earlier, use a FQDN or a Bonjour name instead of localhost if you need remote clients to access HBase. But if you don’t use localhost here, make sure you do the same in the Hadoop config.

hbase-env.sh

Add the following line below the commented out JAVA_HOME line is in hbase-env.sh

export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home

Add the following line below the commented out HBASE_CLASSPATH= line

export HBASE_CLASSPATH=${HOME}/work/hadoop/conf

hbase-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://localhost:9000/hbase</value>
    <description>The directory shared by region servers.
    </description>
  </property>
</configuration>

Making Sure HBase is Working

If you do a ps ax | grep hbase you should see two java processes. One should end with:
org.apache.hadoop.hbase.zookeeper.HQuorumPeer start
And the other should end with:
org.apache.hadoop.hbase.master.HMaster start
Since we are running in the Pseudo-Distributed mode, there will not be any explicit regionservers running. If you have problems, check the logs in ~/work/hbase/logs/*.{out,log}

Testing HBase using the HBase Shell

From the unix prompt give the following command:

~/work/hbase/bin/hbase shell

Here is some example commands from the Apache HBase Installation Instructions:

base> # Type "help" to see shell help screen
hbase> help
hbase> # To create a table named "mylittletable" with a column family of "mylittlecolumnfamily", type
hbase> create "mylittletable", "mylittlecolumnfamily"
hbase> # To see the schema for you just created "mylittletable" table and its single "mylittlecolumnfamily", type
hbase> describe "mylittletable"
hbase> # To add a row whose id is "myrow", to the column "mylittlecolumnfamily:x" with a value of 'v', do
hbase> put "mylittletable", "myrow", "mylittlecolumnfamily:x", "v"
hbase> # To get the cell just added, do
hbase> get "mylittletable", "myrow"
hbase> # To scan you new table, do
hbase> scan "mylittletable"

You can stop hbase with the command:

~/work/hbase/bin/stop-hbase.sh

Once that has stopped you can stop hadoop:

~/work/hadoop/bin/stop-all.sh

Conclusion

You should now have a fully working Pseudo-Distributed Hadoop / HBase setup on your Mac. This is not suitable for any kind of large data or production project. In fact it will probably fail if you try to do anything with lots of data or high volumes of I/O. HBase seems to not like to work well until you get 4 – 5 regionservers.

But this Pseudo-Distributed version should be fine for doing experiments with tools and small data sets.

Now I can get on with playing with Cascading-Clojure and Cascalog!

Runa.com, the startup where I am CTO, is looking for great developers to join our small agile team. We’re an early stage, pre-series-A startup (presently funded with strategic investments from two large corporations). Runa offers [...]]]> Opportunity Knocks

Runa.com, the startup where I am CTO, is looking for great developers to join our small agile team. We’re an early stage, pre-series-A startup (presently funded with strategic investments from two large corporations). Runa offers a SaaS to on-line merchant that allows them to offer dynamic product and consumer specific promotions embeded in their website. This will be a very large positive disruption to the online retailing world.

Techie keywords: clojure, hadoop, hbase, rabbitmq, erlang, chef, swarm computing, ruby, rails, javascript, amazon EC2, emacs, Macintosh, Linux, selenium, test/behavior driven development, agile, lean, XP, scalability

If you’re interested, email  jobs@runa.com

If you want to know more, read on!

What do we do

Runa aims to provide the top of the long tail thru the middle of the top 500 online retailers with tools/services that companies like amazon.com use/provide. These smaller guys can’t afford or don’t have the resources to do anything on that scale, but by using our SaaS services, they can make more money while providing customers with greater value.

The first service we’re building is what we call Dynamic Sale Price.

It’s a simple concept – it allows the online-retailer to offer a sale price for each product on his site, personalized to the individual consumer who is browsing it. By using this service, merchants are able to -

• Increase conversion (get them to buy!) and
• Offer consumers a special price which maximizes the merchant’s profit

This is different from “dumb-discounting” where something is marked-down, and everyone sees the same price. This service is more like airline or hotel pricing which varies from day to day, but much more dynamic and real-time. Further, it is based on broad statistical factors AND individual consumer behavior. After all, if you lower prices enough, consumers will buy. Instead, we dynamically lower prices to a point where statistically, that consumer is most likely to buy.

How we do it

Runa does this by performing statistical analysis and pattern recognition of what consumers are doing on the merchant sites. This includes browsing products on various pages, adding and removing items from carts, and purchasing or abandoning the carts. We track consumers as they browse, and collect vast quantities of this click-stream data. By mining this data and applying algorithms to determine a price point per consumer based on their behavior, we’re able to  maximize both conversion (getting the consumer to buy) AND merchant profit.

We also offer the merchant comprehensive reports based on analysis of the mountains of data we collect. Since the data tracks consumer activity down to the individual product SKU level (for each individual consumer), we can provide very rich analytics.  This is a tool that merchants need today, but don’t have the resources to build for themselves.

The business model

For reference, it is useful to understand the affiliate marketing space. Small-to-medium merchants (our target audience) pay affiliates up to 40% of a sale price. Yes, 40%. The average is in the 20% range.

We charge our merchants around 10% of sales the Runa delivers. Our merchants are happy to pay it, because it is a performance-based pay, lower than what they pay affiliates, and there is zero up-front cost to the service. In fact, the above mentioned analytics reports are free.

We’re targeting e-commerce PLATFORMS (as opposed to individual merchants); in this way, we’re able to scale up merchant-acquisition. We have 10 early-customer merchants right now, with about 100 more planned to go live in the next 2-3 months. By the end of next year, we’re targeting about 1,000 merchants and 10,000 merchants the following year. Our channel deployment model makes these goals achievable.

At something like a 5 to 10% service charge, and a typical merchant having between 500K to 1M in sales per year, this is a VERY profitable business model. That is, of course, if we’re successful… but we’re seeing very positive signs so far.

Technology

Most of our front-end stuff (like the merchant-dashboard, reports, campaign management) is built with Ruby on Rails. Our merchant integration requires browser-side Javascript magic. All our analytics (batch-processing) and real-time pricing services are written in Clojure. We use RabbitMQ for all our messaging needs. We store data in HBase. We’re deployed on Amazon’s EC2.

Here are a few blog postings about what we’ve been up to -

Distributed Clojure system in production
Using messaging for scalability
Capjure: a simple HBase persistence layer
Clojure in production
Experience installing Hbase 0.20.0 Cluster on Ubuntu 9.04 and EC2

We’ve also open-sourced a few of our projects -

swarmiji – A distributed computing system to write and run Clojure code in parallel, across CPUs
capjure – Clojure persistence for HBase

Culture at Runa

We’re a small team, very passionate about what we do. We’re focused on delivering a ground-breaking, disruptive service that will allow merchants to really change the way they sell online. We work start-up hours, but we’re flexible and laid-back about it. We know that a healthy personal life is important for a good professional life. We work with each other to support it.

We use an agile process with a lot of influences from the “Lean”:http://en.wikipedia.org/wiki/Lean_software_development and “Kanban”:http://leansoftwareengineering.com/2007/08/29/kanban-systems-for-software-development/ world. We use “Mingle”:http://studios.thoughtworks.com/mingle-agile-project-management to run our development process. Everything, OK mostly everything is covered by automated tests, so we can change things as needed.

We’re all Apple in the office – developers get a MacPro with a nice 30″ screen, and a nice 17″ MacBook Pro.  We deploy on Ubuntu servers.  Aeron chairs are cliché, yes; but, very comfy.

The environment is chilled out… you can wear shorts and sandals to work…  Very flat organization, very non-bureaucratic… nice open spaces (no cubes!). Lunch is brought in on most days! Beer and snacks are always in the fridge.

We’re walking distance to the San Antonio Caltrain station (biking distance from the Mountain View Caltrain/VTA lightrail station).

What’s in it for you

• Competitive salaries, and lots of stock-options
• Cutting edge technology stack
• Fantastic business opportunity, and early-stage (= great time to join!)
• Developer #5 – means plenty of influence on foundational architecture and design
• Smart, full bandwidth, fun people to work with
• Very comfortable, nice office environment
• We have a “No Assholes” policy

OK!

So, if you’re interested, email us at jobs@runa.com

No recruiters please!

We would prefer folks who are already in the Bay Area (but if you not local and are really great let’s talk!)

[After playing with Chef, my conclusion is [...]]]> Update:

The folks who developed iClassify have come out with a total framework that is an alternative to iClassify/Puppet called opscode-chef. I will be looking into that soon and not do anymore work with iClassify (unless Chef turns out to suck or something, but at first glance it looks pretty good!).

[After playing with Chef, my conclusion is forget about iClassify / Puppet and just use Chef unless you are already using Puppet]

iClassify Description

From the creator’s of iClassify HJK Solutions website:

iClassify allows for the easy registration and classification of nodes. Most of the time, a node is a server. With iClassify:

• Nodes register themselves with a central web service, including reporting Facter facts.
• You can then tag those nodes, and add manual attributes.
• You can search the nodes with a full text search engine
• You can write recipies for icagent to auto-classify and auto-attribute your nodes.
• You can tie it in to Puppet as an external node classification tool, enabling you to easily configure hundreds of nodes at a time.
• You can tie it in to Capistrano, and have a dynamic ad-hoc configuration tool.

We are considering using it along with Puppet and Amazon EC2 for deployment of some of our infrastructure.

Install iClassify

Original Instructions are at HJK Solutions. Some of the following quotes liberally from that site but adds the things I learnt along the way as well as how to do it on a Mac.

Prerequisites

Ruby Gems

• Rails 2.0.2
• Rake
• Builder
• UUID Tools
• Mongrel
• Highline
• Net-LDAP

sudo gem rails rake install builder uuidtools mongrel ruby-net-ldap

Non-Gems

• MySQL
• Factor
• Git
• Java
• Runit
• Mongrel Runit

Configure MySQL

The folks at HJK say they use MySQL and it should work with PostgreSQL and sqlite3 as well. We went wtih MySQL.
First create the database iclassify_production

mysqladmin -u root -p  create iclassify_production

Grant it the correct privleges (set yourpass to the password you want to use):

mysql -u root -p iclassify_production
mysql> GRANT ALL ON iclassify_production.* TO 'iclassify'@'localhost' IDENTIFIED BY 'yourpass';

You should have the ruby mysql gem installed

sudo gem install mysql

on the Macintosh OS X Leopard I had to say:

sudo env ARCHFLAGS="-arch i386" gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config

Git

You can install from ubuntu packages

sudo apt-get install git-core

On the Mac Download and install the git OS X package for the Mac from Git OSX Installer on Google Code or

port -uR install git-core

Java

Java is needed for the Solr package that is bundled in the iClassify distro. It can be installed via apt and/or downloaded from the Sun site. Java is already installed on Mac OS X.

Runit

Runit is an alternative / addition to the standard /etc/init.d “systemV” init system. I guess the HJK folks like it and seem to have dependencies on it. They say it should work without it, but I haven’t tried putting this together without the Runit/Mongrel_runit dependencies yet.

Some good info and tips on setting up / running runit on various systems can be found at runit – a UNIX init scheme with service supervision

Runit can be safely installed with apt. It will not replace the standard init system.:

sudo apt-get install runit

For the Mac:

port install runit

On the Mac, you’ll have to start the runit system with the command

sudo launchctl load -w /Library/LaunchDaemons/org.macports.runit.plist

iClassify itself

In a directory you want to keep the source of iclassify:

git clone git://git.hjksolutions.com/iclassify iclassify
cd iclassify

You need to know where you want to install the actual working rails app of iclassify and what user id/group you want to run it under.

The default location and the one we’ll use on Ubuntu is /srv/iclassify and the user id/group is usually the same as the one that runs the apache web services (www-data). Change yourpass to the password used for the iclassify user in MySQL.

sudo rake iclassify:install ICBASE=/srv/iclassify ICUSER=www-data ICGROUP=www-data DBUSER=iclassify DBPASS=yourpass

For the Macintosh:

sudo rake iclassify:install ICBASE=/usr/local/iclassify ICUSER=_www ICGROUP=_www DBUSER=iclassify DBPASS=yourpass

This will create a new iClassify instance in /srv/iclassify, set the right ownership to run iClassify, and set run the migrations to prepare your database instance. I found that I had to run this as root so that it will create the directories properly.

You can test that the iClassify rails app was installed properly by testing it with the built in Rails Server (on the Mac use the /usr/local/iclassify directory and _www user id):

$ cd /srv/iclassify
$ sudo -u www-data env RAILS_ENV=production ./script/server
=> Booting Mongrel (use 'script/server webrick' to force WEBrick)
=> Rails application starting on http://0.0.0.0:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server
** Starting Mongrel listening at 0.0.0.0:3000
** Starting Rails with production environment...
** Rails loaded.
** Loading any Rails specific GemPlugins
** Signals ready.  TERM => stop.  USR2 => restart.  INT => stop (no restart).
** Rails signals registered.  HUP => reload (without restart).  It might not work well.
** Mongrel available at 0.0.0.0:3000
** Use CTRL-C to stop.

You can now point your browser to the local instance of iClassify at http://localhost:3000, hit CTRL-C when you are done to terminate the script/server.

Mongrel Runit

You can download the mongrel_runit gem from the Mongrel Runit page at HJK

Then install it with (on the Mac you’ll have to use the /usr/local/iclassify directory instead of /src/iclassify):

For some reason the HJK folks set their runit_service_dir to be /var/service but the runit ubuntu package puts it in /etc/service. So you might want to edit /srv/iclassify/examples/mongrel_runit_iclassify.yml and set runit_service_dir to /etc/service. You can also change the number of mongrels you want to run in that file. Similarly, the DarwinPorts port install of runit expects it to be in /opt/local/var/service. You can change the /srv/iclassify/examples/mongrel_runit_iclassify.yml or you can make a symbolic link from /opt/local/var/service to /var/service

sudo gem install mongrel_runit-0.2.1.gem
sudo mkdir /etc/mongrel_runit
sudo cp /srv/iclassify/examples/mongrel_runit_iclassify.yml /etc/mongrel_runit/iclassify.yml
sudo mongrel_runit -c /etc/mongrel_runit/iclassify.yml create

You should be able to then run the command

mongrel_runit -v status -c /etc/mongrel_runit/iclassify.yml

and see something like (there should be as many lines as you have set for mongrel servers. I changed the iclassify.yml from 5 to 3):

5000: true: run: /etc/sv/mongrel-iclassify-5000: (pid 4403) 119s; run: log: (pid 4402) 119s
5001: true: run: /etc/sv/mongrel-iclassify-5001: (pid 4401) 119s; run: log: (pid 4400) 119s
5002: true: run: /etc/sv/mongrel-iclassify-5002: (pid 4399) 119s; run: log: (pid 4398) 119s

Solr

First create some directories that will be needed for Solr to run its index as the www user (on the Mac replace /srv with /usr/local).

sudo mkdir -p /srv/iclassify/vendor/plugins/acts_as_solr/solr/solr/data/production
sudo chown -R www-data:www-data /srv/iclassify/vendor/plugins/acts_as_solr/solr/solr/data/

You can then test it with the command:

sudo -u www-data env RAILS_ENV=production rake solr:start

It should start and run with no errors.

You can stop it if you want with:

sudo -u www-data env RAILS_ENV=production rake solr:stop

Then set up runit to run it automatically (use /usr/local instead of /srv on the Mac and /var/service or whatevever your system uses for runit service dir if not on ubuntu and using /etc/service):

sudo mkdir -p /etc/sv/iclassify-solr/log/main
sudo cp /srv/iclassify/examples/solr-run /etc/sv/iclassify-solr/run
sudo cp /srv/iclassify/examples/solr-log /etc/sv/iclassify-solr/log/run
sudo chmod a+x /etc/sv/iclassify-solr/run /etc/sv/iclassify-solr/log/run

The following will start the solr process immediately as well as in the future reboots

sudo ln -s /etc/sv/iclassify-solr /etc/service

Apache

iClassify is best configured as a virtual host under Apache, running with SSL and mod_proxy_balancer. Follow the proper steps for configuring your platforms Apache to use mod_ssl, mod_proxy_balance r and mod_rewrite. Create a virtual host config which resembles the following (works with Mac and Ubuntu, On the Mac just change the refs to /srv to /usr/local and the EXAMPLE.com to your domain in any case)

<VirtualHost *:443>
  DocumentRoot /srv/iclassify/public
  LimitRequestBody 8388608
  ServerName iclassify.EXAMPLE.COM
  ServerAlias iclassify
  <Directory />
    Options FollowSymLinks
    AllowOverride None
  </Directory>

  <Location /server-status>
    SetHandler server-status
    Order Deny,Allow
    Deny from all
    Allow from 127.0.0.1 192.168.0.0/255.255.0.0
  </Location>

  <Proxy balancer://iclassify>
    BalancerMember http://localhost:5000
    BalancerMember http://localhost:5001
    BalancerMember http://localhost:5002
    BalancerMember http://localhost:5003
  </Proxy>

  LogLevel info
  ErrorLog /var/log/apache2/iclassify-error.log
  CustomLog /var/log/apache2/iclassify-access.log combined

  RewriteEngine On
  RewriteLog /var/log/apache2/iclassify-rewrite.log
  RewriteLogLevel 0
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
  RewriteCond %{SCRIPT_FILENAME} !maintenance.html
  RewriteRule ^.*$ /system/maintenance.html [L]
  RewriteRule ^/server-status$ /server-status$1 [L]

  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  RequestHeader set X_FORWARDED_PROTO 'https'
  RewriteRule ^/(.*)$ balancer://iclassify%{REQUEST_URI} [P,QSA,L]
  ProxyPassReverse / balancer://iclassify
  SetEnv proxy-nokeepalive 1

  SSLEngine on
  SSLCertificateFile /etc/apache2/ssl/iclassify.crt
  SSLCertificateKeyFile /etc/apache2/ssl/iclassify.key

  BrowserMatch ".*MSIE.*" \
    nokeepalive ssl-unclean-shutdown \
    downgrade-1.0 force-response-1.0
</VirtualHost>

<VirtualHost *:80>
  DocumentRoot /srv/iclassify/public
  LimitRequestBody 8388608
  ServerName iclassify.EXAMPLE.COM
  ServerAlias iclassify

  RewriteEngine On
  RewriteCond %{HTTPS} !=on
  RewriteRule ^/(.*) https://%{SERVER_NAME}/ [R,L]
</VirtualHost>

Basic instructions for setting up ssl certificates can be found at Just Samuels blog post HOWTO: Create a self-signed (wildcard) SSL certificate

Take the resulting hosts.cert and copy it to /etc/apache2/ssl/iclassify.crt and hosts.key to /etc/apache2/ssl/iclassify.key (or whereever you put your ssl keys and make sure the SSLCertificateKeyFile and SSLCertificateFile are set the same in your vhosts conf file.

Conclusion

That should get you up and running with iClassify. In a future post I will install Puppet and then figure out how to use these together to deploy to Amazon EC2.