Deploying the Agaric way

Background

Where we come from

After journeying from Capistrano to Vlad we eventually use pure Rake tasks for our deployment. While Vlad was a great improvement it also had downsides. Lack of feedback from tasks made it hard to tell why they failed. Also the most interesting feature of those deployment tools is not required in our case: The ability to roll out to many servers at the same time.

We're calling this Drupal-focused Rake Drake. It is primarily developed by Stefan Freudenberg of Agaric.

Requirements

• Set up and update a number of testing/staging/demo sites and a live site
• Synchronize content between each of those and a local environment
• Create CSS from SASS before deployment to avoid generated CSS in the repository
• Environment specific settings.php files under version control

Build Tools

The road we have chosen is to use a build tool for deployment. It would have been great to find a build tool like Make for PHP but it seems the PHP community is not agreeing on a common approach the Drupal community could just hook up with. So there's Drush for interacting with Drupal from the command line and Drush Make which turns out to be not quite like Make because it lacks dependency processing. Having already some experience with Ruby and Rake I chose to create our deployment tasks with Rake using Drush whenever possible.

Deployment

One of the awesome features of Rake and other build tools is the ability to autogenerate tasks. This helps a lot considering our requirement of having different environments to deploy to. Deployment is a series of steps that need to be executed in a particular order: Preparing the build object requires to generate the CSS from SASS sources and picking the correct settings.php file. When the website is nicely packaged it has to be uploaded to the target environment, permissions have to be setup correctly, drush cc all has to be executed. All this can be programmed in a very convenient fashion with dependency based programming.

Content Synchronisation

We also have autogenerated tasks for syncing databases between our environments (they depend on a database backup task to be executed on the target of the process). Because it uses Drush to do the dump and import there is no overhead in terms of stored passwords on the developers machine or mysql configuration files on the remote servers. For syncing files rsync is used. Not having to remember passwords, filesystem paths and command line options makes those tasks very convenient.

How it actually works

Prerequisites

(You'll need Rake installed, see setting up computer for Drupal work, Agaric style.)

A good deal of our Rake tasks can be used in all of our projects. Hence it is convenient to have them available in a central place outside of any project's working copy. All things Drupal start with a D here at Agaric so we put it in ~/.drake. You can download Drake from our repository using this command git clone git://gitorious.org/drake/drake.git ~/.drake. We use the Rakefile in a project's root directory to define the remote environments for deployment (and any project specific tasks).

To include our tasks in ~/.drake rake is invoked with the --rakelibdir option:
{syntaxhighlighter class="brush:bash;gutter:false"}
$ rake --rakelibdir ~/.drake
{/syntaxhighlighter}

An alias can be defined in your .bashrc to alleviate us from typing that all the time:
{syntaxhighlighter class="brush:bash;gutter:false"}
alias drake='rake --rakelibdir ~/.drake'
{/syntaxhighlighter}

(If you've set up your computer with .bashrc-agaric, this is already included.)

Remote environments for deployment are defined by adding an array to the ENVIRONMENTS hash in a project's Rakefile, consisting of a SSH connection string, e.g. [user@]example.com, a path where the web server expects Drupal's index.php and an optional tag to deploy.

{syntaxhighlighter class="brush:ruby;gutter:false"}
ENVIRONMENTS = {
"live" => ["user@example.com",
"/path/to/document/root",
"stable",
],
}
{/syntaxhighlighter}

For some environments only tagged commits shall be deployed. Tasks to tag commits in accordance with a common pattern are being generated for each value in the TAGNAMES array which must also be defined in the Rakefile.

{syntaxhighlighter class="brush:ruby;gutter:false"}
TAGNAMES = %w(qa stable)
{/syntaxhighlighter}

Creating build artifacts

Deploying a Drupal site is relatively easy, usually not more than uploading the local working copy and putting the right settings in place; maybe changing file permissions. So in order to prepare deployment we only need to build a copy of the Drupal root directory with the correct settings.

Environment specific settings files are created by the file tasks etc/<env>.settings.php, e.g. etc/live.settings.php in the project root. Before deploying to the target server the database configuration must be entered there. Additional configuration that's specific to the environment can be put there, too. Those files are kept under version control.

In order to create the build artifact for an environment its task is invoked, e.g.:

{syntaxhighlighter class="brush:bash;gutter:false"}
$ rake build_live
{/syntaxhighlighter}

Calling rake without any arguments invokes the build task for each environment. In case a tag is defined for a target environment the task takes care of checking out the tag first.

Deploying

For each environment there's a dedicated deploy task that uploads the build artifact runs drush updatedb and drush cc all:
{syntaxhighlighter class="brush:bash;gutter:false"}
$ rake deploy_live
{/syntaxhighlighter}
Calling a deploy task will invoke the build task for the environment first. During the build process a file called BUILD_VERSION.txt is added to the Drupal directory. It contains a string indicating the git revision of the build. Use the BUILD_VERSION.txt bookmarklet to view the file when browsing a deployed site.

Synchronizing Content

Our rake library generates tasks for synchronizinging content between each environment using drush and bzip for databases and rsync for files. Some examples:
{syntaxhighlighter class="brush:bash;gutter:false"}
$ rake db_sync_live_to_test
$ rake db_sync_test_to_local
$ rake file_sync_live_to_test
$ rake file_sync_test_to_local
{/syntaxhighlighter}

Walkthrough

Starting a new Drupal project

To start a new project named example, create a directory example in your local development environment. Here are the steps that would be repeated for each project:
{syntaxhighlighter class="brush:bash;gutter:false"}
$ mkdir example
$ cd example
$ drush -y dl --drupal-project-rename=web
$ mkdir web/sites/all/modules/contrib
$ mkdir web/sites/default/files
$ cp ~/.drake/Rakefile.example Rakefile
$ git init
$ git add .
$ git commit -m "Start new Drupal project"
{/syntaxhighlighter}

Note: The command agc example NEW in Agaric's scripts repository automates the above, assuming ~/code/ as the home for site projects.

Ignoring some files

There are a few configuration files you want git to ignore. Here are the steps that would be repeated for each project:
{syntaxhighlighter class="brush:bash;gutter:false"}
$ drake .gitignore
$ drake .gitattributes
{/syntaxhighlighter}

Note: You may want to change the .htacces file REWRITEBASE to / you will find .htacces in the root of your site projects.

Configuration

The first thing to do is changing the remote environments in your local Rakefile. Below, the word 'example' refers to the project name. Your Rakefile will be uploaded to the server and all people working on this project will use the same Rakefile.
{syntaxhighlighter class="brush:ruby;gutter:false"}
ENVIRONMENTS = {
"live" => ["example@sojourner.mayfirst.org",
"/home/members/agariclabs/sites/example.com/web",
"stable",
],
"test" => ["simone.mayfirst.org",
"/var/local/drupal/example-test/web",
],
}
{/syntaxhighlighter}

Our tasks use SSH via FileUtils::sh for executing commands on remote servers. This means all settings in .ssh/config like usernames, gateways, agent forwarding, etc. are respected.

Set up a remote repository

Having made just the basic changes you should create a repository on the test server to share and deploy from. There's a task for that:

{syntaxhighlighter class="brush:bash;gutter:false"}
$ rake init_repo
{/syntaxhighlighter}

It creates a new bare repository on the infrastructure server and pushes the local repository to this new remote. It also takes care of adding this remote repository to the .git/config file.

Deploying to a test environment

Create a database on the test server (for instance mysqladmin create drupal-example-test and deploy the code to the test environment defined in Rakefile.

{syntaxhighlighter class="brush:bash;gutter:false"}
$ rake upload_test
{/syntaxhighlighter}

You either have to bring up your database started elsewhere (for instance drush sql-dump | ssh  simone.mayfirst.org mysql drupal-example-test to transfer, or run install.php.

Subsequently:

{syntaxhighlighter class="brush:bash;gutter:false"}
$ rake deploy_test
{/syntaxhighlighter}

Quality Assurance

Make sure your Rakefile is pointing qa to the latest tag you want to test for. In the final test, you can even use stable here.

Also remember to sync the database before the final test:

Deploying a new site to Live

Create a database and a Unix user for the site on the production server (most Agaric sites do this in the MayFirst control panel, Add new hosting order). Then create the new database first, and the database user second, assigned full rights to that database.