Backing up EC2 MySQL and configuration files to S3

I’ve been spending a few hours setting up a good back-up strategy for my EC2 server, running NOMP.se.

The service runs on a single reserved small instance at present. It’s using Amazon’s Linux distro with an Elastic Block Storage (EBS) root disk.

The first thing you should do after setting up an EC2 host is to make an EBS snapshot. An EBS snapshot is a full disk device dump (like “dd” produces if you’re a Unix hacker). While EBS snapshots are a great feature, and should be a cornerstone in any EC2 backup strategy, they are full volume dumps, and hence take a lot space.

To compliment my EBS snapshots, which I run manually before and after bigger changes (yum update, package installs etc), I hacked together a little shell script in 1337 bytes (really) to backup my MySQL databases in a supported manner (mysqldump) and also backing up a number of configuration files from the file system. The script makes use of a great tool called s3cmd which is used to upload files to S3 (Amazon’s Simple Storage Service).

How to set up the script (all steps as root):

1. Install s3cmd
2. Run s3cmd –configure
3. Copy the generated .s3cfg file to /etc
4. Download the S3 backup script to /etc/cron.daily/
5. Edit the script to suit your needs.

I hope someone finds this useful!

The s3 console after a successful run

Here’s what the script looks like:

## Specify data base schemas to backup and credentials
DATABASES="nompdb wp_blog"

## Syntax databasename as per above _USER and _PW
## _USER is mandatory _PW is optional
nompdb_USER=root
wp_blog_USER=root

## Specify directories to backup (it's clever to use relaive paths)
DIRECTORIES="root etc/cron.daily etc/httpd etc/tomcat6 tmp/jenkinsbackup" 

## Initialize some variables
DATE=$(date +%Y%m%d)
DATETIME=$(date +%Y%m%d-%H%m)
BACKUP_DIRECTORY=/tmp/backups
S3_CMD="/usr/bin/s3cmd --config /etc/s3cfg"

## Specify where the backups should be placed
S3_BUCKET_URL=s3://nomp-backup/$DATE/

## The script
cd /
mkdir -p $BACKUP_DIRECTORY
rm -rf $BACKUP_DIRECTORY/*

## Backup MySQL:s
for DB in $DATABASES
do
BACKUP_FILE=$BACKUP_DIRECTORY/${DATETIME}_${DB}.sql
if [ ! -n "${DB}_PW"  ]
then
  PASSWORD=$(eval echo \$${DB}_PW)
  USER=$(eval echo \$${DB}_USER)
  /usr/bin/mysqldump -v --user $USER --password $PASSWORD -h localhost -r $BACKUP_FILE $DB 2>&1
else
  /usr/bin/mysqldump -v --user $USER -h localhost -r $BACKUP_FILE $DB 2>&1
fi
/bin/gzip $BACKUP_FILE 2>&1
$S3_CMD put ${BACKUP_FILE}.gz $S3_BUCKET_URL 2>&1
done

## Backup of config directories
for DIR in $DIRECTORIES
do
BACKUP_FILE=${DATETIME}_$(echo $DIR | sed 's/\//-/g').tgz
/bin/tar zcvf ${BACKUP_FILE} $DIR 2>&1
$S3_CMD put ${BACKUP_FILE} $S3_BUCKET_URL 2>&1
done