Backing up EC2 MySQL and configuration files to S3

I’ve been spending a few hours setting up a good back-up strategy for my EC2 server, running NOMP.se.

The service runs on a single reserved small instance at present. It’s using Amazon’s Linux distro with an Elastic Block Storage (EBS) root disk.

The first thing you should do after setting up an EC2 host is to make an EBS snapshot. An EBS snapshot is a full disk device dump (like “dd” produces if you’re a Unix hacker). While EBS snapshots are a great feature, and should be a cornerstone in any EC2 backup strategy, they are full volume dumps, and hence take a lot space.

To compliment my EBS snapshots, which I run manually before and after bigger changes (yum update, package installs etc), I hacked together a little shell script in 1337 bytes (really) to backup my MySQL databases in a supported manner (mysqldump) and also backing up a number of configuration files from the file system. The script makes use of a great tool called s3cmd which is used to upload files to S3 (Amazon’s Simple Storage Service).

How to set up the script (all steps as root):

  1. Install s3cmd
  2. Run s3cmd –configure
  3. Copy the generated .s3cfg file to /etc
  4. Download the S3 backup script to /etc/cron.daily/
  5. Edit the script to suit your needs.

I hope someone finds this useful!

The s3 console after a successful run

Here’s what the script looks like:

## Specify data base schemas to backup and credentials
DATABASES="nompdb wp_blog"

## Syntax databasename as per above _USER and _PW
## _USER is mandatory _PW is optional
nompdb_USER=root
wp_blog_USER=root

## Specify directories to backup (it's clever to use relaive paths)
DIRECTORIES="root etc/cron.daily etc/httpd etc/tomcat6 tmp/jenkinsbackup" 

## Initialize some variables
DATE=$(date +%Y%m%d)
DATETIME=$(date +%Y%m%d-%H%m)
BACKUP_DIRECTORY=/tmp/backups
S3_CMD="/usr/bin/s3cmd --config /etc/s3cfg"

## Specify where the backups should be placed
S3_BUCKET_URL=s3://nomp-backup/$DATE/

## The script
cd /
mkdir -p $BACKUP_DIRECTORY
rm -rf $BACKUP_DIRECTORY/*

## Backup MySQL:s
for DB in $DATABASES
do
BACKUP_FILE=$BACKUP_DIRECTORY/${DATETIME}_${DB}.sql
if [ ! -n "${DB}_PW"  ]
then
  PASSWORD=$(eval echo \$${DB}_PW)
  USER=$(eval echo \$${DB}_USER)
  /usr/bin/mysqldump -v --user $USER --password $PASSWORD -h localhost -r $BACKUP_FILE $DB 2>&1
else
  /usr/bin/mysqldump -v --user $USER -h localhost -r $BACKUP_FILE $DB 2>&1
fi
/bin/gzip $BACKUP_FILE 2>&1
$S3_CMD put ${BACKUP_FILE}.gz $S3_BUCKET_URL 2>&1
done

## Backup of config directories
for DIR in $DIRECTORIES
do
BACKUP_FILE=${DATETIME}_$(echo $DIR | sed 's/\//-/g').tgz
/bin/tar zcvf ${BACKUP_FILE} $DIR 2>&1
$S3_CMD put ${BACKUP_FILE} $S3_BUCKET_URL 2>&1
done

 

Oracle kills Open Source Java with a really big rock?

Being known as the guy that called Oracle evil in a blog post, I feel I gotta comment on today’s announcement that Oracle is buying Sun Microsystems for 7.4 billion US dollars. As you can imagine, I’m not very optimistic.

What does the deal really mean to the Open Source Java community? Isn’t this just business as usual? And wouldn’t we be worse off if Big Blue would have bought Sun a couple of weeks back?

As you might have guessed, I would have preferred IBM to buy Sun for many reasons. Perhaps the main thing is that I feel IBM has been embracing open source, whereas Oracle hasn’t. It makes all the difference. Let’s hope Oracle sees the light and doesn’t screw up everything Java!

The Sun assets at stake:
Development Tools: NetBeans
Middleware: OpenSSO, Glassfish, MySQL, Java Hotspot JVM, Java Real Time System
Consumer Technology: OpenOffice, JavaFX / JavaFX Mobile

I’ll try to describe what I think is a likely outcome of the assets above by comparing them to Oracle’s current product line and let’s see how bad this actually can get…

Sun NetBeans vs Oracle JDeveloper

This is easy – no one uses JDeveloper, and it would surprise me if Oracle didn’t bite the bullet and ditch JDeveloper for NetBeans which has become a really good (the best?) IDE recently.

Sun OpenSSO vs Oracle Access Manager

There will be no point for Oracle to invest any money in OpenSSO when they already have a good offering in their Fusion middleware suite. OpenSSO is toast.

Sun Glassfish vs Oracle Weblogic

Oracle has a stronger app server in Weblogic than Glassfish is. I think that Glassfish will be put to the axe. Quickly. Oracle is not known for giving away software and they will only open source software that is needing life support. Some recent examples include TopLink and ADF Faces.

Sun MySQL vs Oracle RDBMS

I think this is _the_ most obvious: MySQL is TheirSQL now and also R.I.P.

Sun Java Hotspot vs Oracle JRockit

Being the cynical person I am, I think Oracle can kill two birds with one stone here. One might think that Oracle would merge the two VM efforts into one, but the result might not be what you think. Let’s just assume that Oracle takes the good stuff from JRockit and puts it into the Hotspot JVM reference implementation. Is this a likely scenario? Hell no. Oracle is in the software business to make money, and if you want to run a production-grade Java Server VM, then you will have to get it from Oracle for a fee. By doing this they also effectively kill Terracotta, the only viable contender to Oracle Coherence. See, Terracotta will not run on JRockit… The consumer JVM will be named Sun JVM. Oracle will of course keep the Java Real Time VM as it’s profitable business.

OpenOffice, JavaFX / JavaFX Mobile

Oracle’s track-record in building consumers applications is, well, not great. Anyone that’s ever tried to install an Oracle product knows what I’m talking about. So I don’t think OpenOffice will survive either. On the other hand Larry may want to keep pushing it just to be a thorn in Microsoft’s side… As for JavaFX, it doesn’t really stand a chance versus Adobe – its too little, and too late. Oracle knows this and will kill it. Quietly.

So, is this good for anyone at all? Yes. Oracle and Microsoft. Everyone else loses. IBM is in a really awkward situation and JavaOne this year may be the ultimate funeral service for (free) open source Java.

Keep in mind that when Oracle says open they generally mean open standards whereas when IBM and Sun says open they generally mean free open source software.

I’ll close with a few Larry Ellison quotes from the conference call:

“Java is the foundation of the Oracle Fusion middleware and its the second most important software asset we’ve ever acquired.”

“We acquired BEA because they had the leading Java virtual machine”

Facelift and EDA

I’m getting back in shape! Since my last post I’ve become a dad again and life is good. I’ve taken up exercise again, and run almost every day. I’ve lost 10 kg:s and is starting to look somewhat fit again…

Operation Facelift

Fortunately for the company – we’re getting in shape at work too! We’re currently moving to XHTML 1.1 strict and a floating layout with skinning support. We’re also moving to 100% YUI and optimizing for SEO and performance. TTM and TCO will decrease significantly too. Man, I love to work with great front-end people. This is going to rock!

Event Driven Architecture

I’ve also kicked off a huge push where all the product teams will start aligning their architectures to an Event Driven Architecture (EDA) model. EDA is great for separation of concerns (the registration module of the customer system will let everyone who cares know that a new customer has registered) and also for getting a scalable architecture (async async async!).

Maven2 and Hudson

We’ve gotten all the product teams up on Maven2 from the god-forgotten shell-script/ant mess we had a year ago. Hudson is used for continuous building. I love Hudson – highly recommended! We also migrated to Subversion (finally).

On the way to and from work

I ride my bike to and from work every day between May and October. It’s a lovely 10 km ride through Stockholm, mostly along the water. The other time of the year I take the tram or the subway.
During this time I prefer listening to podcasts my iPod touch. The iPod touch is great, yet I’m looking forward to replace it with an iPhone 2.0 with high speed internet.

I try to follow the following podcasts:

Entrepreneurial Thought Leaders

The DFJ Entrepreneurial Thought Leaders Seminar (ETL) is a weekly seminar series on entrepreneurship, co-sponsored by BASES (a student entrepreneurship group), Stanford Technology Ventures Program, and the Department of Management Science and Engineering.

http://www.stanford.edu/group/edcorner/uploads/podcast/EducatorsCorner.xml

Some of my personal favorites are:

BusinessWeek – CEO Guide To Technology

I love this BusinessWeek site that has as its mission to educate CEO:s on new trends in technology. It is very useful to listen to the way that BusinessWeek explains tech to management.

http://www.businessweek.com/search/podcasts/guide_to_tech.rss

Robert X Cringely – the Pulpit

I’m a long time fan of Cringely’s since Wired and Accidental Empires. I try to catch his show this PBS as often as I can. Its just under 10 minutes long and usually brings a smile on my face.

http://www.pbs.org/cringely/pulpit/

Javaposse

News, Interviews and more on the Java Language with Tor Norbye (Sun Microsystems), Carl Quinn (Google), Dick Wall (Google), Joe Nuxoll (Navigenics)

http://feeds.feedburner.com/javaposse

Swedish Public Radio

One of the very best podcasts I listen to is P3 Dokumentär (in Swedish) from Sveriges Radio, the Swedish Public Service Radio.

Switched to Mac

I admit it. I’ve been secretly in love with Apple since OS X. I’ve been very impressed with the OS – I love what Apple did with Leopard – but I never really liked the hardware. Until the MacBook Air. It must be the best laptop on the planet.

I have a corporate laptop from HP. Its not a bad laptop, but the company security guys and the PC admins literally killed it. It takes well over two minutes to start, and they do not seem to support sleep mode or hibernation for security reasons. It runs Pointsec, group policies, and an antivirus software that kills it. On top of this they removed administrative rights on all laptops recently and this rendered it totally useless.

So, being an old firewall guy, I set up the VPN client from my home pc (this is what happens when the security dept screws people’s tools up), so I could still work from home, but I was still forced to use the laptop while traveling.

About a week ago, I traveled to London with a consultant from the office. He brought his Macbook Air, and I brought my HP-killed-by-the-IT-dept-laptop.
We decided to do some work on the flight down on a presentation, and we fired up the laptops. He was going at it (flipped the mac open, used spotlight to find the presentation in the mail, and opened it) before I had a chance to enter my Pointsec password. Three minutes later I was still waiting for the drive to settle down and for Windows to give me a cursor. I must’ve looked exactly like the PC guy in the Apple commercials.

I bought my MacBook Air (the SSD version) yesterday. I run Microsoft Office (for OSX) and the recently released Checkpoint VPN client for OSX 10.5 to communicate with work. I also bought VMware to run a couple of apps at work – mainly Smart Draw 2008.

This is the list of software I installed so far:

  • Checkpoint VPN Client
  • Microsoft Office 2008 for OS X
  • TextMate
  • VMware Fusion
  • MSN Messenger 7 for OS X

So far I am extremely happy. I’m blogging from TextMate, a fantastic text editor. I’ve moved to Mac, I am never going going back.

A week away from home

I’ll be spending next week in London. I’ll be at the Unibet HQ in Wimbledon Monday until Wednesday. Thursday and Friday I’ll be at at QCon London. I like spending time at the UK office, but spending a whole week away from Marianne and Matilda is painful. Really looking forward to QCon though…

“QCon is designed with the technical depth and enterprise focus of interest to technical team leads, architects, and project managers. Some of the speakers include Martin Fowler, Amazon.com CTO Werner Vogels, Spring creator Rod Johnson, Hibernate creator Gavin King, Scrum co-founder Jeff Sutherland, LINQ inventor Erik Meijer and many more. The tracks at QCon were chosen to reflect both the hard hitting problems facing enterprise software today as well as the cutting edge that will become mainstream tomorrow.”

Yeah baby!