More for less

Since I started working with IT, I’ve always focused on helping organizations increase productivity and/or cut costs. Since I joined Unibet I constantly am challenged by my managers to cut operational cost, with at the same time making the system and platform more performant, available and scalable. You might think that this would suck, but it’s really great fun. Its very rewarding and not too difficult really. My approach is to question everything. Start asking questions! Why? How much does it cost? What value does it provide?

So far I’m looking on a number of different areas where we can cut some cost.

Example #1 – KISS = Effectiveness

After a few month in my new position I started to question the current technical setup we had. We have one site in Malta and another one in Costa Rica. The latter site was set up a few years ago, and they moved some of our markets there for legal reasons. The thing that I was surprised with was that no one was challenging this decision or reevaluating it even though it seemed to cause major issues in production and increased development costs by quite a bit – obviously TTM suffered too. So, I decided to look at what our competitors are doing and I quickly come to the conclusion that they seemed to have a much more straight forward IT infrastructure.

The next step was obviously to try to change this so that we could be more effective, provide a better service, and increase TTM – while at the save time cutting costs. And the way to do this in any organization is to present a business case that explains what the rationale for making the change is. With the help of the colleagues in the IT management team I delivered a business case to my managers which in turn was presented to legal (who had been advocating for setting up the second site in the first place). They were baffled what the actual cost was for the current setup, and also that no one really explained to them before what the implications of their requirement was. As there was very hard to justify the direct and indirect costs by having the second data center in production, we are now, four months later, not in Costa Rica anymore.

Example #2 – Bandwidth costs

So, we run our business solely off Malta, a not particularly interesting rock in the Mediterranean. Bandwidth costs are insanely high in Malta due to the lack of competition in this space – and we require quite a lot of it. Most of our competitors run their systems closer to mainland Europe (London, Vienna, Gibraltar, Isle-of-Man and Madrid to name the most popular hosting locations for e-gaming). Legally they probably take a slightly higher risk by doing so, but they gain better performance as they are closer to the customers and they have lower cost – hosting and bandwidth costs are 30-50% of what we pay in Malta.

For this reason I was curious if it was allowed to run off a Maltese e-gaming license outside of Malta. After reading up on the regulations for the Maltese LGA’s laws and regulations, I found out that its allowed to have everything except the very core pieces of the site outside of Malta.

So, we move more and more stuff onto the Content Delivery Network. Currently we are diverting more than 50% of the traffic to the CDN and hence we could reduce our bandwidths costs in Malta by a lot. ROI from day one!

Example #3 – Support and software license costs

Another huge operational expense is the license fees we’re paying to companies such as Oracle, Bea. Oracle has a really good product that I don’t mind paying for but the issue here was that we payed too much (for too many CPU:s). We had database (disaster) replication using Oracle Data Guard to a server on the same site. We also had as many CPU:s active in the Data Guard as in the production databases. I read up on Oracle license agreement fine print and quickly came to two conclusions: We shouldn’t use more than one or two CPU:s in the replication database. We can consolidate smaller databases and save license costs. On top of this it was fairly easy to look at parts of the application and rewrite it to minimize load on the production database. In about four months we managed to reduce the load on the main database by 50% or more, hence cutting the Oracle licensing costs by the same amount.

As for Bea Weblogic costs, I don’t really see a point in paying them going forward. Application Servers are becoming commodity (as in open, free software), and Bea’s product isn’t really providing the business value to justify its cost. Bea support is infamous for its terrible offshore first line in India, and you get no help from them unless you reproduce the problem yourself, write the test case and submit it. You’re ending up doing what you pay them to do for you. Let me just say, I’m eating a hat if we’re paying for Bea’s services in eight months from now.

Way to go Adobe!

I must say I quite like what Adobe has been up to recently. I actually had a need for a small Flash app today, and since I never get a chance to code anything at work I thought this might be a good chance to look closer on ActionScript and Flex.

I installed Adobe’s IDE (Eclipse plugin) and after about two hours (I had never even seen Flash code before) I had a working little app. I must say Flash development seems to be quick, easy and straightforward. I don’t mind paying for good IDE:s, but I think it is important that the Flex SDK is available for free.

Adobe AIR is basically Flash apps on the desktop (ie outside the browser) with offline support. It’s going to be really interesting to see if it takes off. There are some really impressive apps in Adobe’s AIR Showcase site.

I’ve also been looking at Adobe BlazeDS, which is basically a message-queue push bridge between Flash clients talking AMF and enterprise apps talking JMS. Adobe recently released this product under LGPL. You’re probably going to see some apps using it soon 🙂

Divide and Conquerer

I listened to Randy Shoup at QCon. Randy works in the architecture team at eBay. The thing that I was impressed by with his presentation was the “just-the-facts-and-nothing-but-the-facts” approach and the complete lack of buzzwords and product talk. It was like listening to a really good and concise O’Reilly book. Although I didn’t learn anything new from listening to Randy, it’s always good to get a distilled and well-presented summary of what really works regardless of technology fads.

Partition everything! Partition your system (“functional split”) and your data (“horizontal split”). It doesn’t matter what tool or technology you use. If you can’t split it you can’t scale it. Simple as that. Regardless if you’re using a fancy grid solution or just multiple databases.

Use asynchronous processing everywhere! If you have synchronous coupled systems they scale together and fail together. The least scalable system will limit scalability and the least available system will limit your uptime. If you use asynchronous, decoupled, systems then the system can scale independently of each other.

Would you want to live in your code?

I’ve had many discussions with developers on if their code is considered to be “good”, “maintainable” or “effective”. Starting today I’ll ask them if their code is habitable.

Any (building) architect would likely want to live in a house that they found had good architecture. It should be solid, functional and effective. The last thing you want is to live in a maze of abstract rooms or to open five doors to get into the kitchen.

To me great code has the following qualities:

  1. It solves the business problem, and only that.
  2. It maximizes communication. This is really key. Your code should be easy to parse and understand for humans. People are not telepathic so please write code that people can read without having to solve a puzzle or slide into a hell of recursive roller coasters. Is writing documentation often boring? Yes. Does great code need much documentation? No.
  3. It minimizes “accidental” complexity introduced by crappy frameworks (remember EJB 2.1?) or old code patterns. Keep it simple. Focus on the business problem.
  4. Great code does not take into consideration what might happen at a future date. No one can see into the future, so please just don’t try to as you are adding complexity and impacting time to market and project budget.

A great developer understands that business value provided per dollar spent is what counts.

Two truths in one hour

After a busy day at the office I managed to get to QCon just in time for the keynote by Martin Fowler and Jim Webber. Both of them are as you probably know great speakers and the keynote was entertaining. They had slides on why EAI, SOA and ESB:s are a joke (as a concept, to maintain), but I can’t say they provided any alternatives. They basically said:

  1. Focus on business value from the first iteration
  2. Keep it simple

In reality though, you often get the most business value by buying 3rd party software. And it needs to be integrated… Not that I enjoy it though. If you build it yourself, then obviously light HTTP services is the way to go.

A week away from home

I’ll be spending next week in London. I’ll be at the Unibet HQ in Wimbledon Monday until Wednesday. Thursday and Friday I’ll be at at QCon London. I like spending time at the UK office, but spending a whole week away from Marianne and Matilda is painful. Really looking forward to QCon though…

“QCon is designed with the technical depth and enterprise focus of interest to technical team leads, architects, and project managers. Some of the speakers include Martin Fowler, Amazon.com CTO Werner Vogels, Spring creator Rod Johnson, Hibernate creator Gavin King, Scrum co-founder Jeff Sutherland, LINQ inventor Erik Meijer and many more. The tracks at QCon were chosen to reflect both the hard hitting problems facing enterprise software today as well as the cutting edge that will become mainstream tomorrow.”

Yeah baby!

E-gaming vs the Financial industry

I work long hours at Unibet. I spend too much time in meeting rooms. But to be perfectly honest it beats the hell out of working at a bank (which I did as a consultant a few times) or at an insurance agency (god forbid)…

The pace is incredible, and the average age in the company is 27 or something. Plus I get to manage a group of really smart and friendly people (the architects!) . And I have a say on most important decisions that are being made in the company. Cool stuff!

In the Limelight

One of the first things I did when I joined Unibet was to set up a Content Delivery Network. I did some research and ended up with a shortlist.

A few factors were limiting my options:

  1. The CDN provider must do business with e-gaming companies
  2. The CDN provider must have an SSL CDN service

The first point effectively rules out Akamai and a number of other companies. The second point rules out even more companies.

I ended up talking to Limelight, and despite some screwups at their London sales office, I must say their CDN service is really awesome. Highly recommended. We currently use them for web site acceleration, to host downloadable clients and for banner serving.