Have you walked down the ORM road of death?

A friend of mine asked me a really good question tonight:

Hey Stefan,
It would be great if you could please give me a sense for how many development teams get hit by a database bottleneck in JEE / Java / 3-tier / ORM / JPA land? And, how they go about addressing it? What exactly causes their bottleneck?

I think most successful apps – scaling problems are hopefully a sign that people are actually using the stuff, right? – built with Hibernate/JPA hit db contention pretty early on. From what I’ve seen this is usually caused by doing excessive round-trips over the wire or returning too large data sets.

And then we spend time fixing all the obvious broken data access patterns, by first to use HQL over standard eager/lazy fetching, or tuning existing HQL and then direct SQL if needed.

I believe the next step after this is typically to try to scale vertically, both in the db and app tier. Throwing more hardware at the problem may get us quite a bit further at this point.

Then we might get to the point where the app gets fixed so that it actually makes sense to scale horizontally in the app tier. We will probably have to add a load balancer to the mix and use sticky sessions by now.

And then then we will perhaps find out that we will not do that very well without a distributed 2nd level cache, and that all our direct SQL code writing to the DB (that bypass the 2nd level cache) won’t allow us to use a 2nd level cache for reads either…

Here is where I think there are many options and I’m not sure how people tend go from here. Here we might see some people abandoning ORM, while others may try to get the 2nd level cache to work?

Are these the typical steps for scaling up a Java Hibernate/JPA app? What’s your experience?

Unibet Privacy Proxy

One of the cat and mouse games we play in the e-gaming space is with the regulating authorities is the blocking/anti-blocking game.

To give you some background on what’s going on we need to look at the legal landscape in the EU for e-gaming.

Most EU member states try to enforce a (state-owned) monopoly on offline and online gaming. The EU on the other hand is pro-competition and opening up the markets on equal terms for privately owned operators with a licensing process for each country in line with articles 59 and 60 of the Rome treaty.

Article 59. Within the framework of the provisions set out below, restrictions on freedom to provide services within the Community shall be progressively abolished during the transitional period in respect of nationals of Member States who are established in a State of the Community other than that of the person for whom the services are intended.

The Council may, acting by a qualified majority on a proposal from the Commission, extend the provisions of this Chapter to nationals of a third country who provide services and who are established within the Community.

Article 60. Services shall be considered to be ’services’ within the meaning of this Treaty where they are normally provided for remuneration, in so far as they are not governed by the provisions relating to freedom of movement for goods, capital and persons.

‘Services’ shall in particular include:

* (a) activities of an industrial character;
* (b) activities of a commercial character;
* (c) activities of craftsmen;
* (d) activities of the professions.

Without prejudice to the provisions of the Chapter relating to the right of establishment, the person providing a service may, in order to do so, temporarily pursue his activity in the State where the service is provided, under the same conditions as are imposed by that State on its own nationals.

However, the member states hasn’t been very keen on letting a huge amount of profit from it’s fully-owned state operated lotteries, casino and betting companies be subject to outside competition. Not to mention revenue from the tax on gaming…

As you can imagine there is not a whole lot of interest from the member states to open up the monopolies and risk being subject to competitor. So the EU is taking legal action against these member states and fining them until they do open up the markets.

So not too many member states have had an open market, the exceptions being the UK and Italy. As the UK has had privately held operators in a regulated market for many years the government doesn’t have any interests to protect, but this is far from the case in the rest of the member states.

One of the quirks with what I guess one can call “reversed e-commerce” (when the customers occasionally gain money from using a service), is that the a EU citizen can use any e provider outside of the national borders, but is still still subject to national tax. Also, if a privately held operator wants to apply for a license to operate in a EU member state that decides to open up an regulated market, that operator need to withhold the tax for the customers and also pay tax on its profits.

In order to force the nationals of a member state to only play with the licensed companies in a regulated market, the strategy is to try to cut off access to other online e-gaming services by DNS-blocking (being practiced in Italy at the moment). Sweden has said that they will use IP-address level blocking in addition to DNS blocking to restrict Swedes to only access the licensed betting sites.

This really annoys me. One of the reasons Internet has become what it is today is openness, and I am very concerned that countries in the free world are now using Chinese mentality approach to protect revenue streams from gambling.

And when I get annoyed, I try to help people circumvent these communist approaches

Enter the Unibet Privacy Proxy

So, I decided to set up a proxy in the cloud – in this case using Amazon EC2. I spent an evening setting up a Linux image with Squid (a proxy server) and an Apache web server.

On the Apache web server I serve a proxy auto-configuration file that proxies only URL:s used by Unibet via the proxy. All other URL:s doesn’t use the proxy.

The Squid proxy is an open proxy, but does only proxy URL:s for Unibet servers.

Then, to simplify use and configuration, I spent a few nights writing an add-on for Firefox. I’m looking into doing the same for Internet Explorer.

I love these projects, because I learn a lot by doing them. In this case I learned a lot about both Amazon EC2 and Firefox add-on development! Hopefully I can manage to get the IE add-on done too at some point!

Happy proxying!

The (almost) perfect (rich) website

I am personally a fan of light-weight web pages that use W3C standards based elements and layout. However, many commercial web sites seem to want to move to a more “print-like” experience.

The cost of moving to a richer experience is usually higher maintainance cost and round trip time – you need the graphics or flash guys for many changes. SEO (Search Engine Optimization) suffers as the graphics can’t be indexed by the web crawlers, and you usually take a hit on page load times too.

Wouldn’t it be great if you could make a web site that is:

• Great looking
• SEO friendly
• Quick to load and render
• and is XHTML compliant

We have come a long way at unibet.com, but we made some compromise in look and feel for speed and we also do still have article headers using generated images. This has bothered me for some time. One of our consultant mentioned that he know of someone that used Flash for rendering headlines, and it sounded like a good idea to me. I did some research and stumbled upon sIFR.

sIFR (or Scalable Inman Flash Replacement) is a technology that allows you to replace text elements on screen with Flash equivalents. Put simply, sIFR allows website headings, pull-quotes and other elements to be styled in whatever font the designer chooses – be that Foundry Monoline, Gill Sans, Impact, Frutiger or any other font – without the user having it installed on their machine. sIFR provides some javascript files and a Flash movie in source code format (.fla) that you can embed your fonts into. It’s really easy to set up.

To use sIFR on your website you embed the font (be careful to encode all (but only) the chars you will need) to minimize the size of the Flash movie. Typically the SWF movie is between 8-70kB. This may seem like a lot more than an image, but remember that the SWF will be cached for a very long time in to browser if you’ve set up your web server correctly. Effectively the font flash will only be downloaded once or not at all per site visit.

When you have made the SWF:s you need, just add a few lines of sIFR code into the web page and that’s it.

The following explains the sIFR process in the browser:

1. A web page is requested and loaded by the browser.
2. Javascript detects if Flash 6 or greater is installed.
3. If no Flash is detected, the page is drawn as normal.
4. If Flash is detected, the HTML element of the page is immediately given the class “hasFlash”. This effectively hides all text areas to be replaced but keeps their bounds intact. The text is hidden because of a style in the style sheet which only applies to elements that are children of the html.hasFlash element.
5. The javascript traverses through the DOM and finds all elements to be replaced. Once found, the script measures the offsetWidth and offsetHeight of the element and replaces it with a Flash movie of the same dimensions.
6. The Flash movie, knowing its textual content, creates a dynamic text field and render the text at a very large size (96pt).
7. The Flash movie reduces the point size of the text until it all fits within the overall size of the movie.

sIFR is a clever hack, but none the less a hack. The result is really amazing however. It’s hardly noticeable to the end user and meets all the four requirements I set up in my “what if…” list above so we’re moving to sIFR for the next release of unibet.com.

While sIFR gives us better typography today, it is clearly not the solution for the next 20 years.

Further reading:

The sIFR3 Wiki

Would you want to live in your code?

I’ve had many discussions with developers on if their code is considered to be “good”, “maintainable” or “effective”. Starting today I’ll ask them if their code is habitable.

Any (building) architect would likely want to live in a house that they found had good architecture. It should be solid, functional and effective. The last thing you want is to live in a maze of abstract rooms or to open five doors to get into the kitchen.

To me great code has the following qualities:

1. It solves the business problem, and only that.
2. It maximizes communication. This is really key. Your code should be easy to parse and understand for humans. People are not telepathic so please write code that people can read without having to solve a puzzle or slide into a hell of recursive roller coasters. Is writing documentation often boring? Yes. Does great code need much documentation? No.
3. It minimizes “accidental” complexity introduced by crappy frameworks (remember EJB 2.1?) or old code patterns. Keep it simple. Focus on the business problem.
4. Great code does not take into consideration what might happen at a future date. No one can see into the future, so please just don’t try to as you are adding complexity and impacting time to market and project budget.

A great developer understands that business value provided per dollar spent is what counts.