Hello, I’m Imad Mouline, CTO of Gomez and like Eric, I’m very excited and proud of last week’s Platform Summer Release, especially the positive feedback we’ve been receiving from customers and the industry.

I’ll be a frequent voice here on the Gomez Blog and I’m looking forward to sharing my observations about the world of Web performance and Web application delivery management. In fact, I’m looking forward to having more than 140 characters to communicate about these things than I do when I tweet about them on Twitter (where you can find me at @imadmouline.) It’s been sometimes painful spending more time trying to squeeze into 140 characters the results of a data mining exercise than it did to formulate the thesis, create the query, run it across billions of records, and analyze the data! Maybe I’m just old-fashioned, or, as some of my colleagues would argue, long-winded. I haven’t given up on tweeting so only time will tell if I can get proficient, or at least, efficient, on that medium.

So back to the topic at hand: why does our recent platform upgrade focusing on the need for multi-browser testing and monitoring? Sure, cross-browser testing is important. However, shouldn’t it be limited to design validation and functional testing? Why on earth would anyone want to or need to monitor a Web application across multiple browsers?

How big can the difference in browser performance really be? Let’s take a look.

There are lots of sites that do browser benchmarking. I am particularly impressed by a Microsoft benchmark (link: http://www.microsoft.com/windows/internet-explorer/videos.aspx Click on Case Study Videos -> Performance Testing) where a team painstakingly built a carefully calibrated environment where no parameter was left un-controlled: various layers of caching, bandwidth, latency at multiple levels, all were controlled and documented. A camera was set up to capture, every 1/100th of a second, how the various browsers build the target Web pages. The team captured the performance of various browsers as they were loading the top 25 most popular Web pages in the world. What was most revealing to me wasn’t which browser won at the end of the day. It was that, when you broke the data down, IE8 was the fastest in 12 out of the 25 cases. Just because Browser A is faster than Browser B when loading page 1, doesn’t mean it’ll be faster when loading page 2. A little less obvious, but just as important: just because page 1 loads faster than page 2 in Browser A doesn’t mean that it’ll load faster in Browser B. Do you see where I’m going with this?

We decided to get out of the lab and look at some data captured in the wild.

We looked at the aggregate data that our SaaS-based real-user monitoring solution collects. We took a sample of over 40 million pages across several dozen Web sites, measured over a 5-day period. We restricted the measurements to just those real users connecting from broadband connections in the United States. We further narrowed it down by excluding anyone using a hand-held device or PDA. The latter restriction was made so we could compare not just raw page load times, but also perceived page load time (i.e. the amount of time it takes for the visible portion of the page to appear to load in the browser), which tends to vary widely based on the size of the screen. Don’t worry; we’ll spend some time on that topic in a future blog post.

First, a quick reminder of how page load time, perceived render time, and response time relate to each other. In a nutshell, page load time, a.k.a. raw page load time, is the amount of time it takes for the page to load. Perceived render time is the amount of time it takes for all the visible portion of the page, typically images, to load or error out. Response time is the amount of time it takes to navigate from the previous page until the current page loads.

Before we dive into the numbers, a quick disclaimer: We are not stating that Browser A is the fastest, or that Browser B is faster than Browser C. As I mentioned, one cannot generalize and make that leap. For now, we’ll keep the actual names and versions of the browsers hidden, so we can give you the opportunity to focus on what the data is telling us. This also gives you an opportunity to speculate. (I’ll send a prize to the first person who guesses which browser is which!) While we have measurements for about 100 browser types and versions, we selected the most notable. Included in these numbers are measurements for IE6, IE7, IE8, Firefox 3, Firefox 3.5, Safari 3, Safari 4, Opera 9.6, Opera 9.8, Chrome 1, Chrome 2, and Chrome 3.

Now, let’s take a look at the data. (All times in milliseconds)

We could spend a lot of time and a lot of virtual ink analyzing and discussing these numbers and some of the subtleties behind them. But before I share my analysis, I’d like to get your reaction to this data.

So, what are your thoughts on the methodology? And the results? Is this representative of your experience with different browsers? Is there a different slice of the data you’d like to see?

Let us know!