What's the fastest browser? Maybe you're measuring wrong

17.04.2012
Most of today's browser speed tests focus on aspects such as JavaScript speed, HTML5 performance, or web page loading times. Supposedly, these tests are here to tell us something about a browser and help us either be proud of our choice or convince us to switch. In this article, I'll question typical browser benchmarks, such as SunSpider or V8, and show you why they're no more than meaningless values that are likely affected by an ocean full of non-controllable variables. Second, we'll talk about the memory footprint of web browsers and why this often-overlooked metric matters more than you think.

Let's face reality: While browser speeds are still a theoretically valid criteria in testing browsers, the latest browser generations have improved on web performance so drastically that some of these benchmarks hardly translate to real-world scenarios anymore. In fact, I'd argue that even IT pros don't realize how scores on these speed tests translate to effects on their day-to-day browsing.

Among all those very scientific benchmarks we forget one factor: the user. Our human eye is incapable of distinguishing the 221 ms load time of a JavaScript applet in IE9 from the 220 ms load time in Firefox 17. Do we even notice if Chrome opens a website in 778 ms versus IE's 953 ms? Humans think in seconds, not milliseconds. Do we really notice that our Facebook timeline appears on screen a fraction of a second sooner in a particular browser? Would anyone ever care? Of course, we geeks love our milliseconds, but we may get lost in perfection and this obsession has infected the entire industry: today, literally all browser makers push out raw numbers in order keep out-marketing each other.

And all users fall for it: from tech journalists to IT pros to the beginner. Again, scripting and rendering speeds may still be a valid criteria to web developers or in certain scenarios (I'm thinking browser-based automation), but it's starting to reach a point where the differences can't be made out by the user. Let's take things a bit further: What really determines how fast (or slow) a website is displayed on your screen is much more than the browser. In fact, the browser is just one link in a chain of many technologies, applications, and devices that determine how fast ITworld or Facebook appears on your screen. Among those are:

I'm sure I forgot half a dozen other things that lurk between the web server and your eyes!

A website's loading time is determined by all of these factors. If one or several links in the chain fail to deliver in time, there's no way to determine the speed of a browser. Therefore, judging a browser by its scripting performance or overall website display speed is -- in my opinion -- an extremely theoretical undertaking with little to no effect on any real-world scenario. And even if browser speed made a visible difference, you'd have to set up a local web server on your own PC and measure browsing speeds for local websites in order to avoid most (not all!) of the factors described above.

Then there's the factor of browser behavior: how do power users actually use a browser? Well, maybe we browse to a website and use a middle- or Ctrl-click to open one or many links in new tabs while we keep reading the current website. In the background, another website is loading, and will likely be ready and waiting for us once we're done reading the active page. That's just one of many different user routines. We're not all robots who type in the URL of a website and wait for it to fully load until we're able to read it. Of course, I don't want to make superficial statements about everyone's browsing habits, but you need to take a good look on how you actually surf the web before choosing your browser based on precise speeds.

Let's put the issue of classic speed tests aside. What (most) of the popular web browser comparisons neglect is that today's browser generation has become more resource intensive than ever. Operating systems like Windows 8 have become increasingly "slimmer" in order to fit on lower end or low-powered mobile devices (tablets, first-generation ultrabooks) or thin clients. One the other side, the browser, which is arguably the most important application on any PC, has become more bloated. And while a big memory footprint on high-end systems may only result in some annoying lag or crashes, it affects the performance of the aforementioned devices, including battery life and overall responsiveness, drastically. And that has a visible effect on your web browsing experience.

Running 15 tabs in your browser isn't an uncommon scenario, right? Have you ever taken a good look at your Task Manager while doing so? On one of my test beds (a 3 GHz Core 2 Duo machine with 4 GB of memory running Windows 7 SP1), Internet Explorer 9 with 15 tabs open takes up a whopping 1,056 MB of RAM. That's a gig of RAM that's being taken up just by a seemingly small application. Plus, these 15 tabs caused IE9 to create nine iexplore.exe processes, each taking up more (basic process units to which the OS allocates processor time) and (the amount of OS resources being accessed) than any other application or service that's running on most systems. Again, on faster machines with tons of memory, this resource hogging will manifest as reliability issues -- some lags and crashes. On netbooks, laptops, older desktops, first-generation ultrabooks, and tablets, it will cause noticeable performance problems.

No matter how you look at it, the browser ranks among the most resource-intensive processes on your machine. Why? First, a web browser is not a document viewer and a website is not a simple document. Browsers are complex and full-blown programming environments. Technologies such as HTML5, JavaScript, and Flash are hugely more powerful and complicated than any document. They're running code! It's no wonder browsers require lots of CPU time and memory. It's a question of what has less impact on your system:

Again, the more memory your browser takes up, the slower it will go, and it will affect the performance of your other applications as well. Plus, keeping your hardware busy means a decrease in battery life on notebooks. Keep in mind that Chrome, Firefox, and IE9 are built as basic 32-bit applications. This restricts them to 2 GB of memory, which increases the chances of reliability issues and freezes. (You could use the 64-bit versions, but these are incompatible with many add-ons.)

Among aspects like security, usability, stability, and compatibility I'd rank memory usage as one of the more important factors when deciding between browsers, as it affects not just browsing speed but reliability as well as overall PC performance.

To determine the memory usage of browsers, I prepared the following scenario:

To test the memory scaling of browsers, I staged three different test runs for each browsers:

All tabs were opened in intervals of 5 seconds. The websites I used represent a multitude of content types, such as static HTML articles, search engines, video content, massive dynamic content and social media sites. I made sure that all websites are made up of mostly static content. For example, I did not use the start page of ITworld but rather the printer-friendly version of an article; I also used the Facebook profile of an inactive user rather than my front page. By doing so I eliminate drastic fluctuations in my memory tests.

I used Process Explorer to determine total memory consumption and to keep an eye on excessive utilization of Windows resources such as handles, CPU time, or threads. After each test, I rebooted the machine to restore all locked memory.

I tested the following browser versions, comparing IE 9 with not just the stable versions of competing browsers but some beta and developer versions as well:

So, how'd they do?

In their official stable versions, IE9 and Chrome used the most memory in the 15-tab scenario. (Expect that number increase drastically if you're the type who switches between 40 or more tabs.) Firefox 10, however, beat them all: It consumed around half (!) of the memory that the others needed in this scenario. Although its basic resource usage ran at 68 MB, it scaled quite well up to 218 MB when loading five tabs.

How does this translate to real-world performance? Both IE9 and Chrome felt sluggish, with noticeable delays in switching between tabs. Also, overall system performance and responsiveness degraded. Firefox had no discernible effect on performance whatsoever.

Now let's move on to the beta releases of Opera, Firefox, and Chrome.

Firefox's beta did not achieve a significant improvement over the production version when opening one or or five tabs, tough it did shave another 70 MB off on the 15-tab test. Google Chrome, too, improved things a tad when only a handful of tabs were running, but it failed miserably with a usage of 1,084 MB at just 15 tabs -- unnacceptable! The beta version of Opera had a difficulty with one of our websites as its memory increased to 789 MB at five tabs. At 15 tabs, over 1,112 MB of memory was consumed, which shows that the beta version needs to go a long, long way to match the memory consumption of version 10. All browsers, with the exception of Firefox, showed visible lags -- it was no fun surfing the web or trying to work with other programs.

Looking at the dev versions of the browsers, Firefox again took the crown, improving its memory usage still further. It's unbelievable. Even though the browser froze up for five seconds during our 15-tab test, it still achieved the best overall performance.

Chrome came in the heaviest this time yet again, with a combined memory utilization of 1,118 MB, followed closely by the alpha version of Opera 12. The screenshot below shows Chrome eating up memory in Process Explorer.

Now, I completely realize that these benchmarks do not necessarily represent your real-life usage. Everyone has their own browsing behavior, which may or may not exceed what I've tested here. Also, factors such as third-party plug-ins, operating systems, background tasks, and many of the other variables mentioned above may result a slightly different picture. But this test should at least give you a general picture of the browsers that tend to use the least and most memory.

If you never open more than five or ten content-heavy websites at the same time, you should never base your browser choice on memory consumption. It simply doesn't matter a lot if your browser consumes 50, 200, or 400 MB of RAM, even on lower-end machines. However, heavy tab multitaskers should steer clear of IE9, Chrome (in all its incarnations), and Opera (Beta). Just go with Firefox. With RAM levels quickly reaching 1 GB with just 15 tabs, you'll encounter sluggishness even on faster machines. Again, it simply doesn't matter if you've got 4, 8, or 16 GB of RAM -- a browser taking up 1 to 1.5 GB of RAM quickly reaches the limits of both the Windows' and its own memory management capabilities. Threads and handles run wild, paging starts kicking in, and overall reliability goes down with responsiveness.

Firefox is the clear winner of the bunch. It was the only browser that did not slow things down and I recommended it for both lower-end mobile devices and high-end desktops.

This article, "," was originally published at . For the latest , analysis and how-tos, follow ITworld on and .

.