An Open Letter to Roger Capriotti, Microsoft
Exactly three months ago today, you posted on the IE Blog - "Understanding Browser Usage Share Data". In the post you compared StatCounter Global Stats and Net Applications (aka Net Market Share) data.
In the intervening period we have had several press enquiries regarding your post (which criticizes our methodology and results). Initially, we didn't pay too much attention to your article. However, as these enquiries are still trickling in, we've now decided to explicitly address the points you raised.
Firstly, let us make it clear that we have no argument with Net Applications. They do what they do and we do what we do and we wish them every success. We are concerned, however, that you may not fully understand browser usage share data. For example, why else would you omit key facts (e.g. our vastly superior sample size)?
By the way Roger - we also noticed a strange coincidence. You posted your article on 18 March - the very same day that Chrome first overtook IE in our stats. Interesting... ;) To be perfectly honest, we hadn't even noticed this daily milestone until your blog post appeared and a comment to the post led us to examine our stats a bit more closely.
Now let's examine some unfortunate errors and omissions in your blog post:
ERROR: Browser Usage Stats ARE measured with Page Views
Roger - you made a fundamental error in your blog. Your post has "Browser Usage" in the title, but then goes on to criticize us for utilizing page views instead of unique visitors to measure this.
Despite your claims to the contrary, page views is the only valid metric to look at when talking about browser usage as it directly measures how much activity (or usage) is happening on each browser. Our methodology takes account of how frequently browsers are used and also tracks multi-browser usage by individuals. It is just plain wrong to claim that "browser usage" is measured using unique visitors.
This video might make things clearer for you:
In contrast, Net Apps only 'count' one unique visitor to each of their network sites per day. Here is an extract from their site:
This means that Net Apps do NOT take account of how much a browser is actually used. Think about it. Whichever browser you use first on a given day is all that counts in the Net Apps analysis. Your amount of use of that browser is ignored. Any subsequent use of a different browser is also completely ignored.
You also made a comment regarding unique visitors: "It is also seen as a more accurate way to determine actual browser usage because it is less susceptible to fraud. Wikipedia notes "Measuring browser usage in the number of requests (page hits) made by each user agent can be misleading.""
Roger, and we hope you will forgive us for saying this but, that's nonsense. How exactly are unique visitor stats less susceptible to fraud? We'd love to hear your explanation on that one. Please don't rely on the old Wikipedia-says-it-so-it-must-be-true argument though!
OMISSION: Sample Size - 75x larger for StatCounter Global Stats
A very large difference between us and Net Applications (which you choose to quote) is sample size.
You omitted to mention this Roger - are you not aware that sample size is one of the most critical factors when it comes to comparing statistics?
At StatCounter Global Stats our sample pool covers 3,000,000 websites. Net Apps stats are based on 40,000 sites.
This means that the StatCounter Global Stats sample pool of websites is 75 times larger than the Net Applications pool. This is hugely significant Roger as larger sample sizes generally lead to more reliable results.
Furthermore, Net Applications only offer paid web analytics services which may introduce a bias towards profit-making, ecommerce-type sites in their sample pool. Our free service with optional paid upgrades gives us better variety and coverage of all site types.
OMISSION: Sample Size - Page Views
Across our member websites, we track over 15 billion page views per month. Net Apps do not disclose the number of page views they track.
Net Apps DO state that they track 160 million unique visitors per month. We don't track unique visitors for StatCounter Global Stats so we can't give you comparable information.
Let's examine this a little further though. If we guess that each of those Net Apps visitors is responsible for an average of 5 page views, then that means the Net Apps sample would be 800 million compared to our 15 billion. OR if the average visitor does 10 page views, the Net Apps sample would be 1.6 billion which is again dwarfed by our 15 billion page views.
We believe that this is very important information which was left out of your analysis.
OMISSION: IE Bundled with other browsers by Net Applications
Net Apps state that they bundle IE with other unrelated browsers in their stats. Here is an extract from their site:
We don't. We separately track and report IE, Maxthon and Lunascape - while they may use the same rendering engine these are separate and independent browsers. Bundling these third party browsers with IE is simply incorrect - and it wrongly inflates the IE stats.
Looking at our stats for May 2012 for China IE is at 71.76% while Maxthon is at 3.17%.
|StatCounter Global Stats
Methodology: Separately report Maxthon
Methodology: Inflate IE with Maxthon numbers
Net Applications include Maxthon in the IE number so would incorrectly report IE at nearly 75% (instead of 71.8%) due to this strange methodology decision.
Forgive us for bringing this to your attention Roger, but you happened to omit this information.
OMISSION: Geo Weighting - Re-running the Numbers
Let's take a look at some of the numbers you published on your blog.
You applied the CIA Internet User data as weights to our stats. We welcome you and all our users to conduct further analysis and build upon our stats - that's why we license them appropriately and publish our sample sizes. You failed, however, to provide some extra information: When China is excluded, there is no difference at all to the IE market share as reported in our stats.
|StatCounter Worldwide Share Feb 2012|
|StatCounter (geo weighted with CIA data ex China)||35.8%||27.8%||26.7%|
In other words, if you accept the CIA data as accurate and appropriate then the ONLY country contributing to an increase in the IE global market share position is China. This is interesting for a couple of reasons:
Most of the other browser firms we talk to specifically want to examine stats for "worldwide minus China". They tell us that they use this segmentation for their own internal analytics and describe the browser situation in China as a "huge anomaly". Obviously, though Roger, your preferred metrics are your prerogative - it's just interesting to note your different approach.
In China, IE6 usage is still at over 34% and 9 out of 10 copies of Windows are pirated. Roger, you yourself have spoken out against IE6 and Microsoft takes a strong line on piracy... so it seems odd to rely on China to boost your global browser usage figures. Your prerogative though Roger - usage is usage, huh?
ERROR: Geo Weighting - Incorrect Weighting Data
Roger - you don't seem to have a good grasp of geo weighting. If you did, you certainly wouldn't be advocating inappropriate weights like the CIA Internet User figures.
Your entire argument with regard to geo weighting is predicated on this inappropriate data. This is another significant error in your blog. How can you honestly advocate using the CIA data to "improve" browser usage stats when the following issues arise?
Out of Date
Have you looked at the CIA data? It dates back to 2008 and 2009. It's ridiculously out of date and therefore ignores recent growth in economies like Brazil and India (where Chrome happens to be the lead browser at the moment).
Inappropriate - Includes Mobile Users
This renders the data wholly inappropriate for application to desktop browser stats. Take China for example, the CNNIC estimates Chinese mobile internet usage at 69%. Using the CIA data to adjust desktop browser stats therefore incorrectly inflates IE browser usage share based on Chinese mobile internet users.
Mismatched - User (not usage) Data
Roger - it's completely wrong to use data based on USERS to weight USAGE stats. The two metrics simply do NOT match. Internet user data should NEVER be used to weight browser usage stats - this is because web usage per user varies hugely from country to country. If you insist on weighting our browser usage stats, then please use appropriate data that approximates actual web usage per country e.g. Cisco traffic data
China Influence - vastly overstated
The CIA Internet User data does NOT distinguish between people who access the internet several times per day and those who only go online once every few months. This means that the data vastly overstates the influence of China, for example, when compared to appropriate data sources which measure actual web usage e.g. Cisco.
In fact, when you use a more appropriate measure which reflects the actual web usage per country, China's relative weighting compared to the US plummets. Basically Roger - your decision to advocate inappropriate weighting data based on users means that people in China who access the internet a few times a year are inflating the IE stats - that's just not correct on any level.
CIA Internet Users data is available here. Cisco Traffic data can be obtained here via their Advanced Editor. View Data by Country and exclude Consumer Video, File Sharing, Online Gaming, Voice and Video Communications. Exclude Non-Internet IP and Mobile.
Due to the above issues with the CIA data, we do not believe that anyone should rely on it for the purposes of adjusting browser usage stats. We simply will not apply out of date, inappropriate and mismatched weightings to our data and pretend that the resultant output is somehow "better". Applying the CIA data to our stats would simply mean that the weighted output would be meaningless.
OMISSION: Geo Weighting - Other Important Considerations
Roger - look - you became involved in marketing for IE in early 2010 so let's take a moment to look back to when we launched StatCounter Global Stats in March 2009. Just 5 months later, Net Apps decided to start geo weighting their data. We assume you are aware of that?
This contemporaneous article notes: "What the company [Net Apps] doesn't mention -- but several readers have pointed out -- is that [after applying geo weighting] their operating system numbers are now much more in line with StatCounter's global stats."
Roger you also stated on your blog that "You could of course use other data sources [apart from the CIA data] available to you to account for real world internet population".
What you FAILED to mention was that ONLY StatCounter Global Stats allows you to apply those other data sources. Net Applications apply the CIA weights to their stats by default; they do not disclose their individual country sample sizes and they do not publish their unweighted stats... which does beg the question why not?
ERROR: Pre-rendering - No Impact
Honestly Roger - we couldn't believe you made such an issue of this. If you wanted us to remove prerendering - all you had to do was ask - you know how to contact us.
To your credit, you have updated your blog to mention this... however you also threw in a little comment that "there is no indication of either methodology or what percentage of Chrome share is being removed from StatCounter data". Well Roger, and it really grieves us to say this but, again - you're wrong. We clearly explain our adjustment for prerendering in our FAQ and we also publish a graph of all prerendered pages we track in Chrome. If there's anything else you're unclear about - just ask.
In case you're still having trouble with this - let's look specifically at May 2012 - prerendered pages (not actually viewed) accounted for approximately 1.5% of all Chrome hits. (Compare this to the 4.3% impact it had on Net Apps stats - yet another good reason to use page views instead of daily unique visitors). When this is taken as a % of all hits from all browsers, the figure is negligible. This is why there is no discernible impact on our stats. (There is an impact - but it's so small it's not relevant).
Why DO the stats for Net Apps and StatCounter Global Stats differ?
Let's put aside the verbal arguments and look at the stats to figure this out.
Net Applications show that Internet Explorer INCREASED from 51.87% in Dec 2011 to 54.09% in April 2012.
Over that period, StatCounter Global Stats shows that IE actually DECREASED from 38.65% to 34.07%.
Roger - what countries are driving that increase in IE in the Net Apps stats? Given that you are so confident in the Net Apps data, please do give us some details!
In StatCounter Global Stats, IE is trending DOWN in virtually every single country for that period. So without question, therefore, this difference in stats has absolutely nothing to do with geo weighting as you attempted to claim. If we apply your CIA data as weights to our stats, IE is *still* trending downwards. Check it out:
|StatCounter Global Stats
(adjusted by CIA Data)
(CIA Adjusted by default)
How do you explain that? We can think of no reason other than this: Net Applications must have poor initial data in a large number of countries... and no amount of geo weighting can fix that.
So Roger - let's be clear. You claimed that there were three key reasons why you prefer Net Applications stats:
Prerendering - no longer an issue.
Unique Visitors - you're wrong on this as we've clearly explained - so this issue is gone too.
Geo Weighting - despite our objections to your preferred methodology, we have accommodated you and adjusted our stats above by applying your chosen CIA weights - therefore this issue is no longer relevant either.
The key difference that remains is sample size and we beat Net Apps on that hands down.
Therefore Roger, you simply must concede that our (adjusted) numbers are more accurate (given your stated preference for weighting browser usage stats with CIA user data) than Net Applications. So, as our stats are more accurate and directly contradict Net Apps, we're sorry to have to break it to you that IE actually trended DOWN (and not up) over the period Dec 2011 to April 2012.
If you only have a small sample, you run a high risk of failing to obtain an accurate picture of browser usage in individual countries. This means that your initial country data is flawed and not representative of true browser usage. In our view, if initial data is flawed, then multiplying flawed stats by weights, just compounds the problem.
We therefore suspect that the difference in stats between StatCounter Global Stats and Net Applications is largely due to Net Applications having small and unrepresentative samples in many countries.
This suspicion could be easily investigated. Will Net Apps publish their individual country sample sizes so we can compare? We already publish our sample sizes.
In summary - the Net Applications sample is small (it is based on 40,000 websites compared to our 3,000,000). Weighting the initial unrepresentative data from that small sample will NOT produce meaningful information. Instead, applying a weighting factor will simply inflate inaccuracies due to the small sample size. (And then the fact that the CIA data is wholly inappropriate just makes matters worse.)
Comscore View - StatCounter Global Stats
Roger - you are firmly behind Net Applications (not surprising given their stats are positive for IE and Microsoft). We have also heard Sundar Pichai (Google SVP for Chrome) confirm that our stats match their own internal analytics (which again, you could say is not surprising given that our stats are positive for Chrome).
However a spokesman from ComScore is on record as saying that StatCounter browser stats match what he is seeing: "ComScore, another company that tracks Internet traffic, does not release comparable numbers. But spokesman Andrew Lipsman said in an e-mail that StatCounter's numbers are "consistent with what I've seen.""
Remember Roger - we have absolutely no problem with you or with Microsoft or IE... our problem is with misleading and biased information being published under the guise of a fair and balanced analysis. We have no preference for one browser over another. We are independent and report what we see.
At the end of the day Roger, whether you like it or not, we're telling you this:
On a sample exceeding 15 billion page views per months to over 3 million global websites IE is not doing very well... Don't shoot the messenger though Roger! We just record the stats.
As always, we welcome comments/questions from you (and from all our users) so feel free to drop us a line. We also kindly request that you update your blog to correct the errors and omissions - if you need any assistance with that we'd be happy to help.
All the best,
The Team at StatCounter Global Stats
18 June 2012
Note to Net Applications:
We're sure that, like us, you are working hard to provide the best service you can. We would never have engaged in a public critique of your stats if it were not for the misleading, ill-informed and one-sided article published by Roger on the IE blog. Forgive him - we're sure he never intended to be so unfair.
In the interests of providing a fair and balanced comparison of our services, we warmly welcome your comments and we will happily publish them in full as an addendum to this article. If you prefer not to comment though, that's ok too.
If you are ever In Dublin, Ireland - be sure to let us know. We'd love a chat over dinner and drinks!
(Our treat, of course ;))