When the Web Analytics Association (WAA) reviewed the 2008 edition of the CMS Watch Web Analytics Report, they noted that while the "report is well worth the price," there "was no effort to discern vendors' current or planned compliance with the WAA standards." The WAA standards represent an effort by the Association to set common definitions and descriptions for the various measurement concepts used by the web analytics solutions. A new version was released in September, 2008.

It was a fair criticism, and we decided to address this in the 2009 edition of our research. We asked all twenty vendors we evaluated to provide the same point-by-point explanation of their solution in the manner set out by IndexTools' (now Yahoo! Web Analytics) Dennis Mortensen.

The results: 50 percent of the vendors responded.  They are
Amethon
Bango Analytics
Coremetrics
Google Analytics
Intellitracker
Lyris HQ: ClickTracks
Mobilytics
Nedstat
Unica
WebTrends

We put all their responses into a matrix that's included as an appendix in our report.  But you can now download the matrix here for free.  See our release about it here.

We were pleased to see that most vendors we contacted wanted to participate. At the same time it was disappointing that not all did, especially mega-vendor Omniture. Omniture indicated that they would be releasing this information on their own sometime this year.  We gave a pass to the video analytics vendors, Visible Measures and GlanceGuide, since the WAA Standards did not apply to them in a complete manner.   While there is certainly debate among web analytics insiders as to the relative value of the Web Analytics Definitions, we think you can derive some “on the ground” benefits from using them as a reference in a couple of key ways.

1. Vendor evaluations
You’ll see when comparing responses that not all vendors answer "Yes" in their being able to calculate all of the WAA standard metrics. This may come as a bit of a surprise to those who have sat through demos where vendors spin out beautiful reports, or to those who don’t really think about their metrics requirements until after the vendor selection process. For example, Repeat Visitors is not a metric you can get in Google Analytics, Lyris HQ:ClickTracks or Nedstat. This becomes even more interesting in viewing the responses of mobile analytics vendors who indicate that the metrics are collected, but not available in reports.And, those vendors who do indicate compliance with the WAA standards, may choose wording that you not find to be entirely clear, or may by omission of details, invite additional questions, such as: "What do you mean by that?" Or, "Can you explain that in more detail?" Or, "Could you give me an example?"  The definition of a "Page" provides some illustration. Most vendors answer that their definition of a Page is the same as the WAA, "A page is an analyst definable unit of content." That in of itself is a bit obtuse, but the vendors by and large don’t try to explain in any detail how "they" define a page; much less files or media content. Only WebTrends describes in their definition, "Images, sound files and video are not considered pages."

You may also ask vendors whether they provide reports that are compliant with the definitions "out of the box", or whether they require calculation

2. Report analysis
How many times have you been asked by your web analytics report consumers, "how are unique visitors calculated" or "how is visit duration defined," or a range of other questions that are central to coming to terms with report accuracy and intention? Referring to the vendor responses provides a basis for your reference, as well as establishing a benchmark for your own report development. You may also find that using these definitions as the basis for report specifications prior to development helps educate your report consumers and helps to constrain multiple iterations of report review cycles caused by a lack of understanding of metrics definitions and terminology.   Are the Web Analytics Association Web Analytics Definitions a gold standard? No, but they are a useful tool in your evaluation of vendors, as well as your understanding of the metrics and reports you can produce once you have the solution in production. We invite you to take a look for yourself and check out how your current solution stacks up on its own and against the other vendors. We think it makes for some interesting reading.

Again, you can download the WAA Compatibility ratings here.