Performance testing is a notoriously difficult undertaking. So much so, in fact, that it is sometimes not done at all, or only done when a performance problem arises in production, making some sort of investigation unavoidable.

Testing the performance characteristics of a system in advance of its rollout is particularly difficult, because it's hard to know how to simulate real-world usage situations. Developer and QA-lab setups rarely replicate real-world environments. In the real world, machines have fragmented hard disks, superfluous extra files on the file system, anti-virus and other software running, etc., while end-users do crazy things like start and stop applications, run hard-disk searches, flush the browser cache, leave Gmail, Skype, and other "chatty" applications running in the background, and so on.  This can all affect the performance of WCM or DAM applications in particular. The real world of end-users (and of real servers running in real data centers) is not easily duplicated in a sandbox environment.

Some CMS vendors (for example, PaperThin and Sitecore) provide built-in reporting capability for determining time-to-render for various content elements. But in general, onboard profiling capability is woefully lacking from most WCM and DAM systems.

A few vendors are beginning to delve more deeply into this area. One that does is Day Software: The next release of its Communiqué offering (version 5.3, slated for March) has what I might call (tongue in cheek) pervasive thermometry. Almost any operation that takes (or can take) a noticeable length of time has a thermometer bar or other progress indicator associated with it, and in many cases a comparative bar-graph is available at the click of a button. The bar graphs are drawn using the Google Charts API, which means that a graph can be stored, sent, and managed as a bookmark -- the charts are essentially REST URLs.

As with any vendor -- and especially Day -- you need to be careful that the engineering vision of a reporting subsystem is matched by its usability.  So Day customers will want to test closely when it comes out. 

Until more CMS or DAM systems start offering good tools for profiling or performance monitoring, we recommend that you be sure to address those requirements up front.  Make it part of your requirements process, before undertaking a system implementation, or for that matter, before choosing a vendor. Determine ahead of time what your performance goals are -- then put them in writing, in your RFPs and RFIs. Structure them into purchase agreements as well. "Pay for performance" isn't a bad policy. But if you don't spell it out, it's like anything else; it won't get implemented.