The Problem of Data Quality for Unstructured Content

For me this raises the question of quality -- especially when dealing with unstructured data.

"On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' ... I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question." (Charles Babbage, Passages from the Life of a Philosopher).

Babbage's thoughts on the subject of data quality, were neatly summarized by George Fuechsel a century or so later as, "garbage in, garbage out," or "GIGO."

Understanding of data quality in the world of structured data (think ERP, CRM, BI) has reached a very high level of maturity. Unfortunately, the same cannot be said for the world of unstructured data, a.k.a., content.  Ensuring that same level of quality for unstructured data such that it doesn't skew subsequent analysis is much harder to apply.

Use Cases on Offer

Listening to Oracle discussing the possible outcomes from Big Data, you hear many references to use cases such as "smart meters" in domestic scenarios, or medical sensing equipment attached to patients. These examples certainly when scaled-out can produce vast quantities of data, and almost certainly that data will provide valuable insight once analyzed.

I would argue though that these are pretty limited use cases that simply extend existing applications.

They ignore the massive amount of true content: from short-form social, to long-form document text. Is this because such content is inherently not useful, or that the problem of quality makes it too hard to glean actionable results?

Reading IBM's own commentary on how they plan to apply their new Vivisimo technology to this problem suggests that they have at least recognized the issue exists. IBM envisions Vivisimo as a kind of content curation tool: federating sources and assembling data sets that have been filtered for quality and faceted together into logical collections. However, while this appears to be sensible in theory, it begs a question.  Why Vivisimo, rather than their pre-existing Content Analytics/Omnifind technologies? Might Enterprise Search find a new role across the board in this emerging area?

What You Should Do

There is certainly right now a paucity of solid business cases for Big Data in the enterprise. Certainly not a shortage of ideas and theories, but customers are still primarily sandboxing sub-sets of data, looking for indications that there are a demonstrable returns on investment to come. As you look for suitable use cases, and your Big Data explorations turn more to unstructured data, remember GIGO and don't lose sight of data quality.

The project in question has weathered some turbulence – execution delays, budget overruns and most critically, lack of end user enthusiasm for the delivered solution.  

At the surface level, they seemingly did everything right and all the boxes can be safely checked off. However, careful reflection reveals that they ended up where they ended up and not where they wanted because of the disconnect between Marketing and IT. In this instance, marketing drove the project with the assistance of a 3rd party integrator, and the internal IT team was not fully on-board till very late in the game. Important issues like global training, scaling up, ongoing support and service levels were left as an afterthought.

Suffice it to say a sound DAM (or for that matter, any IT) project requires all stakeholders to be aligned from the beginning or else you'll end up with bad karma, and a system that is not fully adopted.

In addition to our cornerstone evaluations of technology vendors, in our DAM Report you will also find sage counsel about the pitfalls that you'll encounter during your DAM project life cycle. While attaining DAM Nirvana is a difficult goal, we at RSG do our bit by at least pointing you in the right direction.

Had any enlightenment of your own recently? Tell us about your experience.

It is not a big financial deal ($50 million US) -- but still significant for Lexmark if previous experience is any guide. The company has tried agressively to take Perceptive beyond its comfortable North American home market and raise its stature in case management-centric industries such as Government and Healthcare worldwide. 

From Perceptive's perspective Pallas-Athenas has some interesting products, "BPMone" being the flagship. What differentiates this product from other small BPM vendors is its focus on process mining and analytics, capabilities more often found in much more expensive product sets that target Case Management. I think the challenges here are going to not only be the obvious one of integrating two disparate product sets (BPMone  will make little sense as a standalone product at Lexmark) but rather in selling Case Management to Lexmark's existing customer base of SMBs (small and medium businesses).  Of course, it doesn't matter to me whether Lexmark is successful here or not. 

What is of concern and interest to me is the rapid rise in demand for process-driven Case Management in larger organizations. This is an application-based approach to Document Management that has come with  an equally rapid drop in interest in using the same software as an enterprise platform. The E in ECM is quickly becoming anachronistic, with a back-to-basics approach from buyers of the technology. Buyers seems to be saying, "To hell with my broad enterprise needs, I have a very specific problem  -- application processing, accounts payable, contract management, and so on -- and I want it fixed now."

Case Management applications theoretically answer that call by building on workflow and document management technology. This is not something particularly new, but in the past building such applications was massively complex and an onerous task to take on at all. But there have been huge improvements over the past couple of years in terms of basic usability, a move toward configuration over customization, and most importantly insight (via analytics) into active and planned processes. Together these make Case Management relatively easy to use and able to deliver some kind of value on the initial investment. Case Management is a trend I don't see falling off soon, but then again everything is cyclic -- like the size of a collar or the length of a skirt. In a few years time perhaps disillusionment will have set in and we will be back to talking about document management as an infrastructure component once again...

The answer people always prefer is of course, "Actually quite a lot, we've been working in this sector for a while."  By sector I mean industry "vertical," such as manufacturing, retailing, banking, insurance, healthcare, and so forth.  If you understand my business you are a part of my world; if you do not then no matter how clever you may be, you will always be an outsider.

That's basic, yet in the world of content management it's not a message or practice that is commonly followed. Most vendors try (with mixed success) to sell to anyone and everyone.  They argue that the technology is "horizontal" as opposed to "vertical," and that it will work equally well in any industry.  There is a validity to this argument, particularly in the world of web publishing.

Nevertheless the more complex the underlying business process the more the need for somebody who understands your business, My goal here is not to give business advice to sellers of technology, but rather to posit the opinion that buyers are quite right to have a heavy bias toward those that understand their specific industry.

True, most organizations are nothing like as unique as they think they are. Once you have seen one transactional document process you have to some extent seen them all. Likewise one media heavy website is much like another.  Further, the underlying technology used in these situations may have different branding, but they tend to be very similar at core. 

The difference and the value is in detail.  Whether you call it a document, content, a case item, a record, or a deliverable matters. Whether you navigate via product types and sub-types or locations and offerings matters.  As a buyer you want your content management system, sites, and services to be better than your competition; you also want it to be comparable to the competition.  Familiarity matters.  You want to work with a supplier that you can understand, that has an intimate understanding of your world and concerns.  Suppliers you can have conversations with, and and built a long term relationship with. 

The horizontal approach to selling and buying technology makes a lot of sense in the consumer world, but much less so in the enterprise.  The one size fits all mentality seldom works, though that does not stop a lot of suppliers from pursuing any and every opportunity no matter how far outside of their typical market reach and comfort zone. 

It's why we at the Real Story Group consider vertical expertise to be an important factor in our ECM product evaluations, for example.  A vendor might have a great technical solution but zero knowledge of the insurance sector (for example) will often rule them as unsuitable to a insurance-focused buyer.  Speeds and feeds are important, usability and functionality equally so, but a simple and genuine understanding of a buyer's world has just as much, if not more importance in the technology buying process.

Recently, I’ve seen numerous organizations churning out massive amounts of content using faux-CMS methods such as:

Content entry: Word-based forms
Workflow: E-mail and phone calls
Content reuse: server-side include files tracked via spreadsheets
Collaboration: shared network drives
Versioning: multiple versions stored on shared network drives
Archiving: PDFs in shared network drives
Deployment: Simple FTP

To their credit, these organizations have done an impressive job of cobbling together a variety of tools and processes to mimic the functionality that comes with most content management systems. In most cases, these homegrown solutions have served these organizations well -- thusfar.

However, the changing demands of content consumers are putting stress on these methods. With increased content demands, more and more production departments are hitting their breaking points and becoming bottlenecks for publishing important content.

Is your organization one of these?  The good news is that in 2011 there are more options than ever for a Web CMS buyer – we cover 44 of the most significant. All 44 of these commercial products and open-source projects have been around for years and have matured to the point where their base functionality could provide operational wins for you. Your challenge will be to find the right partner to support your visions for the next three years and beyond.

Over the past few years a number of leading US hospitals have subscribed to our research and advisory services as they attempt to better leverage and co-ordinate broad content management needs. The panoply of use-cases -- EMR/EHR, diagnostic tools, rules driven scenarios, patient management, self management, and education -- all demand (in theory at least) a seamless operating and integration environment.  Inconsistency and potential contradictions in information can be critical in clinical settings.

Of course, none of this is new.  However, a co-ordinated and updated approach to managing information and content is rising on many hospital agendas -- due in part to increased US government funding, a general drive for greater efficiency, and a highly litigious and competitive commercial environment. 

Alas, the fact remains that few suppliers of content management technology have any substantial presence in this sector. Fewer still have the domain expertise and proper grasp of these highly complex and industry-specific requirements to be able to deliver much real value.

That said, some vendors are trying to build up their presence and offer better solutions. Hyland has long had a presence, and now Xerox is trying to get a foothold for its DocuShare Suite via its acquisition of WaterWare (a small firm that had build an EHR application on top of DocuShare).  But the major players you would expect to see regularly in this space -- such as EMC, Oracle, IBM, and Microsoft -- have patchy reputations and a history of trying to fit generic solutions into specialized environments.

In fairness, the healthcare sector is itself disconnected and difficult to navigate, where suppliers are severely challenged to develop high quality, co-ordinated solutions.  Clinicians usually rule the roost, and their expectations and requirements are often out of line with IT reality. Similarly major hospitals consist of a multitude of near autonomous departments, making co-ordination and alignment between them near impossible at the best of times. 

Nevertheless, healthcare is an area where ECM can deliver profound benefits. Healthcare (in both its private and public incarnations) also represents an industry sector in real need of serious improvement when it comes to managing information resources.

Current investment by document management and broader ECM vendors in developing better BPM, analytics, integration, and case management capabilities could ultimately deliver some valuable new technology solutions.  But in today's healthcare world, ECM is poorly represented in the broader technology portfolio.

That question has been asked every year for the past decade, and though in early 2008 we stated that DAM was "always the bridesmaid and never the bride," it seems at least it's on a serious date. Pure-play DAM vendors like North Plains are expanding services teams, ADAM landed a few big-name clients in 2010 and grew its team by 20%, MediaBeacon added a raft of new functionality to its software, while Autonomy and OpenText are attempting to keep their longtime MediaBin and Artesia customers via an upgrade to major new platforms. 

The action and growth in the DAM world can in some part be credited to SMBs outside the media, advertising, and broadcast industries taking up an interest in this area. We have subscribers in the non-profit, medical, and government sectors who have started to ask how they can better manage their photographs and video, in particular, and are educating themselves in the discipline of DAM, as well as vendor weaknesses and strengths.

Meanwhile, longtime DAM system users from the M&E or ad agency world are looking at taking the next step - be it upgrading their DAM to a more modern, multi-faceted tool, or turning their system into something outward-facing to service customers or other 3rd parties. 

Our 2011 DAM market analysis takes a look at these trends, and delves into depth on how each of the 20+ vendors we cover plays a part.  Stay tuned - if you're not already managing multimedia or brand assets, you will be soon.

Transactional Document Management encompasses process-centric functions, from imaging, forms and capture, through workflow, to case management.  It's not sexy or exciting like social collaboration; there are no funky, Facebook-like features. Watching transactional products getting demonstrated is like watching paint try.  The user interfaces are essentially Windows 3.1 lookalikes, and the most exciting thing you can expect to see is a form moving magically from one folder to another. It can be hard to perceive the magic at times, but it is there, under the covers, if you know where to look, and you need to look deep.

We evaluate a wide range of vendors and products that play in the transactional space, from IBM and Oracle, through Hyland, Perceptive, and NewGen in our ECM Report.  I can tell you that some transactional products do well in simple, high-volume environments, while others excel at complex capture, and others still at sophisticated business process management.

They differ further in the vendor's understanding of particular vertical environments. For example, one vendor does really well in healthcare, where they have a very thorough understanding of the particular business processes around client billing. Another understands eGovernment and all the related forms processing required there. Another still does well in customer support, analytics, and exception management.

It's easy to forget about the primary importance of this branch of ECM, as the collaborative branch gets all the headlines (think SharePoint and Social Media), on account of seeming so much more exciting -- or at least less dull.

But the transactional side of the equation is something we have no intention of forgetting. If anything we'll be furthering our coverage in this area in the new year. The reason is simple. Buyers of ECM systems are increasingly asking us about transactional systems. That is what drives our business: you the buyer, not market noise and hype.

• Help manuals for your products or services
• Electronic bills sent by utility service providers
• Insurance quotes and policies

Dynamic Publishing is a catch-all phrase and is often referred to in context of various disciplines such as Component Content Management, Automatic Document Assembly, Document Output Management and Document Automation. In our recently released advisory, we explore Dynamic Publishing Systems and what what their capabilities should be. We also explain how these should be considered not in isolation but in the context of your wider Enterprise Content Management (ECM)  initiatives. To quote from the advisory:

Don't consider DPS in isolation: Dynamic Publishing should not be considered as a standalone system. Content and documents that are within the scope of Dynamic Publishing are often subject to the same stages and processes as any other content, such as publishing workflows, version management, archiving and so forth. Hence, you should consider Dynamic Publishing as part of a broader content management initiative. It is very important that the DPS you select has basic ECM capabilities (repository services), or provides connectors to integrate with external ECM, search, and portal tools.

Here's the table of content of the advisory paper Dynamic Publishing Systems -- A Technical Overview

• Key Takeaways
• What is Dynamic Publishing
• Functional Requirements of Dynamic Publishing
  • Content Assembly
  • Rendering, Transformation, and Conversions
  • Graphics Support
  • Process Management
  • Output Management
  • Multichannel Translation Management 
• Key Advice

This acquisition is an unusual move, as for many years Hyland has been synonymous with document management and workflow solutions for the mid-sized banking and healthcare sectors. With this acquisition, Hyland takes a fairly serious stake in the higher education sector. But not a dominant stake: the higher ed sector has many players; for example Perceptive Software has over 400 clients there, and its a highly competitive market to get into.

Of course it's up to Hyland to decide how to run their business, and where to target their efforts, but it's for their customers and the likes of me to question if this acquisition will prove to be a distraction to the current customer base. I can't help but conclude that it will, as the Hershey acquisition not only takes them into a big new market (though to be clear Hyland did have a few higher ed customers prior to this) but it also delivers a completely new and overlapping ECM technology set. Hyland have not just bought market share, they have bought a lot of new technology. For a company that has long built and developed its own technology, it's hard to predict how this will play out. 

It may be that Hyland will take the best of both technology sets and develop something impressive, but that is hard for even the likes of Oracle and IBM to pull off. They may continue to run the two product sets completely separately, but then there will be overlapping resources and functionality and potential rivalry and confusion. Hyland could just let Hershey do their own thing but as Autonomy and Open Text can attest, that isn't easy either.

Many analysts including yours truly, as well as Hyland's arch competitors, tend to speak respectfully about them as they epitomize the ideals of midwestern honesty and friendliness when doing business.  This acquisition though will test that culture I think, and though I can see no reason for existing customers to get worried quite yet, it wouldn't be a bad idea to watch how things progress with a cautious eye. There may be lots of kisses now at Hershey, but they may not last. (Sorry, I couldn't resist that one!)

Enterprise Content Management IBM-style is typically a million miles away from ECM Microsoft-style. Rather than the broad collaborative environments SharePoint targets, IBM goes after high volume imaging, intensive and complex rules, as well as case and process management. Layering analytics into the equation is only really possible when you have normalized incoming information/documents at the capture stage. Hence this acquisition.

IBM already had some pretty good capture capabilities, but Datacap offers better, and in particular Datacap had a number of industry specific solutions that could be of great interest to IBM. In fact it may well be this ability to extend FileNet more deeply into sectors such as Healthcare and Logistics, as well as further enhancing their existing presence in Insurance and Government, that drove the deal forward.

From a buyers' perspective many acquisitions are problematic, but this one might prove positive.

Let me explain.  To achieve solid business goals with ECM you often have to bolt incompatible components together and hope for the best. Vendors' response -- the ECM Suite where everything worked out of the box and in any configuration -- is still just a dream. Nevertheless, some ECM Suite offerings are starting to look better integrated and a bit more uniform, whilst yet others still look like Frankenstein's children.  Some ECM acquisitions are little more than standalone cash cows, others (like this one) could enhance existing products.

I don't doubt for a second that we will see more ECM related acquisitions (some of which will be capture related) before the year is out. But far from this being a sign that the market is consolidating and shrinking, it is rather a sign that the market is growing and getting ever more competitive.  And as always I will make a point of saying that only time will tell how well this acquisition works out. On paper today it seems pretty sound.

To give a practical example, a healthcare professional will need awareness of all the documents related to a particular patient. These documents and records are sorted and managed through their lifecycle as a "Case" even though they may reside in different locations, have different owners, other relationships, and different retention policies.  Other individuals may also need to interact with these documents for the purposes of billing or insurance. Same documents, different purpose. There may also be multiple legal and compliancy requirements to attend to.

In theory at least, Case Management provides you with the tools to pre-define and orchestrate those requirements. Permissions, rules, metadata, and processes all play a part in what can be a highly complex system. 

For some organizations, Case Management applications built from ECM platforms form the core of their business, and more will in the future. The need to better manage the massive volumes of transactional documentation is growing more acute, and Case Management will certainly play an increasingly important role. 

Yet almost more than any other scenario, Case Management demands good information governance and squeaky clean relevant data. Without it everything falls apart. The fact that so many organizations are lacking here is another key reason Case Management is not as widely deployed as it could be. 

Selecting the right software to meet your Case Management needs is difficult, since everyone claims to do it,  but very few do it well.  The nightmare scenario for a buyer of a Case Management system is to buy a vanilla ECM software system and then just bring in a .NET or Java developer. You are not only buying technical functionality you should also be buying deep and very specific domain expertise, and without the right combination of the two you can be in trouble quick.

ECM vendors such as Hyland, Objective, Open Text, EMC, Autonomy and IBM all have deep expertise and knowledge in the particular industry sectors that they design systems for (Pharma, Legal, Government, Intelligence, Healthcare, Insurance,  Law Enforcement, Retail etc). They know (mostly) what works and what does not, and they understand industry specific business processes right down to the task level. You are paying as much for that knowledge, as you are for their software.

Assuming though that you do have your document house in order, and already utilize Case Management, there are some interesting developments on the near horizon -- most notably the use of business intelligence and analytics tools to extract further value from what is already a rich information set.  Consider the possibilities of early fraud and discrepancy detection or new and emerging trend analysis from the very rich data within your documents.  BI has long been locked solely into the 20% of data that is structured in the enterprise, and is a valued tool set. But very large and clean volumes of documentation that have been given a tight structure can be mined these days too, and those documents theoretically at least, represent the other 80% of the data we deal with in business.  In fact some organizations are already starting to use tools like Cognos, Hyperion, and Business Objects in Case Management deployments, and they are liking what they see.

And remember it's our job here to ensure that technology buyers make the right decisions via the use of our research, and one of the best ways for us to do that is to continuously talk with buyers and users who are at the coalface.  So if you are an organization that is using Case Management along with some kind of BI tool, then  I would love to chat with you in confidence to hear more about what works and what does not just drop me a note and we can chat.

Ironically, it's early days for a combination of technologies that have been with us separately for many years. Yet this could prove to be a very good long term marriage.

Though not really a surprise that Documentum wants to build up its Case Management credentials in this burgeoning area (just think Healthcare, Government, Litigation and Insurance), it's far from clear why anyone would chose EMC Documentum over any other vendor's Case Management offering.

Case Management solutions vary widely, from pretenders to fully fledged players. Everyone from Autonomy to FabaSoft to Oracle has an offering,  There are even some third-party options available for SharePoint. But how each vendor approaches Case Management differs, since specific buyer requirements differ widely too.  Some enterprise buyers seek good BPM capabilities,  while others seek complex metadata management services, records management and retention or even the relative ability to manage files that can range from physical items through to rich media in the same context.  

There is much talk amongst consultants and vendors of the "commoditization" of ECM, but I see little evidence of that in the real world.  Sure, anyone can offer you check in/out and version control and workflow -- but not everyone does those things well, or even adequately -- and some do them too well and deliver you complete confusion and overkill.

One thing is for sure: one size most definitely does not fit all for Case Management.  Therefore, any buyer without a very solid handle on their business requirements and processes is walking into a selection minefield.

As for EMC Documentum, I gave up trying to understand their ever changing strategy(ies) some time back.  From once being the clear market leader they are now in an ever more crowded and ever more specialized and complex field (we cover about 30 in our evaluation research).  That said, EMC Documentum continues to be a key player, and we will continue to evaluate them, and on occasion recommend them to some advisory clients, assuming Documentum makes sense for that client's very specific needs.

People outside of highly regulated environments typically assume that records and retention management in sectors like healthcare, financial services, and energy is state of the art.  This is far from the case.

Most often Records Management is underfunded,  inadequately resourced, and unloved.  It is no cliche to say it is also often run from underground, literally in basements out of sight of those who need to do "real work." 

In places where records are actively managed, they are usually managed well, with detailed and well-maintained file plans and retention schedules.  But many organizations only manage a fraction of the records they should be.  For example very few records management departments include e-mail as part of their remit, yet e-mail is where all the "stuff" happens. 

But maybe, just maybe, the tide is starting to turn.

Here at the Real Story Group we currently support a number of large advisory customers who are looking at records management strategically across highly complex working environments.  We have others who are looking at the whole issue of information management and ECM more strategically than in the past, and have begun to include RM as a component or a recognized future element of their work. 

That might not sound like much, but just a year or so ago it seemed like nobody cared.  Things like e-government initiatives, healthcare reform, demand or need for more self-service applications, increased regulations, and so forth seldom have the immediate impact people expect. And in a world where decisions can get driven by today's opinion polls, it's easy to lose sight of the fact that many of the biggest and most profound changes we encounter in our society occur at a far slower pace.

The most overused phrase in my personal lexicon is, "time will tell." But I fervently hope that time will tell us that awareness of the importance of RM and archiving has slowly risen to the fore.

"Remember the clip board and all the forms you had to fill in, and remember also that the healthcare professional seemed to have a lot more paperwork in addition to yours?" I then explain that Document Management is the process and management of all that paperwork, often with the goal to improving the process and making the paper electronic.  As a working example it tends to resonate with people, and they typically "get it." 

Of course after that, they tend to say either out loud or to themselves, "Wow, that must be the most boring job in the world." 

Hence I could not help but be struck by a new marketing and sales campaign launched by Hyland this week. They're promoting a new offering that combines OnBase document management and workflow software with kiosk hardware from PFU, a Fujitsu group company. They have launched industry-specific (healthcare, government, higher education) kiosks for customers to self serve their forms and related paperwork.  It such a simple and smart concept that I am frankly surprised nobody has done it before. 

Hyland is one of the vendors we evaluate in depth in our ECM research stream, so if you are one of the organizations out there considering their products check out our critical reviews first. 

As an aside, I also hope is that Dr Gustafson on Ayer Road along with the Nashoba General Hospital in Groton will read this short post and agree with me that self service kiosks might be a good idea. 

To quote:

Workflow services can help minimize the cost and time required to coordinate common approval processes -- but only if the service does what you want it to do, and users don't "work around" the system....

Subscribers to our WCM and DAM research streams can download the workflow paper here.

The launch of Apple's iPad last week has caught the imagination of armchair critics worldwide:

"What's it for?"

"What a funny name (snigger snigger)"

"It's just a big iPhone"

And so on.  But whether the Apple device itself is a success or not, we seem to be approaching a tipping point for improved user interfaces to manage documents.

Be it the Kindle, Tablet, Nook, or iPad, keyboardless document reader devices are on the verge of becoming mainstream -- at least for consumers. For browsing and reading through large volumes of files, or large documents containing multiple pages (books and libraries), such devices are infinitely more user friendly than the current desktop or laptop paradigm.

Take that one step further and it is logical to reach the conclusion that such devices might work well for reading through airliner maintenance manuals, consulting documentary evidence in court,  searching archives, or accessing a patient's medical records and images?  So while the pundits mock the iPad, I see real potential here for the world of case, document and records management.  That said, I already own too many Apple devices, and may have just drunk too much Apple-flavored Kool Aid. But surely it can only be time before some enterprising vendor starts to deliver secure organizational and access applications for these devices.

I for one wish them luck, for as somebody who has spent his career digging through virtual crates, accessing electronic files that I then need to print to actually read, I know for sure that there has to be a better way.

Last week, I blogged about the increasing trend toward specialization in the Search & Information Access space. As you may know if you've been reading our Digital & Media Asset Management Research, the DAM industry is yet another area where specialization is ongoing. One trend that's helping drive the verticalization of DAM is the broader use of video in Web publishing and in enterprise scenarios.

Video, as a digital mode of communication, is nearly ubiquitous. This means video asset management (as a capability within DAM) will assume ever-greater importance in the months to come. If you're in the market for a DAM system, you'll want to think about what this may mean for your overall content management strategy -- and take video requirements into account when shopping for a DAM system.

Depending on the business you're in, your use of video may not be extensive today, but it may well become a key content category for you in the near future. Just as the podcast phenomenon suddenly found many companies in the business of managing MP3 files "overnight," pervasive video will likely find many Web CMS owners wishing they'd thought through the vicissitudes of Flash and MPEG4 ahead of time.

Video is becoming more important in broad intra-enterprise cases as well.  Many (if not most) companies have security cameras stationed in their stores, offices, or on company grounds. What happens to all the security-video footage? In some cases, old material is simply destroyed after a certain amount of time. But it still has to be cataloged and stored short-term (then dispositioned appropriately). Is it safe to just manage such footage in ad-hoc fashion? Maybe. But maybe not. What happens if an employee sues the company after (for example) suffering an accident on the job? If the accident was caught on video, the video becomes a key piece of evidence. What if the employee's lawyers claim that the accident was part of a series of similar events? If archival footage of all similar events, across time, is available (and can be found with the company's search technology), it could decide the case.

Video also plays an important (and increasingly critical) role in health care. Nowadays, at major hospitals, all surgical procedures are recorded, for legal reasons. This results in huge volumes of video files that need to be cataloged, archived, and dispositioned.

Highway-patrol cars are (more often than not) videocamera-equipped. Every traffic ticket, every arrest, every roadside assist, is video-recorded. All of that material has to end up somewhere. It's best if it ends up in a repository, managed.

Law enforcement agencies routinely videotape suspect interrogations. Again, this creates enormous quantities of video information that needs to be cataloged and managed -- preferably in such a way that footage can be semantically searched later on, if needed. According to Herndon, VA-based MediaSolv Corporation (which sells video asset management systems specifically designed for police use -- a prime example of the increased verticalization we're seeing in DAM), 28% of U.S. states currently require the recording of "custodial interviews" (i.e., police interrogations), and fully half of all states have already passed evidence-preservation legislation. This essentially amounts to state-mandated use of DAM.

Take a look at your own organization. Do you see video management in your future? If the answer is "yes" (and it probably is), you'll want to consider availing yourself of our Digital & Media Asset Management Research, where we rate each of the 20+ vendors we evaluate on their video-management capabilities. We can help you get a handle on your media management needs -- even if you're still deciding what they are.

Here's an example that's been tried around the world with varying success.  Say you're running the web presence of some medical organization. As such, you have information on how to deal with various diseases, both on a general level (hygiene and disease prevention) and very specific (what to do when a flu breaks out).

Suddenly there is an outbreak of a new disease; let's say the elephant flu. You could prepare a news bulletin, which would automatically appear on information portals for medical professionals, consumers, etc. -- all the sites targeted to specific audiences for which this news might be of interest.

Better still, since you've already built up a large repository of information, it would be easy to launch an elephant flu theme site: just define the kind of content you'd want in there, and hey presto, with one click you've got an entire site with all the information (http://www.allaboutelephantflu.org). Content specifically on the elephant flu, but also the more generic topics on how to deal with a disease or whom to contact.

You can imagine why this is a compelling concept. Which is probably why I've seen attempts in many different areas, ranging from media companies, governments, product marketing companies, and insurance companies.

But, I said, elusive dream. Many have tried (I count myself among them), and many have failed (unfortunately, I can't really discount myself entirely from that group, either). That's because there are three major problems when you've actually implemented the infrastructure to do it. Since this is only a blog post, I'll pick the most obvious one for now.

The content needs metadata for this to work. Many will tell you that "people won't tag." No, seriously, they won't tag content with the right labels, add the right metadata, or correctly categorize, "even if threatened with being fired." And even if they do tag, it will be haphazard and inconsistent.

This is a very real problem. But at the same time it's complete nonsense. Because if this were the case, why would people meticulously tag and file their holiday snapshots on Flickr and Facebook? Somehow, in their spare time, they do identify the people in a picture, add keywords to a shot, give it a meaningful title, and actually describe it. Without having to be threatened with being fired, or even having to be beaten with a stick.

Partly this is because they get the feedback that makes it worth their while to do so. If you identify your friends in a picture on Facebook, they (and then their friends) will immediately find it and start commenting, which creates a positive feedback loop to tag some more. More importantly though, it's really easy.

If you get back to work the next day, and have to laboriously click ten times, scroll, add, categorize, while thinking what the right category within the taxonomy would be, it all feels like an insubordinate amount of trouble to go through. In most WCM systems and implementations (and dare I say it -- most ECM implementations are much, much worse) it's just too much trouble.

Fortunately, I'm beginning to see some change. There are now quite a few ways in which CMSs can make it easier on your editors to identify the content they're producing:

• Using a "free-for-all" folksonomy, where you can just quickly type in a few keywords. The problem of course, is that the tags will often be wildly inconsistent and ambiguous. Check a tag cloud near you for tags like "New York," "NY," "newyork," and of course, the typo that got away, "new yok." This can be made easier by type-ahead auto-completion of tags. Some systems will start listing suggestions as you type, and helpfully, with some, what you type doesn't have to be the beginning of the tag ("york" will also suggest "New York.") The auto-complete effectively normalizes the tags (i.e., at least all of them will be "New York.") It may still be ambiguous and inconsistent, but at least for many purposes, it'll be workable.
• Using suggestions. Usually with the help of an embedded search engine such as Lucene, the system comes up with tags and related links for your content (based on similar content it finds.) At first, this will need quite a bit of training, but the great thing is that the more content is accurately identified, the better the system gets. The suggestions can be used to completely automate the process, but since you'll still have the original author at hand in the editing screen, you can take advantage of this and ask them to validate the suggestions as they are saving the content. That's a lot easier than having to think them up themselves.

It's still more common to see any folksonomy functionality smoothly integrated into Social Software, and auto-categorization or auto-classification is an area where Search & Information Access systems are usually way ahead. But a few web CMSs are making headway into this territory, as well. For GOSS, which has mostly customers in the UK government, its ability to suggest related content, and categorize it in the IPSV (the Integrated Public Sector Vocabulary) is a unique selling point, and the company has been honing this for the past few years. Hippo (who, incidentally, recently won a large Dutch government account, so there may be a pattern there) is working on releasing similar functionality this fall.

There are others with such capabilities; but at the same time, many are lagging. A system that hides keywords and categories on the fourth tab, three items down under "metadata," and then makes users jump hoops to enter the information isn't likely to help. And some products are so thin on metadata, no amount of customization is ever going to make it work for your users. Carefully check before you buy.

But the good news is that if you share this dream of at least partly automating a metadata- driven architecture, the metadata part of that dream can be realized. Of course, that means there are at least two other major hurdles to take -- but that's enough for now.  I'll return to this topic again...

Consider the following:

• Despite the labyrinthine complexity of the US healthcare system - somehow it "works" and has done for years, so why change?
• The healthcare system in the US consists of two equally complex meta processes - insurance/billing & patient care - so where to start?
• Standards and agreed procedures are not uniform across the United States: Federal, State, City and Local entities all have to have a say
• The healthcare system is so highly fragmented, that there is in fact no "system". Instead there is a loose amalgam of public and private entities all with vested interests, and interests that often conflict
• Patients have the temerity at times to live to beyond 90, change state residency, insurance companies, employers and sometimes go without coverage at all

I am only scratching the surface of issues above.... though the ECM sector is getting very excited indeed about billions of dollars coming available from Federal government to help with EHR (Electronic Health Records). The lessons from the UK's National Health Service suggest that though lots of money will undoubtably be spent, not that much will change.

For sure there will be some 'state of the art' hospitals using joined up and impressive ECM systems, just as there will be adventurous local clinics that will sport staff with tablet devices. But fundamentally paper records will be a mainstay of the US health system for many years to come, and will likely run alongside electronic records for decades.

My advice to the buyers within healthcare is to cautious of the snake oil salesman. Those who come to you with grand visions of unified information systems across regions should be treated with great caution. Those who offer transitional routes via the use of imaging and scanning systems, and who support and will work with you on simplified standards and processes, should be given a little more ear time.

Despite the Obama administration's efforts to move EHR along, I would wager that the targets being set will be missed time and again. $20 billion sounds like a lot of money, because it is a lot of money. But it will not fundamentally change the current situation, though it is enough money for a start. Like all ECM projects that go wrong, the focus is currently on EHR and the technology that supports EHR, when the focus should in fact be on the processes and process transformations that will need to be managed and changed across a wide array of conflicting and disparate working units.

EHR, just like ECM is a process of business change utilizing some underlying and supporting technologies. Any project that sees it the other way around is doomed from the start. I fear that is sadly what has already happened here.

Within that service we cover XML authoring tools: the software for creating and editing highly complex documentation, be it DITA structured technical documentation, complex translation work, or product information designed for multiple re-use. But our research goes beyond the editing and authoring options at the front end, and evaluates all the leading CCM (Component Content Management) tools that have been designed to manage the publishing and workflow of these complex XML document components throughout their lifecycle.

On other occasions we separate technologies into different categories, where others may have bundled them together. An obvious case here is our Web Content Management research and our Enterprise Content Management Suites research. Just in the past fortnight I have been asked by more than one person why we have two subscription services for the same technology. Well the simple answer is that it is not the same technology: the technology to manage outward (web) facing content and sites, versus the technology to manage inward-facing (documents and files) content and filing systems is quite different. But nevertheless it is a very fair question to ask, because to someone outside the industry (and even some insiders) that distinction is not always obvious.

Confusion can come about due to vendor branding strategies or a plethora of nebulous acronyms used within the industry. For example Interwoven (recently acquired by Autonomy) did a good job of branding their WorkSite (document & files) offering quite separately from their TeamSite (web content) offering. Other vendors such as EMC Documentum or Open Text use a single brand moniker (such as "digital media") to cover all their content-focused software offerings regardless of each component's specific purpose, while in other instances product names can even be synoymous with the firm's brand, as with ADAM and Sitecore.

Then there is the issue of acronyms. We use the term ECM as it is the most commonly used term for document-based technologies, however in different regions and industry segments ECM is referred to as EDMS, EDM, EDRM, Document Control, IM, IDM even KM. As an advisory firm we have to call our research and services something, and we are faced with picking and choosing amongst a myriad of terms, typically choosing the most well-known term currently in use.

Our job at CMS Watch is to make sense of the complexity of the technology offerings out there for you, the buyer - and that can be a challenge at the best of times. We believe it's safe to say that we have more research available to our subscribers on content technologies than all of our competitors combined. But volume, depth and breadth is not all that separates us from our competitors, we are in addition avowedly a firm that follows the market rather than "makes" it. Thus, we don't try and come up with new acronyms, terminology or market segments, we don't cheerlead for the industry or get involved in market sizing, we stick to the basics of evaluating current release products side by side and evaluating the importance of current trends.

So here is my challenge to you. We evaluate over 200 products and sub-divide those into 10 subscription services, which in turn are sub-divided into about 5-8 sub-categories each. We try to use common terminology so that you the buyer can relate to and locate the research that meets your needs, but that does not always work as well as it could. So if you find the product evaluation(s) you need, but locate them in a place - or grouped in a way you find 'surprising' or 'interesting', let us know. Tell us how you and your colleagues perceive these technologies, what you call them, how you would categorize them differently, how you explain them to your internal clients or project teams. Truly we would love to hear from you as this industry remains a dynamic and very moveable feast at times....