I'm not referring here to Yammer's now famous "freemium" approach of getting your colleagues hooked on a free version and then upselling a more enterprisey edition. The real problem is this: employees signing up by virtue of having an enterprise email address believe that the free version of Yammer is an enterprise-sanctioned and perhaps even enterprise-managed solution.

This is of course a complete delusion, albeit a totally understandable one.

As subscribers to our Enterprise Collaboration and Social Software research know, Yammer's free version is a legal conundrum for the typical large enterprise. In discussions with senior leaders from among our enterprise subscribers, a recurrent theme has emerged about the constant efforts they must take to educate colleagues about the platform.

Savvier enterprises whose employees have taken to the free version emphasize that their public social media policies apply to Yammer, rather than their internal collaboration policies. That means, for example, that you shouldn't share sensitive data or documents via that channel. Busy employees may not always ingest that message. And at a time when many industry gurus don't recognize the difference between external social media and internal social networking, you can understand why some of your employees may not grasp the nuances either.

At the same time, we don't counsel our subscribers to kill off Yammer eruptions either. Enterprise social networking is hard enough to nurture without suffocating it in the crib. (On the other hand many Yammer experiments end up suffocating on their own for some very specific reasons, but that's another story...)

You need to understand the implications of the free version, and do what you can and should do to mitigate risks until you come up with a sanctioned, supported alternative. That alternative might well include licensing the paid version of Yammer, though in the event, you'll want to review your agreement very closely, since like many cloud providers, Yammer's enterprise edition SLA isn't so hot either.

Other collaboration and social computing providers are watching the Yammer model closely. The more free services the get targeted at enterprise employees, the more you need to educate about acceptable use. Better yet, devise a roadmap to get out in front of those needs. Let us know if we can help.

It's a question that many of our advisory customers have been asking us, so we decided to answer it in briefing format.

Cloud-based file-sharing services have obtained some traction, even among larger organizations, because of ease of provisioning and mobile access, as well as for exchanging files with external parties. At least five major providers now target enterprise customers for hosted file sharing: Box, Dropbox, Huddle, Oxygen, and ShareFile.

Just today in fact when talking with a major industrial manufacturer, the excitement and possibilities of working with such suppliers was tempered with concerns over their readiness and maturity to deal with real enterprise needs. To quote this prospective customer, "All these vendors tell us they work with big enterprises, but when we speak to them it feels like it's the first time a real enterprise has asked real enterprise questions of them..."

Certainly cloud providers are here to stay, and if employed in the right circumstances can bring real benefits, but it's still early days as our research paper details.

I wonder if this is really the case.

When a vendor OEMs a product, this typically means that they are embedding a subsystem or specialized technology from a 3rd-party supplier. The key thing here is that embedding someone else's technology can be better, cheaper, and easier than building that same functionality yourself. So for example many ECM vendors embed Oracle's (formerly Stellent) INSO document viewer in their offerings.  Another ECM example would be the OEM of Crystal Reports, to provide reporting capabilities drawing from the base audit trail in an the ECM platform.   

The questions raised in Autonomy's case are twofold: first, how do you embed IDOL at all, and secondly how many people are actually doing so?

Leslie Owens over at Forrester recently wrote a piece on this issue, and described IDOL in terms of a vacuum cleaner. It's an excellent analogy. You see, IDOL does not embed itself in other vendor's products as much as it sits in the corner and sucks data out for processing.  It is essentially a separate black box, and as such is one that could be easily replaced by any other search engine. 

So, far from other vendors becoming dependent on IDOL, it is in fact loosely coupled and easily swapped out for alternatives.  For example, Oracle and Sybase have already replaced Autonomy search technologies with alternatives. Many others have followed suit, particularly since there are now viable and considerably cheaper open source options in the various forms of Lucene/Solr.  And finally, it's important to note that where Autonomy is present in 3rd-party software, it is more typically the old (and very basic) Verity engine, not IDOL.

The argument goes that IDOL may well be expensive and complicated to use, but at the high end of the market it is the best, or as other analyst firms like to say it a "market leader." I can't help but wonder whether this is a victory of marketing over substance. Sure there may be some organizations (the much vaunted law enforcement / security / intelligence buyers) who fully leverage IDOL, but what about the much greater mass of regular enterprises and public organizations?  In our experience most of them are struggling with basic search, with both Microsoft  options and the Google Search Appliance prevalent throughout pockets of their organizations.

For buyers, the HP acquisition of Autonomy could prove enlightening. It will doubtless prompt a shakeout and insight into the true nature of a very secretive company, but more importantly it is reawakening.  It reminds us that there are many enterprise search options available, that there is no defacto standard, and that search in general still typically under-delivers.

If nothing else the move by HP should be taken as a time for current IDOL buyers to consider their options, and to recognize that they in fact do have many options available ranging from Oracle and IBM to specialists such as Recommind, Exalead, Coveo, and ISYS, some of which are much cheaper and easier to use than IDOL. Hopefully, the marketplace will create still more options in the coming years, as most customers would testify that they're not satisfied with their incumbent systems...

EntropySoft, a content integration and migration vendor, has released a new connector for Twitter. EntropySoft already has a set of connectors that get OEM'ed into various packaged content and document management systems, many of which we cover in our evaluation reports.

This new connector enables enterprises to access content (tweets and other information) stored in Twitter. EntropySoft says the primary objective here is to archive those tweets in a corporate archiving system. Once the Twitter content is in one of your own repositories, you can do pretty much anything that the target system allows you to do -- such as declaring tweets as records and applying retention policies on them.

Many organizations have been experimenting with various alternatives for managing their employees' tweets. Some enterprises, for example, employ their Web Content Management system or a Portal-type application as a tweeting interface, so that they can subsequently manage those tweets as content in their larger application. By using a connector-based approach, in contrast, employees can tweet using any of their favorite tools but those tweets can still be brought back into an internal system for subsequent management.

There are some challenges that you'd need to address with the latter model, though. Besides the practicality of considering tweets and more generally social content as records, there are also issues of ownership and legality related to extraction of tweets from Twitter and archiving these in your own systems.

In any case, the ability to access social content via such tools has many uses. Many of the products we cover in our various reports -- such as Document Management and ECM tools like Alfresco and EMC Documentum as well as search engines like Endeca and Exalead -- turn to with EntropySoft to supply connectors within their products. We don't know yet if they plan to use this new Twitter connector too but if and when they start using it, it could become marginally easier to search, index, and archive social content from within your existing systems.

Let's look at what's driving this. The last decade has seen continuous improvements to search technology. At least in theory, today's search engines can crawl more complex repositories, can handle many more documents, and run faster and more efficiently. All you need -- the story goes -- is the ability to convert search results to an RSS feed, build some kind of a simple widget-based environment that displays those feeds, and you are up and running with a functional portal application.

But don't stop there.  The same logic can also get extended to other web technologies. Search engines can act as replacement to traditional integration mechanisms since they can crawl multiple repositories. They can also handle more sophisticated applications such as e-discovery.  So, why invest in portal or portal-like technologies at all?

My advice: don't buy into this hype.

Before you decide to build a web or portal application that's driven by your search engine's index, first consider if that's the best approach to get at all types of information you want to display. You'll quickly find some inherent limitations.

For example, most search engines make certain assumptions and approximations in the way versions, security, and duplicates get handled. The security profile on the document might have changed after your search engine indexed it. Sure, you can take advantage of your search engine's "late-binding" security capabilities, but that only adds more overhead to processing time.

Moreover, in those cases where your application knows exactly what to get from the repository, using search indexes and queries adds even more overhead. "Showing 1 of 1" is hardly the best way to display the local temperature inside a weather gadget.

The point then is that a search-based query is not the most optimum way to access your back-end data for the typical diversity of enterprise portal scenarios. Before making a major platform decision, be sure to consider alternatives that could prove faster more accurate.

This then got me to thinking about how a search result is only as good as the content getting searched, and how people have no problem throwing huge sums of time and money at search solutions from IBM, Oracle, Vivisimo, Coveo, and the rest, but balk at spending any time or money cleaning up their content disaster zones.

On the radio and television I have recently been bombarded with adverts for really expensive diet pills and potions, and as a man of a certain age and paunch I find myself drawn to them with disturbing regularity. For a year or so back I was suffering from chronic tiredness, this worried me as something was clearly very wrong. So, I spent time and money on going through a series of medical checks to find the cause. The result was that there was nothing wrong with me that a bit of exercise and a better (lesser) diet wouldn't fix (my doctor was less diplomatic than that).  So the result of all those tests was that I was very relieved to hear that there was nothing wrong with me, and that I could carry on merrily with my regular lifestyle.  Yet, now that I am starting to look like the cocktail stick that swallowed the cherry, those diet pills are starting to look good, even though I know they probably won't really work. At least they won't work without a change in my diet and exercise regimen.

Search tools are much the same. They are the result of many years and millions in R&D, and hence have become very clever indeed.  But if as organizations we were to diet a little (cut down on the junk) and exercise a little (apply some lifecycle management to content so that it doesn't just accumulate), then search tools could work much, much better. If we were to get a bit more serious and join a gym (start to tag content in a more structured manner) then the search tool can deliver outstanding results. But that all takes effort, and we don't much like effort, and the magic solution is so much more appealing, as you are promised all the benefits with none of the effort.

Just as in life a little self discipline with content can go a very long way.  And after I finish this sandwich and can of cola I am thinking of trying some for myself.

• Key take-aways
• Introduction
• Key Factors that Determine Whether SharePoint Is Right for You
• Conclusion

Records Management is becoming an increasingly hot topic for SharePoint as the volume of critical business information residing in SharePoint sites and collections grows, and I hope our advisory paper will add constructively to that discussion. However, it's not a topic that gets discussed much outside of hardcore RM circles, though inside those RM circles it's a raging debate.  Questions have been raised about SharePoint's basic approach to RM, including concerns over what some experts see as fundamental flaws in the design. (We assess those flaws in the advisory.)

Of course, like everything else in SharePoint, there are third-party vendors that can sell you add on modules for RM. But the majority of SharePoint implementers, consultants, and integrators understand little of RM or of its importance. Hence it's not too surprising that flaws can potentially slip through the cracks, and that SharePoint continues to fall short for RM. 

To quote:

There’s a bigger picture here: information management is becoming gamified, consumerized, socialized, mobilized, cloudified, and caramelized. Enterprises should turn their attention away from core information assets and focus all digital energies on linking up with the next Facebook, whatever that may be.

To get the real story on what's going on in across the different content technology marketplaces, download some research samples.

What actually transpired was a bit more interesting.

We quickly agreed that traditional Business Intelligence (BI) has little real value in the world of Enterprise Content Management (ECM), beyond introducing the concept of dashboards for monitoring some activity.  The reason being that BI delivers true value when it searches rows and columns across multiple databases (or Marts or Warehouses) of structured data.  ECM systems are by definition filled full of unstructured files of information -- a messy, complex, and generally unsuitable dataset for BI tools.

That said there are exceptions. For example, if you make extensive use of Business Process Management (BPM), with sufficiently high throughput, you will build a substantial volume of structured data. This data will likely relate to when content moved between steps, what triggers were fired, times taken to execute tasks, and so forth. This is important data, that can be correlated, exposed in a dashboard, and provide rich monitoring information.

Beyond this though we move into a sort of limbo world that exists somewhere between real BI and Enterprise Search.  This world gets described loosely as that of "analytics."  It's an inexact term to be sure, but it makes some sense. Think of it this way....we use search to find a thing that is difficult to locate, while we use analytics to understand patterns across and within multiple things. Not rocket science (though the technology itself can be), but nevertheless profoundly different scenarios.

Search is the scenario that most people are most familiar with, and maybe most disillusioned about. Though I don't really buy into the belief that Search is a mature technology, the fact remains that enterprise search tools are typically asked to work against a corpus of junk, and no matter how smart the search tools get, the equation junk in = junk out remains a constant. 

Analytics though comes from quite a different angle. The goal here is not to find a specific thing, rather it is to analyze trends, relationships, and patterns, and deliver back a parsed (translated) message.  Consider for example analyzing five years of medical billing records, insurance claims forms, or for that matter police arrest reports.  I don't know specifically what questions one might want to ask, but it doesn't take a huge stretch of the imagination to recognize that there is potentially a wealth of knowledge hidden within these particular types of information mountains.

Analytics tools have been with us in one form or another for many years -- but they have remained highly specialized. And just like BI technology, content analytics tools are typically expensive, complex to use, and appeal to a very niche audience. These tools will probably always remain nichey, yet many traditional names in the search world from Autonomy through IBM to Vivisimo are re-orienting their efforts in the analytics direction. In doing so they're edging  away from the more traditional search market that's seemingly (but not really) dominated by Microsoft & Google offerings.

Hopefully this evolution can bring richer and more usable analytic functionality to a broader cross-section of the enterprise, by digging more deeply and widely into enterprise information stacks.  It's an ongoing shift that we plan to watch very carefully over the coming year, because it's one that not only vendors are pursuing, but buyers too. There is a very measurable and definable need in enterprises today to "know what the enterprise knows," and it's a need that currently goes unmet.

In 2008 HP bought TRIM, a well established RM system, by acquiring Tower Technology. When combined with HP's own archiving technology, TRIM looked set to offer a compelling and cost effective system to the market. Indeed the sheer scale of HP as an organization, along with its expertise in Archiving and Storage, suggested that they could become a major presence in the ECM (Enterprise Content Management) sector, either leading, or at least competing alongside the likes of EMC and IBM. But as of today, despite the acquisition of TRIM, HP has yet to make a mark.

I have recently been updating the HP TRIM product evaluation for our ECM vendor evaluations, and in the process of that work can at least acknowledge that HP seems to have finally figured out what they want to do with this technology.  They want it to be the cornerstone of what they would describe as a "Governance Engine," a unified a approach to centralizing the work of retention, disposition, and archiving of unstructured (think documents) and structured (think databases) information across an enterprise's many systems.

This lies in contrast to their earlier plans to deliver (probably via further acquisitions) a broad-based ECM platform. The apparent impetus for this new HP strategy has been viral SharePoint deployments, and as TRIM was always a Microsoft-centric product set, it has not been too hard for HP and redirect TRIM toward that market.

As we have often stated here, RM is an important but underfunded and under-acknowledged enterprise need. How well HP will do I cannot say, but what gives me pause is that their present strategy seems very reminiscent of fellow IT giant CA.  CA bought a collection of RM and archiving technologies, designated them as a central part of their enterprise governance strategy, and after failing to make any progress, they sold the lot to Autonomy.  CA's strategy was a total failure, and the best one can say of it is that they got a bit of money back at the end of the experiment.

Enterprises need RM solutions that are user friendly, affordable, and efficient. They also need a broad community that more effectively champions RM and better articulates its real value. Hence I'll watch HP's progress with considerable interest, as the role of chief RM cheerleader in this market is currently vacant, and is in need of being filled. With luck and a good wind, other vendors along with HP will also rise to the RM challenge and help to redefine its role within today's enterprise. I am not so much interested in whether HP can sell its products, but as to how they can contribute to a much needed dialogue, and be a part of much needed change.

Enterprises frequently inquire about what format to employ when archiving electronic documents. There are two competing formats for archiving electronic documents: TIFF (Tagged Image File Format) and PDF/A (Portable Document Format - Archival). The TIFF format is the most widely used globally and pre-dates the PDF/A format. However, the PDF/A format has some very distinct advantages and therefore will continue to grow; it could potentially become more popular than the TIFF format in the future.

ECM research stream subscribers can download the briefing here.

Yet many organizations seem to be under the impression that if they buy eDiscovery software something magical will happen: That regardless of the complete and utter chaos that constitutes information management within their organizations, at the push of a button or two the eDiscovery software will find the required information, and then present it in a neat and secure format, thereby meeting the eDiscovery request.  Even in the world of Harry Potter, there is no magic that powerful.

A good example of just how far the hype is from reality was nicely illustrated by Goldman Sachs this week. In responding to an FCIC (Financial Crisis Inquiry Commission) request for information, Goldman provided 5 Terabytes of data, the equivalent of 2.5 billion pages of documents. To quote the FCIC Chairman, "We did not ask them to pull up a dump truck to our offices and dump a load of rubbish."  Now I have no idea whether this was simply a deliberate attempt to obfuscate and hinder the crisis probe -- as the FCIC seems to believe -- but I doubt that it's the whole of the story even if it is a part of it.

eDiscovery requests fall into two main categories: the easy and the impossible.  The easy ones relate to specific individual and a specific transaction. By freezing an individual's email account and network folders almost everything you have been asked to look for is found.  The impossible category consists of almost everything else. In these cases lawyers sort things out in back rooms, usually come to a compromise, and when they can't, either fight the request through the courts or simply settle.  When a government probe like this comes along though you have no real option other than to comply, or at least attempt to as best you can.

To meet eDiscovery requests with ease, you need solid information governance across the whole organization to be in place, and you need a lot more than a search engine on steroids.  When firms have billions of documents or even just millions, knowing exactly which documents are needed, and how they related to each other in a chain of events resulting in an eDiscovery request, will require a lot of manual trawling, lawyers at high hourly rates, and pragmatism for the foreseeable future. 

The only good news is that many firms are now recognizing that no magical software will ever be able to fix this situation. You need good information management practices, and a plethora of content technologies (ECM, Search, Archiving etc) all working in tandem. 

When I advise our clients one to one, or simply chat with people for research purposes there is almost always a common thread.  People tell me that they use SharePoint, or Documentum or Open Text or whatever, for managing their corporate documents, but when I push a little further almost without exception, a much larger number of documents within the organization sit in folders on a Network Drive. The actual document management systems account for only a small fraction of the documents within any organization.

Network drives have never been good places to manage documents, nor will they ever be -- yet that is where most of them sit.  It's depressing really, but nevertheless it is reality. Network drives (or shared drives as they are often referred to) give the illusion that you are doing something constructive to store and manage documents, but really they are just dumping grounds, expensive trash cans.  The only difference in practical terms is that trash cans actually get emptied at some point. Shared drive trash just grows until they become overwhelming mountains. 

But what to do?  A starting point is for organizations big and small to recognize they have a problem, that things are unmanageable and boldly declare "We are document hoarders!" You may even want to do a fearless inventory of your content, and humbly address your shortcomings.  Implement a retention and disposition policy, and learn to let go of documents that no longer have value.

Shared drives are addictive and a hard habit to kick; get some outside help from others that have experience in kicking the same habit.  And if you ever do sort out your shared drives, consider yourself a success, and then turn your attention to the documents sitting in e-mail folders....

The last few weeks has seen a drama, albeit not a Bollywood one, unfold. It all started when the IPL chairman cum commissioner Mr. Modi fired the first salvo on Twitter about a new IPL franchise which was being "mentored" by a ruling party MP. The MP, Mr.Tharoor, quickly tweeted back to start a scandal now being called by various names such as IPL-Gate, Modi-Gate, and Tharoor-Gate. An alert media was quick to latch onto it and generate breaking news, eventually revealing much more than the shenanigans of Modi and Tharoor.

We next saw the two protagonists leaving their coveted positions: Tharoor resigning on an emotional note in the Parliament and Modi being suspended -- ironically via e-mail -- by IPL's governing council.

This is certainly good news for Twitter: people in India who have no clue of the web now want to tweet. But what has this got to do with Content Management?

Well, the IPL scandal that's rocking the Indian Parliament and cricketing world revolves essentially around the issue of compliance. Let us examine how many of the problems could have been avoided had the IPL followed good compliance practices.

And what constitutes good compliance practices? We offer a short online course that lays them out. Let's review some important compliance considerations in the context of this scandal:

1. Executive sponsorship: IPL is a sub-committee of Board of Control for Cricket in India (BCCI). On the face of it, IPL had it's own CEO, Chairman, as well as a Governing Council (GC) of 12 members. So it appears there was a suitable level of top management support. However, it now turns out that the CEO was just a figurehead with most of the decisions being taken by a proxy. The BCCI and the GC assumed that the CEO would do his job and were caught wrong-footed.
2. Well defined governance procedures: Okay, so they created a governing council (GC). The GC boasts eminent members, including well respected cricketers, politicians, and administrators -- but all without any accountability. And just like the CEO, the GC remained a silent spectator to all the wrongdoing, never consulted on important matters. In fact, the GC members remained unaware of major transactions involving billions of dollars, since all of these were handled by just one single person. No procedures existed to ensure the placement of suitable checks and balances.
3. Identified (and prioritized) compliance requirements: What's that? Since everyone involved was making big money, no one in their wildest dreams thought they'd need to comply with legal and ethical rules. The current scandal have revealed (among other things): inconsistencies in shareholding patterns, bid and contract documents gone missing, and some suspect funding sources. No processes were defined to approve contract documents or to keep an audit trail of transactions.
4. Ability to manage change as a result new business processes: Processes were non-existent and every activity was handled in an ad-hoc manner, depending on whims and fancies of individuals.
5. Systems to monitor, audit, and enforce compliance: There were no systems and more importantly no desire to enforce compliance. Everyone was in for the love of money, not for the love of the game
6. Right tools – such as those for managing content and records – to optimize your compliance efforts: The parent body, BCCI is the richest sports body in India and also the richest cricket board in the world. One would assume that a cash-rich cricket board would invest in technology that would help them in managing records and documents, enabling processes for tracking bids and contracts and other dealings. Yet, the BCCI doesn't even have a decent web presence. Use of technology could have facilitated transparency, something they evidently wanted to avoid.

Will applying proper Content Management practices and following the steps above in and of themselves enable organizations to avoid tax evasion, illegal betting, bribes, suspect shareholding patterns and so forth? Of course not. If you have people of questionable integrity and ethics running an organization, no amount of management practices will help.

Nevertheless, technology can help uncover and document misdeeds. If IPL had followed basic compliance principles, things would not have gotten so far out of hand. And we would have enjoyed our greatest passions - cricket and bollywood -- even more.

(Special thanks to Siddharth Gongala for supplying background info. Sid is a Sports Producer for an online channel.)

Obama, of course, is on YouTube; and YouTube started an experiment with automatic closed captioning of videos last year, so some of Obama's speeches have automatically transcribed subtitles.

Sarkozy, on the other hand, resides online on the much more presidential Elysée site. Even if this weren't a perfect case in point for "not invented here," YouTube doesn't understand anything but English, so that wasn't an option. Instead, the French search vendor Exalead provided text-to-speech for sous titrage, and made all the videos keyword searchable. The results point to the exact frame in the video, too.

Of course, the idea itself isn't new. A few vendors in our Search & Information Access research allow you to do the same. If you'd want to search loads of untranscribed recordings -- say, council meetings, or millions of customer recordings from a call center -- you could ask Autonomy or Coveo as well. (Autonomy's technology also powers the Blinkx video search, and Coveo has a module you can also use to enable SharePoint to index audio and video.)

However, if there's ever a technology you'd have to test on your own corpus first, before purchasing, media search is it. The success rate varies wildly, and the most important factor isn't even the technology -- it's the quality of the recording. Exalead's Voxalead has been trained to understand French political speeches, and the Sarkozy recordings are impeccable, so there are relatively few mistakes. The Voxalead News demo is a lot more ambitious (indexing newscasts in five languages), but doesn't fare quite as well, even though the sound quality is usually pretty good. Do this over the phone and speak in vernacular, and results can range from surprisingly good to useless -- but funny. (Note that while YouTube and Google Voice both use the same technology, it's not available as a module for the Google Appliance.)

So though I can think of quite a few uses for the technology, in most cases, don't hold your breath yet. The demos can be amazing, and Exalead proves it's a lot better at keeping tabs on Sarkozy than Google is with Obama. But before you commit, test, test again, and if you think you're done, do more testing. Truly useful and successful implementations are still few and far between.

People outside of highly regulated environments typically assume that records and retention management in sectors like healthcare, financial services, and energy is state of the art.  This is far from the case.

Most often Records Management is underfunded,  inadequately resourced, and unloved.  It is no cliche to say it is also often run from underground, literally in basements out of sight of those who need to do "real work." 

In places where records are actively managed, they are usually managed well, with detailed and well-maintained file plans and retention schedules.  But many organizations only manage a fraction of the records they should be.  For example very few records management departments include e-mail as part of their remit, yet e-mail is where all the "stuff" happens. 

But maybe, just maybe, the tide is starting to turn.

Here at the Real Story Group we currently support a number of large advisory customers who are looking at records management strategically across highly complex working environments.  We have others who are looking at the whole issue of information management and ECM more strategically than in the past, and have begun to include RM as a component or a recognized future element of their work. 

That might not sound like much, but just a year or so ago it seemed like nobody cared.  Things like e-government initiatives, healthcare reform, demand or need for more self-service applications, increased regulations, and so forth seldom have the immediate impact people expect. And in a world where decisions can get driven by today's opinion polls, it's easy to lose sight of the fact that many of the biggest and most profound changes we encounter in our society occur at a far slower pace.

The most overused phrase in my personal lexicon is, "time will tell." But I fervently hope that time will tell us that awareness of the importance of RM and archiving has slowly risen to the fore.

At the core, both enterprise and federated search are about accessing, indexing, and querying diverse and dispersed repositories. Enterprise search is very specifically about searching within the enterprise, which may include an externally-facing web site or extranet(s). 

It's often the case that an enterprise search tool can come across repositories that already have their own search engine that's created its own index. Assume for a moment that you elect  not to have your enterprise search engine re-index all that content itself.  Yet you want to expose items from those repositories.  In this scenario, when the initial search engine is trying to return results from those repositories that are already indexed, it can do one of two things:

1. Directly query those indexes created by other search engines in real-time, or
2. Trigger those search engines to provide results, and in turn aggregate what it collects into a single, enterprise-wide results set

That's federation: leveraging the indexes and potentially the query results of other search services within your enterprise.

However, I've heard some vendors use the terms enterprise search and federated search interchangeably -- which creates confusion.  A scenario where an enterprise search tool crawls other repositories itself and builds an index based on that content in all its unstructured glory is classic enterprise search, even if your vendor calls it "federated search." 

Obviously federated search sounds quite attractive, especially in an environment where enterprises have licensed multiple search products or employ various information management systems that embed their own search facilities.

In practice, however, comprehensive federated search is quite difficult. Even more than with “regular” enterprise search, security becomes an important issue, particularly when multiple indexes or engines follow different security models. Also, in scenarios 1) and 2) above, it can become extremely hard to de-dupe, merge, and provide comprehensive relevancy rankings in a reasonable timeframe for the searcher. A bottleneck in one repository can gum up the overall results.

While hosting a search seminar with Martin White of Intranet Focus in London a couple of years ago, he drew an excellent representation of these federated search challenges, and I have since transformed it to PowerPoint and use it often in my presentations about search technologies. 

Here at Real Story Group, we've added more detail to our search vendor evaluations regarding how vendors handle this highly complicated scenario. Despite the challenges of federated search, there is promise in these technologies, especially those approaches that rely on other native-search engines to provide the core results, and serve as basic aggregators.

It is inconceivable that enterprise-focused vendors, be they niche specialists like Hyland, Open Text or Autonomy, or household names like Microsoft, IBM or Oracle, would ever contemplate the reckless move that Google undertook in deliberately exposing customers' private information to all and sundry with Buzz. Even if they did contemplate such a move, it would never have happened, as these firms not only have too many checks and balances to ensure it could not. But additionally they have a solid understanding that enterprise customers would simply not tolerate, or ever forgive such a move. It would be commercial suicide.

Google is a consumer company that makes a huge amount of money from selling online advertisements. The business model is predicated by the concept that popular equals good. The more hits the better; the more people accessing information the better.  Basically Google is structured to support and develop "cool" functionality that will drive ever more online activity. But the world of the enterprise just isn't like that. In fact most of the time it is the exact opposite. It is a world that focuses on security, control, process, and accuracy.

Google's enterprise business income represents but a tiny fraction of overall revenues -- by some estimates less than 1%. There is simply no way that such an insignificant group will ever seriously challenge, let alone change, Google's popularity-driven development culture.

The ease of exposing a customer's private contact details (GMail has 175 million of them) quite naturally raises questions about the broader suitability of cloud computing for enterprise needs in and of itself. But I think to focus on that at the moment would be a mistake.  More important right now is to focus on Google itself, a culture that has undervalued its customers' confidentiality and privacy. By default this makes Google unsuitable to meet the needs of the enterprise.

The Mimecast-based hosted solution will apparently continue to operate alongside the onsite Mimosa based system, but it will be interesting to see how this eventually works out.

For buyers of Mimosa's system this is likely pretty good news, as Iron Mountain brings more finances and expertise to support and develop Mimosa than its founders could.  For users of the Iron Mountain Mimecast-based service, you will likely want to get firm answers from your account rep about whether and how it will remain supported in the longer term.

There are at least two different ways that software can specialize: Functionally and Vertically. Functional specialization -- sometimes referred to as "horizontal" specialization -- entails targeting a particular, well-known business use-case, such as e-discovery, that spans multiple different industries. Verticalization happens when a product gets tailored for a specific market "vertical" or industry segment, such as Manufacturing. (Verticalization typically implies functional specialization of some kind as well, but different industries beget different types of functional needs.)

In the opposite direction, some vendors try to create omnibus platforms, making a product or product family more broadly capable, and therefore theoretically applicable to many different verticals and business scenarios. This can be an effective strategy, but at some level is the opposite of specialization. An example of this in the WCM world is Ektron's Web CMS, which started out essentially as a rich-text editor and has broadened, functionally, in a manner that would make the designers of the Swiss Army knife proud. Many other WCM and ECM tools have followed a similar path.

The search marketplace has been quite different. Our latest Search & Information Access Research has found such specialization more the rule than the exception. In fact, it is a major -- maybe the major -- industry trend right now.

For example, when a vendor such as Recommind continues to tailor its products in ways that favor addressing enterprise e-discovery use-cases and the legal marketplace, that's an example of of both functional and vertical specialization.

Other examples of functional and vertical specialization in search are legion.

• Endeca continues to pursue categorization, clustering, and other techniques particularly relevant to the online-retail market (and the intelligence community, as well).
• Surfray is striving to differentiate itself as the "SharePoint search solution."
• Adobe Omniture's SiteSearch has concentrated heavily on an analytics-rich approach to search that appeals to marketers.
• Sinequa, with its emphasis on linguistic analysis, continues to tout its prowess as a multi-repository "knowledge access tool," applicable for KM use-cases and the law enforcement sector.
• Coveo and ISYS, meanwhile, continue to strengthen their capabilities in multi-client (palm/lap/desktop) enterprise search. Coveo, in addition, continues to beef up its BI capabilities; it now offers a search solution tailored for call-centers.

For more examples, consult our newly updated Search & Information Access Research.

Of course, the trend towards specialization search is not universal. Mega-vendor Autonomy offers a dizzying array of diverse search and access tools. Even then, Autonomy increasingly emphasizes "meaning-based computing." I'm not sure what that term means, but we've seen demos for functional use cases like e-commerce.

Overall, we view specialization as a positive trend. It means search vendors (and presumably their customers) understand that search is not really a one-solution-fits-all problem. Search is as much a knowledge-discovery problem as it is a problem of "looking things up." It's about finding things you didn't necessarily know existed. And the tactics for doing that are as varied as the information world itself. There is no one right way to find something, so it stands to reason there is no one general-purpose system that can do it all. Getting the right search solution in place means first and foremost understanding your needs. And that's something we can help you with -- through our research and consultation. Don't hesitate to call on us when the time comes.

The question that arose at the end of the day was, "what is it going to take for Digital Asset Management (and related technologies) to match the adoption of Enterprise Content Management and Web Content Management today?"

There are a variety of reasons for why DAM will finally become a priority. I'm afraid one such reason is legal action. Like many things in life, people don't take things seriously until they get in trouble. We have seen many cases where litigation has spawned action to manage e-mails, records, documents, and even web content. With the explosion of digital media within enterprises, the chances that pertinent evidence resides in an audio, video, or image file is greater and greater. If you were sued, would you be able to search through all of your digital assets to find relevant evidence?

Among a variety of functionalities, the need to respond to litigation will undoubtedly challenge a DAM system's capabilities with regards to metadata, search, and rights management. As readers of the Digital Asset Management Report know, digital rights management in particular remains a weak point among most of the products we cover. Litigation will likely cause buyers to demand better rights management in the near future. Unfortunately for some, their current systems are not going to suffice as presently implemented.

Even if you are managing digital assets in your enterprise, are you prepared to put a formal "hold" on the assets in the system? Does your e-discovery technology integrate with your DAM system? As students of our E-Discovery online education course know, e-discovery can be expensive, time-consuming, and painful when dealing with something as seemingly simple as e-mails. It is much more difficult when the smoking gun is embedded in some video.

It's time for enterprises to get serious about properly managing their digital assets just like they are (or should be) doing with other enterprise content.

More and more, IT teams are finding themselves entangled in legal webs as legal actions spawn electronic discovery efforts. E-discovery can quickly become a costly and time-intensive activity -- and can put immense strain on an enterprise's business as usual. So much so, that many enterprises choose to simply settle cases in order to avoid the costly, and often painful, process altogether. Adding to the pain of an e-discovery process is the confusing marketplace of technologies that are classified as "e-discovery" tools. These can range from everything from search engines to full-blown case management tools.

Today we added our newest course, "The Fundamentals of E-Discovery" to our ever-expanding curriculum of online courses. Our hope is that this course will give you the basics that you should know before beginning an e-discovery project and help you minimize your risk by taking a pro-active approach to e-discovery. Armed with these fundamentals, you will be able to help your team see how e-discovery technology can be an essential part of the solution, but not the solution alone.