Structured search (noun) was rooted firmly in the enterprise when publishers of print index resources (e.g. Chemical Abstracts, Index Medicus from the National Library of Medicine, GRA&I from the National Technical Information Service) became available on-line in the early 1970s. The Systems Development Corporation launched ORBIT developed by a team lead by Carlos Cuadra. Orbit was a command driven search tool accessible to professional searchers. In those days searchers were usually special librarians in corporations, large public libraries, government agencies and major universities. Using the ORBIT command language through a terminal connected by a phone line to remote large computers, librarians would type search commands to find data in specific structured fields. These remote computers held electronic versions of paper indices. Citations resulting from a query for specific chemical compounds, diseases, or government reports, would contain information needed to retrieve articles, patents or books from library shelves.
Corporations spent hundreds of thousands of dollars each year to access external specialized, and structured indices, and the journals, conference proceeding, patents and government documents to which the indices pointed. Hard copy (paper or microform) was the only practical way to read content. Computer screens were not accessible to most researchers and even if they had been, content could not be rendered on them in easily readable forms. Also, until computer storage technologies became cheap, indexing large amounts of text (full-text, or unstructured content) was not affordable.
Even with the advent of graphical interfaces, searching for non-specialists made only minor advances in the early-1980s when library systems offered index browsing to find citations. Library users still needed to read content in hard copy. It was only in the late 1980s and early 90s that full-text content began to be searchable by large numbers of library users on CD-ROMs. Users would go to a library computer, which held multiple CD-ROMs containing journals and other subscriptions, and use a menu to find content on the CD-ROMs by typing keywords that would look through all the content to find matches. This was the first routine use of full-text searching by library users.
These technologies are just memories for a few of us, and unknown to most, but they do point to the differentiation between structured and unstructured searching. Both have been around for a couple of decades but it has taken Web search engines to put search in the hands of everyone. Only recently is frustration with retrieving buckets of unfiltered content pushing enterprises to reconfirm the added value of structured searching.
Technical and business users are appreciating the value of being able to search for a precise title, all documents contributed to a specific project, or all presentations delivered by the CEO in the past two years. Each of these searches requires a defined set of data points, stored with the content and retrievable with a search interface that can support the “structured” query.
Yes, librarians have been here before but, just now, the rest of the organization is learning how they managed to get such good search results all along. Structured searching is now a lot simpler than it was in the 1970s. It is only one aspect in enterprise search but it is an important requirement for most enterprise users when they need reliable and clearly defined search results. And, by the way, Carlos is still around building systems for enterprises to manage and search their critical proprietary content.
IBM just launched a very interesting suite of enterprise search products. I have yet to try it or examine the specifications but the marketing message is the right one for the small and medium business enterprise buyer. What I like in the message:
> Pricing ranges from free to reasonable to ? (sky is probably the limit).
> Deployment is simple, intuitive and clean.
> IBM knows that simple and easy is the right call for IT but also the way to keep costs down.
> The solution is scaleable from a departmental solution to the entire business domain on the same basic software platform.
For the short term, I am placing OmniFind on the long list of products to consider for enterprise search. Check out the Web site at:
How smart is it that IBM and Yahoo have chosen to team in this way? It could be a great strategy.
Disclaimer: This is the first product mention in this very new blog. It won’t be the last; I have a lot of interesting products on my list on which to comment. There will be much backfilling in the next few months but I have to start somewhere. The marketing message resonated; I hope OmniFind users will keep us informed by posting case experiences on whether the product delivers on the promises.
I seem to be attending a lot of presentations because I think they are going to be about “enterprise search.” Instead they cover a new offering or positioning strategy by a search company seeking to help enterprises monetize their Web sites. There are great business models in this space as Yahoo, Google and Amazon have illustrated. These will morph to offerings, as yet, unimagined. The trouble is that for my audience, I want to help them understand offerings that will help them with searching content already in the enterprise or from outside that they can leverage for business uses: competitive intelligence, product development, supply chain improvements, marketing collateral development. That is what enterprise search is to most people working inside organizations. Admittedly, this is not a new or “sexy” market. I think that vendors of search may be so worn down by how they can make money offering their search tools for searching inside the organization that they may just be talking up other markets to stave off their own boredom with the “inside the enterprise market.” Horizontal markets are tough to deal with (more about that in a later blog entry).
To all you vendors who would like to cut and run, I respectfully ask that you stay. There is a crying need albeit a very big financial challenge in all of this. Enterprises have no idea what their true monetary loses are because workers can’t find “stuff.” There have been plenty of guesses put forth by analysts, but until we see search solutions that don’t take decades (OK years) to implement, who actually knows what gains could be made by really good and easy search tools that find both structured and unstructured content with a minimum of set-up.
The pricing models of tools need to make better sense, including ways to chunk the expenditures incrementally with “quick start,” and low overhead options. Why do the search tools with the mightiest claims also come with the highest price tags for licensing but the least out-of-the box functionality?
To all you enterprise information technology seekers of “true enterprise search” you bear responsibility for some of the mess this market is in. You’ve got to write better specifications, learn to start small and demand small to get started, and come to the selection process with real users and professional searchers on the team who will test drive products before they are purchased. If a product doesn’t solve the specific search problem you are trying to solve, don’t buy it. May-be it doesn’t feel like your money when you purchase for your enterprise but it really is your professional responsibility. Would you buy a car that will only take you to the beach or a lake but not to the grocery store?
If you signed up for feeds from this site, new posts have been slow coming. Gilbane’s announcement of an Enterprise Search Practice has not gone unnoticed. The past two weeks have resulted in more good will than this analyst could easily digest and filter. The good news is that ideas for posting on “enterprise search” are already accumulating faster than they can get written, and the number of enthusiastic well-wishers is encouraging. It looks like we have an audience and community of practice in the making. Thank you to all who have sent their support and good cheer.
Quite a number of responses have come from companies who want to discuss their technology offerings and positioning. At Gilbane we are following up on those requests and beginning to schedule time for discussions and presentations. With the recognition that vendors/suppliers of technologies want ink, and plenty of it, comes a responsibility of which I am acutely aware because I was one of that community for over 20 years. Having founded, in 1980, and lead an integrated library automation firm in the corporate arena, I know how industry press coverage can make or break the fortunes of even the best offerings. While blogs are intended to launch and promote discussions, even play devil’s advocate, I don’t take this role lightly. Every good intention and hard work by vendors deserves thoughtful and unbiased consideration. It deserves to have analysts who know what they are talking about, and those that would present what they can fairly assess in a useful context. The very definition of analyst (noun) supposes a responsible action, to analyze (verb) the offerings. While my analysis may not focus on what a vendor wants me to consider, it will try to present information that is both helpful and thought-provoking without being mean-spirited or dismissive, and content that helps potential users of the technology focus their own choices and decisions.
Now it’s time to get down to business and start making this a more frequent happening. Based on a number of comments, let’s begin with clarifying what we mean by enterprise search at Gilbane. While the marketplace often categorizes enterprise search as a specific kind of search product, we at Gilbane don’t. Any technology that serves any type of enterprise by helping it find electronic or physical content through an electronic search interface is fair game. Enterprise search is about looking for content in the organization or for the organization. It may be embedded in a specialized application, may be a platform designed to collectively search and aggregate content from many internal silos, or it may combine search of desktops, enterprise hard drives and the Internet. There is a very big universe of content out there; enterprises need all the search tools they can (afford to) leverage to harvest what they need and when they need it.
Now this analyst’s job is to give you a balance between what the vendors are saying and offering, and what the users really need, and get the two engaging more effectively with each other.
Enterprise Search has been an illusive dream for too many organizations for too many years. Search technology is ubiquitous but the “holy grail” for most organizations is to be able to find all content within the organization through a single query interface. My instinct is to give a chronology of search over the past four or five decades to guide your understanding of why enterprise search has remained so “out of reach.” I could also describe the ways in which search technologies have evolved and morphed with hundreds of functions and thousands of features. It would certainly help explain why the typical company has a daunting task narrowing its options but it would probably not quicken the selection process.
For now, one view of the current market segmentation is a starting point. Sue Feldman, Research VP, Content Management and Retrieval Solutions at IDC, gave the audience a high level view of the market in a session at Gilbane Boston 2006. She placed enterprise search technology into three big buckets: Appliances and Downloadable Search, Enterprise Search (software) Platforms, and Application Specific Search embedded with other software. She then broadly described the features and functions that characterize each major type. If you have grown up with search in your professional life for over 30 years as I have, it makes perfect sense that this is what we have come to in the market but differentiating the options is a step far less clear-cut.
After the sessions, 15 conference-goers joined me to continue discussing and learning about enterprise search in a roundtable forum. It was hard to know which end of the search animal we should address first to help everyone speak the same language. That is precisely what is making this marketplace such a tough one. Vendors represent a huge variety of solutions, each positioning product(s) for a problem of their definition, offering technology that targets the specific problem. Buyers have multiple search needs but still want a single solution. Further complicating the mix is a dizzying array of search jargon. With vendors and buyers using their own language the market is, frankly, a real mess.
Take Ms. Feldman’s three big buckets and think of one example of search product in each category. Now think about all the types of searches that people in your organization need to perform just to get their routine work done:
- Looking up an address in a directory
- Finding an image for a presentation
- Retrieving a press release your department issued last year on a new product
- Locating a configuration change to a piece of equipment in manufacturing
- and so on…
Can you imagine any single search interface or product from the tools you know that would give you the means to find all of these pieces of information? Can you imagine a single search tool that would answer your query in a couple of simple steps, and able to perform the functions right out of the box? Simple solutions that address the complexity of business variables and technology standards in most organizations make any single solution an unlikely candidate at a reasonable cost.
Blog readers can request answers to questions, ask for help with sorting out the marketplace or definitions to understand the jargon. I invite readers to tell me what you think needs to be talked about and I’ll give it my best shot. What do you need to know first to tread through the search marketplace?
The Gilbane Group has invited me to lead a new practice area for the Gilbane community. Based on my recent experience at Gilbane Boston, November 2006, there is the foundation for a substantial community of practice around the topic of Enterprise Search. When I invited attendees of conference sessions on search to a preliminary roundtable discussion about enterprise search, over 15 people signed up. About 12 people dropped in for all or part of what turned out to be a two hour lunch break with a fluid conversation about what search means, what users are seeking for their organizations and what vendors (service and product) have to offer.
I am inviting anyone who drops in on this blog to continue the conversation. By sharing needs, product offerings, definitions, problems and case studies, participants in this blog have the opportunity to contribute to a community of practice among a highly diverse audience of professionals concerned with this topic of search. We will learn what is working or not, what tools, tips and processes have been used and leveraged for improving business performance in any type of organization.
You will note a group of categories that I established because I have something I would like to share sooner rather than later about these topics. Not everything I may have to say in the next few weeks or months will fit neatly; expansion is inevitable. The categories are broad until we begin to accumulate content in other areas. One thing is certain; technology changes and changes our thinking. A year from now it will be interesting to look back at recommendations, advice, enthusiasms and endorsements in this time period and see where reversals happen and attitudes morph. We are all capable of love hate relationships with technology and I am as fickle as the next person.
Stay tuned and see where we are January 1, 2008. Still blogging, I hope.