Key Considerations in a PoC

Determining which software vendor delivers the right solution - the most robust, precise, secure and scalable infrastructure - can challenge even the most seasoned CIO and technology buyer. Typically, testing competing technologies can take one of the following two forms. First, a discreet demonstration can occur, in which the vendor receives a set of the customer's data and executes tests based only on the information received. Alternatively, a more comprehensive, onsite exercise can be deployed to ensure that the vendor's product will work in the customer's environment and function in the way it was promoted during the sales process. In this Proof of Concept, the vendor actually installs its software in the customer's environment to determine how the product will function within its organization, and specifically, to differentiate what the vendor claims it can do and what the vendor can actually do.

The following best practices are recommended when you compare software from multiple vendors during a PoC:

1. Always Use Blind Data: To ensure a level playing field among competing vendors, it is imperative that vendors work with blind data as this will prevent the vendor from exploiting a product's advantages and hiding a product's limitations. In addition, this ensures that the testing of data takes place onsite as an offsite exercise opens the door for one or multiple vendors to skew the PoC to their advantage.
2. Always Require More than Keyword Matching to Score Search Relevance: Be careful about relying on relevance scoring that is based only on keyword matching. Keyword search has become a virtual commodity with major search vendors and this should be used as a baseline. Retrieval relevance should be determined by ensuring documents returned are actually those documents your users want.
3. Always Ensure Relevance of Results: In order to preserve the appearance of high level performance, some technologies will not index more than a set number of characters in a document. As a result, relevancy suffers because users are only able to retrieve information across the first few pages in large documents while missing what could be the critical piece of information tucked away on the last few pages of a large document. The remaining unsearched area of the document becomes what is referred to as a "dark object" that goes unrecorded within the index. Make sure you seed your testing data with documents that contain one hundred pages or more, forcing the vendor to search and index documents in their entirety rather than limit retrieval to a reduced portion of a page in an effort to speed up your engine.
4. Always Require Security and Scalability: Building a secure results list can be challenging. But when assessing the security claims of a product in a PoC, it's important to test scalability as well. Before a document can join the list of matches against a user's query, it must first be checked against a user's entitlement in each of the multiple repositories such as Lotus Notes, Exchange, LiveLink, NT, etc. In a typical enterprise, a user may be restricted to only see a fraction of the total number of documents. Millions of query-matching documents may need to be checked in order to return a standard results list of just 10 hits. The only method that guarantees security and provides the high scalability that enterprises require is mapped security. With mapped security, a secure mapped query checks the permission directly within the engine at query time and does not have to make any external reference. In addition, the security map is updated as soon as permissions change in the underlying repository. This means the security is never out-of-date.
5. Always Require Full Performance: Some search products stop looking across an index as soon as they believe they have assembled a large enough group of results, rather than the correct group of results. This means that for any given query an arbitrary set of documents is returned rather than the most complete or accurate set, typically known as a 'jump-out.'
6. Always Require Vendors to Stand Behind their Product with a Viable Warranty: If practical testing logistics actually prohibit a complete testing program, or if additional assurance is desired, the customer should request a warranty from the vendor. If this cannot be given, or is not given in sufficient detail, the customer must question the engineering validity of the technology being proposed.

Only Autonomy is uniquely positioned to stand behind these guidelines and offer you the best possible solution for your enterprise. For a more detailed description of each of these guidelines and on conducting a successful PoC, the reader is referred to the Autonomy white paper: "A Guide to a Successful PoC: The Dos and Don'ts for Selecting Unstructured Data Management Software," available from www.autonomy.com.