Technology-assisted review (“TAR”) is being more widely adopted as a way of saving time and money in large-scale document reviews. But virtually everyone assumes that the advanced text-analysis algorithms underlying TAR and embedded in tools like Relativity, Recommind, Clearwell and Equivio can really only be trusted to perform basic relevance coding, so-called first pass review or maybe issue-based classification.
Can you think of anyone who has suggested that these tools can perform the much more challenging and nuanced task of identifying privileged content? Most lawyers would scoff at the idea. And for very important reasons, their scepticism must be taken seriously. Entrusting priv review to a machine is a high-stakes proposition. Some work in this area as part of TREC 2010 (Interactive Task 304, see pp. 33-35) yielded poor results.
But what if a tool could be trusted to find the kind of text that lawyers should look at, to assess both privilege and waiver? And what if that tool did a better job of finding that kind of text than the average reviewer? Even a well-trained reviewer?
Working with Porfiau, a well-credentialed software development company run by a team of text-classification experts with connections to the University of Waterloo, KPMG Canada has both helped to enhance the computational tool built by Porfiau and developed a set of workflow protocols whereby a new generation of text-classification technology appears to have achieved this goal. The technology uses finite state machines and the workflow is much like those already adopted in TAR situations (start with a seed set, build the algorithm, get a first set of results, have a SME code a sample, retrain the algorithm, and so on). Initial results, based on several iterations and retrainings against a target population of over 300,000 documents, strongly suggest Porfiau’s technology, together with carefully designed processes, can identify potentially privileged content with a level of recall of 0.90 or higher.
And the tool has consistently found important potentially privileged material that the lawyers missed.
This work is described in the final section of a paper that I co-wrote with Chris Paskach and Manfred Gabriel of KPMG US, “The Challenge and Promise of Predictive Coding for Privilege” [PDF] The paper is one of four selected by peer review for presentation at the DESI V Workshop in Rome this coming Friday.