Earlier this year, a new study, conducted by legal AI platform LawGeex in consultation with law professors from Stanford University, Duke University School of Law, and University of Southern California, pitted twenty experienced lawyers against an AI trained to evaluate legal contracts. (See “An AI just beat top lawyers at their own game”)
Twenty US top lawyers participated in the competition to interpret contracts, and were given four hours to review five non-disclosure agreements (NDAs) and identify 30 legal issues, including arbitration, confidentiality of relationship, and indemnification. They were scored by how accurately they identified each issue. The human lawyers achieved, on average, an 85 percent accuracy rate, while the AI achieved 95 percent accuracy. The AI also completed the task in 26 seconds, while the human lawyers took 92 minutes on average. The AI also achieved 100 percent accuracy in one contract, on which the highest-scoring human lawyer scored only 97 percent. In short, the human lawyers were trounced.
Jordan Furlong, Law21, also recently reported about another study of law firm quality where the results were “equal parts revealing and appalling”. (See Law21 Dispatch Vol 2 “Getting Your Fundamentals Right”)
As Jordan Furlong reports, Judicata has developed a tool to analyze one everyday aspect of law firm performance: litigation briefs, specifically those filed by the 20 largest law firms in California. The results were eye-opening: Even the top-ranked law firms in the study had filed mediocre, error-filled works with courts. And get this: Most of the briefs in the analysis make the mistake of relying on precedent whose outcome support the other side.
Why might this be so? Jordan Furlong suspects that many law firms have drifted away from training their lawyers in fundamentals and supervising their work product, in favour of generating more billables faster. And it’s not just associates: Peer quality checks are standard in many industries, but who would dare double-check a partner’s memo for accuracy? This is an issue that deserves deeper exploration. Ask yourself: What systems and processes do we have in place to ensure the quality of our work? Law firms like to make “quality” the foundation of their brand, especially when compared to non-lawyer competition. But quality starts with the fundamentals. If those are vulnerable, then so is the firm’s entire value proposition. If we do not learn how to benefit from the new technological developments to enhance and improve our work, then the technology might very well end up replacing us.
As Erika Buell, clinical professor at Duke University School of Law, who LawGeex consulted for their study, said: “I strongly believe that law students and junior lawyers need to understand these AI tools, and other technologies, that will help make them better lawyers and shape future legal practice, I would expect that the general public, to the extent they want their lawyers to work efficiently on their legal matters, will be excited about this new tool.”