Everyone seems to be jumping on the artificial intelligence (AI) bandwagon these
This lesson was reinforced in the recent case of Matter of Weber. The court ruled that an expert’s AI-generated output was unreliable and, therefore, his opinion was inadmissible as evidence.
In Matter of Weber, a trust beneficiary objected to the trust’s accounting. He alleged that the trustee breached her fiduciary duty by retaining certain real estate in the trust and using it for her personal benefit. The beneficiary claimed that the trustee should have sold the real estate, which would have allowed the trust to enjoy significant profits and invest the proceeds in the Vanguard Balanced Index Fund.
The beneficiary hired a financial expert to support his breach claim and alleged damages. However, the Saratoga County Surrogate’s Court found that the beneficiary failed to prove that the trustee breached her fiduciary duties. It also rejected the expert’s damages calculations, including a supplemental damages report.
To cross-check the calculations for his supplemental damages report, the expert used Microsoft Copilot, a generative-AI chatbot similar to ChatGPT. However, he couldn’t recall the inputs or prompts provided to Copilot or state the sources Copilot relied on. He also couldn’t explain how Copilot works or arrives at its output.
To test Copilot’s accuracy, the judge entered the following prompt on a court system computer: “Can you calculate the value of $250,000 invested in the Vanguard Balanced Index Fund from December 31, 2004, through January 31, 2021?” Copilot returned a value of $949,070.97, which differed from the expert’s conclusion. The court ran the same query on two different computers that generated values of $948,209.63 and just over $951,000, respectively. Although the discrepancies weren’t significant, they “question the reliability and accuracy of Copilot to generate evidence to be relied upon in a court proceeding.”
When the judge asked Copilot, “Are you accurate?” The chatbot replied, “My accuracy is only as good as my sources so for critical matters, it’s always wise to verify.” When asked whether its calculations are reliable enough for use in court, Copilot responded, “When it comes to legal matters, any calculations or data need to meet strict standards. I can provide accurate info, but it should always be verified by experts and accompanied by professional evaluations.”
The court noted that the use of AI is expanding rapidly across various industries. However, that doesn’t make its results admissible in court. Given the “rapid evolution” of AI and its “inherent reliability issues,” the court held that before AI-generated evidence is introduced, “counsel has an affirmative duty to disclose the use of [AI] and the evidence sought to be admitted should properly be subject to a Frye hearing prior to its admission.” (The Frye standard is the benchmark used in New York to evaluate the admissibility of expert testimony.)
AI can be a valuable tool, but as the Weber case demonstrates, it presents numerous challenges when used in a litigation setting, and its output should always be independently verified. AI currently isn’t — and will likely never be — a substitute for the skills and judgment of experienced professionals. For more information, contact us today.