Five weeks to find out whether AI can judge the merit of a medical malpractice case.

Expert Institute connects attorneys with medical experts, and for a potential malpractice matter the work begins with a medical record review, in which a physician reads the case summary and the underlying records and renders an opinion on whether the case has medical merit. Each one takes a physician close to two hours across some forty medical specialties, and represents a constraint on the whole business. Leadership wanted to know whether large language models could produce a credible first-pass opinion, both to accelerate their own physicians and, eventually, to offer a lower-cost self-service assessment to the much larger population of firms that can’t justify a full expert consultation today.

The uncertainty was as much legal as technical. Record packages run to hundreds of pages, arrive as scans, faxes, images and zip files, and the reasoning a physician applies to them is partly tacit and shaped by years of conversations with attorneys. Anything the system produced also had to hold up under scrutiny, since an opinion generated during legal due diligence can become discoverable. Expert Institute asked us to answer the feasibility question in five weeks, without the overhead of a full software build.

The engagement was scoped as a merit predictor, and within two weeks of physician interviews we recommended changing what the tool should predict. Every physician we spoke with told us the final score is tailored to the opinions and risk appetite of the attorney and the firm, that one client’s positive is another client’s borderline, and that it often shifts on the debrief call. A definitive score would also be difficult to validate, because roughly 90% of cases settle privately with no documented outcome. So we redirected the prototype toward explanation: a structured account of standard of care, causation, damages, and the strengths and weaknesses of the case, leaving the merit judgment to the physician or attorney reading it.

We built in Google AI Studio and n8n, working directly against Gemini’s API, which let us change prompts and logic daily without waiting on a development cycle. The interviews showed that physicians evaluate cases in consistent stages, so we structured the workflow as a four-step agentic pipeline that mirrors them, with structured JSON at each step so a reviewer can inspect where the reasoning went wrong. We narrowed the scope to orthopedic malpractice to keep prompting precise, and we passed full record packages to Gemini as PDFs, which proved more accurate than OCR and handled document sizes that would have exceeded the context limits of other frontier models. The evaluation set came from Expert Institute’s own data, including its library of 16,000 recorded physician-attorney consultations. Their physicians graded every output against two bars: useful enough to speed up an internal review, and good enough to send to a client verbatim.

In five weeks, and without writing production software, we answered the feasibility question. Expert Institute’s own board-certified specialists reviewed the prototype’s analysis of real cases and judged 67% of it good enough to accelerate their own case reviews, exceeding OpenAI’s 57.8% benchmark at the time. The evaluation framework also showed a clear pattern in the cases that fell short, most of them traceable to inputs the prototype had never been given or to a case type the prompts weren’t yet tuned for, and correcting those failure modes alone would lift the pass rate to 89%.

We left Expert Institute with a phased roadmap that starts with an internal physician co-pilot and grows into a self-service product for law firms, with a decision point before each round of investment, along with a business case that modeled six-figure annual savings from the first release and a seven-figure revenue opportunity from the second. More than the prototype itself, the engagement gave their leadership a clear view of what an AI merit tool could and couldn’t do, and where it fits in the product line.