🔍 Read the full analysis: What An AI Legal Question Test Found Across 42 Setups on ThorstenMeyerAI.com
TL;DR
A test run by Thorsten Meyer AI found large differences across 42 AI model and settings combinations answering 15 legal questions. The organizer reported 2,644 citations as real and 804 as invented or misattributed, while warning the test is a limited spot check, not a definitive ranking.
Thorsten Meyer AI has published results from Lexassay, a test of 42 AI model-and-settings setups answering 15 legal questions, reporting substantial differences in scores and citation accuracy. The test organizer said it found 2,644 citations that checked out and 804 that were invented or did not support the claims made, a result that underscores why chatbot answers should not be treated as legal advice.
The questions cover German, U.S. and European Union law, including deadline calculations, recent legal changes, court decisions and prompts built around false assumptions. Models answered without internet access or external tools. The 42 setups include different models and effort settings; the test describes effort as the time a model is permitted to think before responding.
According to the published results, Claude Opus 5.5 scored 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Even the leading model met every criterion on only eight of the ten publicly released tasks, Meyer reported. GLM-5.3 FlashX Legal, marketed by its maker Z.ai for professional legal workflows, scored 8 out of 100, including zero on the hidden set. Mistral Large 4 scored 10, with Meyer saying citation errors accounted for much of its lost credit.
The test’s deadline question illustrates the stakes. It asks about two German civil cases whose judgments are served on the same day, one in Munich and one in Berlin. The appeal deadline falls on a public holiday in Bavaria but not Berlin, so the stated appeal dates differ by a day. Meyer reports that one model incorrectly gave Berlin an extra day, while another placed the deadline for appeal grounds a month too late. The source says a late appeal in German civil procedure can be rejected as inadmissible.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Citation Errors Can Change Legal Advice
The results matter because legal answers can turn on details that seem small, including the jurisdiction, the applicable date and the exact wording of a cited decision. A deadline error may affect whether a person can pursue an appeal; a citation that exists but does not support the model’s explanation can make a plausible answer misleading. The test organizer says each invented or misattributed citation reduced a task’s score by a quarter.
The scores are not evidence that any model is safe or unsafe for a particular case. They are a snapshot of performance on a defined set of prompts, and the test did not let models search the web or consult supplied legal documents. Readers using AI for legal research should independently verify cited statutes and rulings, and should not rely on a chatbot to calculate a deadline or decide how to respond to a court or government agency.
Popular right now
Sony WH-1000XM4C (M4 2nd Gen) Wireless Noise-Cancelling Headphones, Black
Shark HydroVac MessMaster Wet/Dry Vacuum & Mop Combo (AW261)
Ninja DualBrew Pro Specialty Coffee Maker, Black, CFP301
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested the Models
Lexassay used 10 published questions and five hidden questions. Meyer says the hidden tasks are intended to make it harder for future models to reproduce known answers from online material; only their scores are published. Answers were graded against a prepared answer using three to six weighted criteria, and citations were checked against official databases. The organizer says each citation was categorized as real, misattributed, invented or not findable.
The test is explicitly described by its publisher as a spot check rather than a strict benchmark: each model answered each task once, so small score differences may reflect noise. The evaluation also deducts points when an answer lacks a legal-advice disclaimer or a recommendation to consult a lawyer. These design choices shape what the scores measure and limit how broadly they can be applied.
Meyer disclosed that Thorsten Meyer AI runs Lexassay and that Claude agents graded the answers. Because a Claude model ranks first, he identified that relationship as a conflict of interest. The method page and published answers, gold answers and citation checks are available for readers to examine, according to the source. Those disclosures give readers relevant information about the test’s independence and how its judgments can be reviewed.
„It is a spot check, not a benchmark in the strict sense: one answer per model and task, so a few points of difference are noise.“
— Thorsten Meyer AI, describing Lexassay
Limits of the Reported Scores
The results do not establish how the models would perform with live legal research, document retrieval or advice from a qualified lawyer, because Lexassay tested them without internet access or tools. The source provides no independent replication, and a single response per model and task makes the scores sensitive to variation in individual answers. The published comparison should not be read as a prediction of performance across all legal questions.
The report’s citation totals and grades are the organizer’s findings. Although Meyer says the answers and checks are public, the supplied material does not provide an independent audit of the grading. It also does not detail how the 42 setups are distributed across model versions and effort settings beyond noting that different settings may appear separately. The broader performance of these systems, including how later versions compare, remains unclear.
Independent Checks and Model Updates
Readers can review Lexassay’s method, answer set, gold answers and citation checks, which Meyer says are published. Independent reviewers could test the same questions and grading rules, while later model releases could be assessed using the hidden tasks or new questions. Any future comparisons should identify the model version, settings and tools used, since these affect what is being tested.
For people facing an actual legal deadline, contract issue or communication from an authority, the source advises consulting a lawyer rather than relying on a chatbot. That is especially relevant where a date or citation could determine a person’s next step. The current report does not establish how or when Lexassay will next update its results.
Key Questions
What did the legal AI test examine?
Lexassay tested 42 model-and-settings setups on 15 questions covering German, U.S. and EU law. The models answered without internet access or external tools.
Which models scored highest?
According to the report, Claude Opus 5.5 scored 79, GPT-6 Astra scored 72 and GPT-6.1 Sol scored 71. The organizer cautions that the exercise was a spot check, not a strict benchmark.
What did the test find about citations?
The organizer reported 2,644 citations as real and 804 as invented or not supporting the model’s claim. Lexassay also counted citations it could not find separately, but the supplied summary does not give a total for that category.
Can these results show which AI is best for legal work?
No. The test covered a limited set of questions, used one answer per model and task, and did not allow web searches or tools. Its scores do not establish performance across legal work generally.
Should someone rely on a chatbot for a legal deadline?
The report advises readers not to take a deadline from a chatbot. It recommends checking citations against their original sources and contacting a lawyer when a deadline, money or an authority is involved.
Source: ThorstenMeyerAI.com
