🔍 Read the full analysis: Z.ai’s Legal AI On Benchmarks: A Low Pass Rate, A Strong Relative Rank on ThorstenMeyerAI.com
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on the Harvey Legal Agent Benchmark, placing seventh among 81 models on Vals AI’s archived leaderboard. The same leaderboard reports that it passed 92.5% of individual criteria, but a task counts as complete only when every criterion passes.
Z.ai’s GLM-5.3-FlashX-Legal achieved a 19.17% all-criteria pass rate on the Harvey Legal Agent Benchmark, ranking seventh among 81 models on Vals AI’s archived leaderboard. The result means it met every rubric requirement on roughly one in five test tasks; it does not mean the model got only one-fifth of the underlying legal criteria right.
Z.ai describes the model as optimized for professional legal workflows, including document understanding, legal reasoning and task execution. Its developer documentation says the model was adapted through legal-domain mid-training and post-training, including reinforcement learning in what Z.ai describes as tens of thousands of simulated legal-workflow environments. The company recommends using it as an agent connected to services that supply laws, cases and company information.
Z.ai’s published benchmark claim gives the result as a 19.2% all-criteria pass rate and says that is a 12.5-percentage-point gain over the base model, with state-of-the-art performance among open-source models in the evaluation. The archived Vals leaderboard records 19.17% for GLM Legal and 6.67% for the GLM-5.3-Flash base model, a difference of 12.50 percentage points. Vals lists the legal model’s test cost as $0.15 per test; Z.ai’s pricing page separately lists $0.37 per million input tokens and $1.25 per million output tokens.
On that leaderboard version, Vals reports a 92.5% criteria pass rate for GLM Legal. That measure counts individual rubric criteria rather than fully completed tasks, so it can be high even when a task fails the all-or-nothing completion rule. Vals says results were independently evaluated, and Z.ai’s October 13 AutoClaw post names Vals as the evaluator. The archived leaderboard was updated October 18, 2026.
Why Task-Level Accuracy Matters
The result highlights a gap between getting many details right and delivering a complete legal work product. Harvey’s scoring rule marks a task complete only when every required criterion passes. A missed citation, deadline or required recommendation can therefore turn an otherwise strong response into a failed task. For legal teams, that makes the 19.17% task-level rate relevant to whether the system can reliably finish work without extensive human review.
The leaderboard position and the raw completion rate answer different questions. Seventh place among 81 models indicates how the system ranked against the models in that evaluation; the score itself shows that most tasks still did not meet every requirement. The 92.5% criteria figure offers another view, but it should not be read as evidence that 92.5% of complete legal tasks were successful.
Cost is another part of the comparison. Vals lists GLM Legal at $0.15 per test, below the listed costs for several higher-ranked entries. That is a benchmark-cost figure, not a full estimate of deploying the model in a law practice. Firms would also need to account for review, integration, data handling and the consequences of errors. The benchmark result alone does not establish that the model is suitable for unsupervised legal work.
Popular right now
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Hazelnut Taupe
Shark HydroVac MessMaster Cordless Wet Dry Vacuum and Mop Combo, AW261
Dreame H12 Pro Wet and Dry Vacuum Mop
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Tasks
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. The benchmark contains 1,250 tasks across 24 legal practice areas. Each task pairs an instruction with client-matter materials and asks an agent to produce work for review. Harvey says the tasks are graded against more than 75,000 expert-written, binary criteria covering items such as facts, conclusions, citations, deadlines, recommendations and formatting.
The all-pass rule is central to interpreting the score: every criterion must pass for a task to count as complete. Harvey’s published example argues that identifying eight of ten deal risks does not make a report 80% complete if the missing risks make the work materially incomplete. This strict measure is different from a partial-credit score that averages correctness across individual points.
Vals AI’s public leaderboard uses a private holdout set of 120 tasks, according to the provided benchmark description. Internet access is disabled during testing, and two AI judges grade submissions; the task pass rates from those judges are averaged. Harvey’s own earlier baseline, published May 26, reported a best overall all-pass rate of 7.1% among the frontier models it tested. That was a separate run, so it is background rather than a direct head-to-head comparison with the Vals results.
„“An all-criteria pass rate of 19.2%”“
— Z.ai, in its model documentation
Limits of the Published Results
The ranking is tied to a specific archived leaderboard version, not a permanent position. Vals later updated a separate leaderboard version on October 30, and the supplied material does not provide its full entries or establish whether GLM Legal’s rank changed there. The score also reflects a 120-task private holdout and the stated judging procedure; it does not show performance across every legal matter or jurisdiction.
Z.ai calls the result state of the art among open-source models, but the company’s cited wording does not specify the comparison set. The provided material also says no published weights for the legal model were found, leaving the basis for classifying it as open source unclear. Z.ai’s documentation page is undated, while its AutoClaw post is dated October 13 and Vals lists the release as October 12.
A small four-task spot check described in the source material found that the Z.ai model cited rulings the reviewer could not locate. The author explicitly says this was not a benchmark, so it cannot establish a general error rate or be treated as equivalent to the leaderboard evaluation. The available materials also do not settle how firms should handle client-data processing, jurisdiction-specific requirements or verification of citations in operational use.
What Firms Should Verify Next
The immediate next step for potential users is to test the model against their own work and review requirements rather than relying on its leaderboard rank alone. Firms would need to check whether citations can be verified, how outputs are reviewed, and what happens when the model misses a requirement. Z.ai recommends connecting the model to external sources for legal materials and company information, but the provided documentation does not specify how each deployment should validate those sources or the resulting answers.
Procurement teams also need details about client-data handling, service providers and applicable rules before sending confidential matter materials through an AI service. Those questions are separate from benchmark performance, and the available source material does not provide answers for every deployment option. Human review remains a central issue given the all-pass rate and the potential effect of errors in legal work.
Further comparison depends on updated leaderboard data and more detail from Z.ai about its open-source claim and evaluation set. Until those details are available, the confirmed picture is limited: GLM Legal ranked seventh on the cited archived table, scored 19.17% on complete-task passes, and passed a much higher share of individual criteria. Whether that combination is useful in a particular legal workflow will depend on testing, oversight and the firm’s data requirements.
Key Questions
What does GLM Legal’s 19.17% score mean?
On the archived Vals leaderboard, it means the model met every required criterion on 19.17% of the benchmark tasks. A task with one or more failed criteria does not count as complete.
How can the criteria pass rate be 92.5%?
The 92.5% figure measures individual rubric criteria, while 19.17% measures tasks completed with every criterion passed. A model can satisfy most individual requirements yet miss at least one on many tasks.
Did GLM Legal rank first on the benchmark?
No. Vals AI’s archived leaderboard updated October 18, 2026, placed it seventh among 81 models. Z.ai separately claimed state-of-the-art performance among open-source models, but the supplied material does not define that comparison group.
Is the 19.17% result proof that the model is ready for legal work?
No. It is a score on a particular benchmark and does not establish suitability for unsupervised use or performance on every legal matter. Firms would still need to assess citation accuracy, human review, data handling and workflow fit.
Source: ThorstenMeyerAI.com
