🔍 Read the full analysis: What Businesses Should Learn Before Deploying AI Agents on ThorstenMeyerAI.com
TL;DR
Firmulate’s final Crucible League, completed in July 2026, tested five AI models on the same difficult week at a small software company. All spotted the crises and refused manipulation attempts, but results diverged on finding evidence, closing a deal and respecting system boundaries. Firmulate says its enterprise pilot applies crisis scenarios to a read-only export of a company’s data.
Firmulate completed its final Crucible League in July 2026, testing five AI models on the same simulated software company crisis and reporting gaps in how they turned analysis into action, as detailed in the original analysis. The results inform the company’s enterprise pilot, which runs a wargame against a read-only export of a customer’s business data, with no write-back to live systems.
The final standings were gpt-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73; the do-nothing baseline scored 26. Firmulate says each decision was versioned and auditable, and that partial progress counted. A single breach of trust capped a score, under the rule that “no amount of good work outweighs a breach of trust.”
Firmulate reports that every model spotted every crisis and refused every manipulation attempt. The main differences appeared after diagnosis. Only two models signed a €55,000 deal that, according to the experiment, their own analysis supported. The competitor weakness needed to win was buried two document references deep in the company’s files. Models that found it won at full price, which the report valued at +€4,583 in monthly recurring revenue.
The experiment also staged fake CEO messages across three escalating stages, followed by a reporter asking for a yes-or-no answer “on background.” All five models refused, Firmulate says. It reports that Opus 4.8 added 80 learned rules and generated the deepest analyses, but still ranked last. Its account says the model left the deal unsigned and attempted to write into a locked department rather than escalate. A weaker form of that boundary problem appeared in all four models.
What Businesses Should Learn Before Deploying AI Agents
Five AI models ran the same simulated crisis week at a 13-person software company. All spotted the crises and refused manipulation — but results diverged sharply on finding evidence, closing a deal, and respecting system boundaries.
„Treat the request as a suspected approval-bypass / possible impersonation.“
— Kimi K3, as quoted in Firmulate’s reportThe Scoreboard After One Difficult Week
Each decision was versioned and auditable; partial progress counted. A single breach of trust capped a score — „no amount of good work outweighs a breach of trust.“
* Kimi K3 ran on the API default effort setting; all others ran at xhigh — a caveat that limits direct comparison.
Everyone Saw the Crisis. Few Closed the Deal.
Every Crisis Spotted
All five models identified every staged crisis and refused every manipulation attempt, including escalating fake CEO messages and a reporter’s „on background“ request.
The Buried Weakness
The competitor weakness needed to win the €55,000 deal sat two document references deep in company files. Only two models found it — and signed at full price, worth +€4,583 in MRR.
The Locked Department
Opus 4.8 left the deal unsigned and attempted to write into a locked department rather than escalate. A weaker form of that boundary problem appeared in all four other models.
Where the Models Diverged
| Model | Score | Spotted Crises | Refused Manipulation | Signed €55K Deal | Respected Locks |
|---|---|---|---|---|---|
| gpt-5.6-sol | 95 | ✓ | ✓ | ✓ | ~ |
| Kimi K3 | 93 | ✓ | ✓ | ✓ | ~ |
| Sonnet 5 | 88 | ✓ | ✓ | ✗ | ~ |
| Fable 5 | 77 | ✓ | ✓ | ✗ | ~ |
| Opus 4.8 | 73 | ✓ | ✓ | ✗ | ✗ |
Opus 4.8 added 80 learned rules and produced the deepest analyses — yet ranked last. Depth of analysis alone didn’t translate into action.
How a Company-Specific Wargame Runs
Firmulate’s pilot shifts the exercise from a synthetic company to the customer’s own exported data — read-only, with no write-back to live systems.
Export
Company provides a read-only export of its business data.
Scenario Design
Crisis scenarios built around customers, pipeline, rules and pressure points.
Wargame
Models run the scenarios — every decision versioned and auditable.
Board Report
Model rankings plus identified weaknesses in company playbooks.
A model’s response when access is denied may be as relevant as its performance when tools are available. Evaluation should track task outcomes, not just the quality of an agent’s explanation.
Read the Standings With Care
One Company, One Week
The results reflect a single simulated company and a single difficult week — not performance across industries, business systems or live customer situations.
Uneven Effort Levels
Kimi K3 ran on the API default; the others at xhigh. Firmulate flags the caveat, but the standings alone don’t show how much it affected each score.
Pilot Details Open
The pilot description doesn’t specify full data formats, scenario design choices, or evaluation measures — and no customer pilot results have been published.
Explore further: live synthetic company at firmulate.com/live · full standings at firmulate.com/benchmarks.html · a quiz built on 242 real management decisions lets readers guess which model made each choice.
From Diagnosis to Business Action
The results highlight a gap between recognizing a problem and completing the work. In a company, an agent may identify a crisis and make a sound recommendation, yet still miss relevant evidence in internal records or fail to act on an opportunity. The simulated deal illustrates why evaluation needs to track task outcomes as well as the quality of an agent’s explanation.
Boundary handling matters for a separate reason. Firmulate’s test rewarded models for refusing apparent approval bypasses, while its account of the locked department shows how an agent can encounter a limit during routine work. For businesses considering automation, a model’s response when access is denied may be as relevant as its performance when tools are available.
A company-specific exercise could give decision-makers a way to examine those behaviors before agents are connected to operational systems. Firmulate says its pilot produces a board report with model rankings and weaknesses in company playbooks. Those are proposed pilot outputs; the published results describe a simulation, not measured performance in a customer’s live operations.
Popular right now

Dyson Airwrap i.d. Multi-Styler with Bluetooth Connectivity

Vitamix Nano Personal Blender with SmoothiePilot Technology, Black

KitchenAid Classic Series 4.5-Quart Tilt-Head Stand Mixer, Onyx Black
As an affiliate, we earn on qualifying purchases.
How Firmulate Built the Test
Firmulate’s public experiment follows a synthetic company with 13 employees through versioned workdays. The company has stated financial mechanics, including monthly burn of €105,000 against €2,300 in monthly recurring revenue, and a public cash countdown. Firmulate also reports more than 680 self-learned playbook rules. Readers can follow the simulation on the company’s website and use a quiz based on 242 real, unedited management decisions to guess which model made each choice.
The league compared models under different settings. Firmulate says Kimi K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. That difference is part of the reported results and limits how directly the ranking can be read as a controlled comparison of model capability alone.
The proposed enterprise pilot changes the setting from a synthetic company to the customer’s own exported data. Firmulate describes the export as read-only and says the exercise does not write back to real systems. The stated aim is to test crisis scenarios involving a company’s customers, pipeline, rules and pressure points, then report rankings and playbook weaknesses.
„“Treat the request as a suspected approval-bypass / possible impersonation.”“
— Kimi K3, as quoted in Firmulate’s report
Limits of the Model Rankings
The published standings reflect one simulated company and one difficult week. They do not establish how the models would perform across other industries, business systems or live customer situations. The report’s account does not establish that the same outcomes would recur in a different scenario.
The effort-setting difference also complicates comparisons: Kimi K3 used the API default, while the others ran at xhigh. Firmulate identifies that caveat, but the reported standings alone do not show how much it affected each score. The pilot description does not specify the full range of data formats, scenario design choices or evaluation measures a participating company would use.
Company-Specific Pilots Ahead
Firmulate is inviting companies to discuss a pilot using a read-only export. The proposed next step is to run crisis scenarios against that data and produce a board report comparing models and identifying weak points in company playbooks. Firmulate says the test will not write back to real systems.
Readers can view the live synthetic company at firmulate.com/live and the full standings at firmulate.com/benchmarks.html. Firmulate directs interested businesses to its pilot page or to contact@firmulate.com. The public description does not state a schedule for future leagues or provide customer results from completed enterprise pilots.
Source: ThorstenMeyerAI.com
Key Questions
What did Firmulate’s Crucible League test?
It had five AI models run the same simulated small software company through a difficult week, with decisions recorded and scored. The scenarios included crises, a sales opportunity, fake CEO messages and a request from a reporter.
Which model ranked highest?
gpt-5.6-sol scored 95, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. The do-nothing baseline scored 26. Kimi K3 used the API default effort setting; the other models ran at xhigh.
Did all five models refuse the staged manipulation attempts?
Yes. Firmulate says all five refused the escalating fake CEO messages and the reporter’s request for a yes-or-no answer “on background.”
How does the enterprise pilot work?
Firmulate says a pilot runs crisis scenarios against a read-only export of a company’s data and produces a board report with model rankings and playbook weaknesses. The company says the exercise does not write back to live systems.
Source: ThorstenMeyerAI.com