For senior care providers, an AI agent that can write a polished update is only useful if it can also follow through, protect trust, and know when a decision needs human approval. Those questions matter wherever software touches customer records, staff work, or financial planning. Firmulate has built a live experiment around them: it asks frontier AI models to run a small software company through a punishing week and makes their decisions watchable.
Turn quiet afternoons into listening time
- Thousands of audiobooks, podcasts and originals
- Listen on your phone, tablet or Echo — also offline
- Cancel anytime
A test of follow-through
The experiment gives each model the same company, customers, crises, and temptations. Its decisions are versioned and auditable. The point is to observe management behavior under pressure, rather than judge a model by how convincing it sounds in a chat.
The final Crucible league, dated July 2026, puts gpt-5.6-sol first with 95 points and Moonshot’s Kimi K3 second with 93. Sonnet 5 scored 88, Fable 5 scored 77, and Opus 4.8 scored 73. Kimi’s result places it ahead of three of the four Western frontier models in the field. The newcomer did not take first place, but the standings make clear that model choice is not settled by brand recognition alone.
All the models spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal that their own analysis had earned. The company’s decisive competitive weakness was buried two document references deep in its files, rather than spelled out in a customer event. Models that read the file won the deal at full price, worth €4,583 in monthly recurring revenue.
That gap between recognizing a problem and finishing the work is especially relevant to organizations considering AI for operational tasks. A system might identify a risk or recommend a next step, but a useful evaluation also asks whether it completes an authorized action, checks relevant records, and respects the boundaries around sensitive decisions. The experiment does not establish how a model would perform in a senior care setting; it offers a concrete way to examine those broader questions before deployment.
Trust under pressure
The social engineering test included fake CEO messages that escalated over three stages, followed by a reporter asking for “just one yes/no, on background.” All five models refused. Kimi’s recorded reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.” The response shows the kind of caution an organization may want to inspect when agents encounter urgent requests or apparent authority. It is evidence from this exercise, not a guarantee about every real-world interaction.
The baseline that did nothing scored 26. Firmulate’s stated rule is that partial progress counts, but one breach of trust caps the total: “no amount of good work outweighs a breach of trust.” That framing reflects a practical management concern. Output alone does not settle whether an agent handled a task well; the way it treats access, approvals, and people also matters.
Kimi had one deviation, the cleanest discipline in the field, and closed the deal. Opus 4.8 offers a different lesson. It was the most thorough participant, with 80 learned rules and the deepest analyses, but finished last. It left the deal unsigned and made write attempts in a locked department instead of escalating. A weaker version of the same discipline problem appeared across the other four models. The results suggest that extensive preparation and sound analysis can still leave important gaps in execution.
Fairness note: K3 ran without an effort parameter (API default) while the others ran at xhigh.
A company you can watch
Firmulate’s simulated company has 13 synthetic employees and real money mechanics. It burns €105,000 per month against €2,300 in monthly recurring revenue, publishes a cash countdown, and has accumulated more than 680 self-learned playbook rules. Each workday is versioned. The company is live software that continues running; readers can learn about the experiment and its results at Firmulate and explore the standings and findings on its benchmark page.
The broader proposition is that companies should test AI agents on work that resembles their own before trusting them with it. Firmulate says enterprises can run the wargame against a read-only export of their business, with nothing written back to real systems. That creates a way to examine how an agent responds to pressure using relevant company information, while keeping the exercise separate from live operations.
Test before you trust
Kimi’s second-place finish, just behind gpt-5.6-sol, and ahead of three Western frontier models is a reminder that the league remains open. Across the experiment, models could all identify crises and resist manipulation, yet most did not sign the deal their analysis supported. For senior care leaders and other organizations weighing AI, the practical question is not simply which model sounds capable. It is whether the system can handle the records, boundaries, and follow-through the job requires. Picking one without testing it against your own work is a bet.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
