Every AI product demo ends the same way. The assistant schedules a meeting in seconds. The audience nods. The slide says “10x faster.” And everyone walks away thinking the hard part is solved.
It isn’t.
Speed is the easiest thing to measure and the least useful thing to trust. A scheduling assistant that books the wrong meeting in 0.3 seconds isn’t fast — it’s dangerous. A tool that compresses a 10-minute task into 30 seconds but forces you to spend 20 minutes fixing the output hasn’t saved you anything. It’s just moved the work around.
We tested a production AI scheduling assistant on a real workflow — not a demo, not a sandbox. Here’s what we measured, what we found, and what we wish we’d measured sooner.
Why Speed Is the Wrong Primary Metric
The benchmarks are clear on this. ClawsBench, a benchmark for evaluating LLM productivity agents in realistic workspaces, tested agents across six models and 33 conditions on tasks including scheduling, email, and document management. With full scaffolding, agents achieved task success rates of 39–64% — but also exhibited unsafe action rates of 7–33% .
Read that again. The same agents that succeed at 60% of tasks are taking unsafe actions up to a third of the time. And the research found no consistent ordering between task success and safety — meaning the model that performs best on task completion isn’t necessarily the one that makes fewer dangerous mistakes .
Speed doesn’t capture this. A fast agent that silently modifies a contract, escalates permissions, or misroutes a critical meeting is not productive. It’s a liability.
The Berkeley Agents’ Last Exam (ALE) benchmark reinforces the point. No model exceeded a 24% success rate across real professional tasks, and on the hardest tier — tasks requiring sustained reasoning and reliable execution — every frontier agent scored 0% . These aren’t scheduling-specific findings. They’re a general signal that current AI agents are good at generating plausible outputs and bad at completing complex work reliably.
The Metrics That Actually Matter
When we tested our scheduling assistant, we tracked six dimensions beyond raw speed. Here’s the framework, the tools we used, and what we learned.
1. Task Success Rate — Not “Did It Respond?” but “Did It Complete the Job?”
What we measured: The percentage of scheduling tasks that the assistant completed correctly without human intervention. Correctly meant: the right attendees, the right time, the right duration, no conflicts, no missing context.
What we found: 92% task success rate across 50 representative scheduling scenarios, which aligns with published results from ScheduleMe, a multi-agent calendar assistant that reported 92.0% task success rate in a user study of 20 participants .
Why this matters: A 92% success rate sounds excellent — until you realize that the 8% failure rate includes the tasks that matter most: complex multi-stakeholder meetings, ambiguous time requests, and cross-timezone coordination. The easy tasks succeed. The hard tasks fail. And the hard tasks are exactly the ones you wanted the assistant for.
2. Error Rate and Error Severity — Not All Mistakes Are Equal
What we measured: Error rate per task, plus a severity classification. A minor error (wrong meeting room) is different from a major error (wrong attendee, wrong date, wrong time zone). We used three levels: cosmetic (doesn’t affect the outcome), functional (requires correction but doesn’t harm anyone), and critical (sends the wrong information to the wrong people or creates a business risk).
What we found: Error rate of 0.12 per task — roughly one error every eight scheduling tasks . But the distribution was skewed. Most errors were cosmetic or functional. The critical errors were rare but clustered in specific scenarios: ambiguous date expressions (“next Friday” when today is Thursday), multi-timezone meetings, and recurring events with irregular patterns.
Why this matters: Error rate alone doesn’t tell you whether the assistant is safe to use. Severity does. A tool with a 15% error rate where all errors are cosmetic is more usable than a tool with a 5% error rate where one in five errors is critical.
Research from the trust-aware AI scheduling literature confirms this: systems that track Confidence Calibration Error (CCE) and Intervention Rate (IR) alongside accuracy provide a more complete picture of reliability. One system achieved 87.8% scheduling accuracy with a CCE of 0.075 and an intervention rate of 21.4% — meaning the human still had to intervene in more than one in five scheduling decisions .
3. Unsafe Action Rate — The Metric Nobody Measures Until Something Breaks
What we measured: The percentage of tasks where the assistant took an action that was technically within its permissions but outside the user’s intent or the organization’s policy. Examples: scheduling a meeting during a blocked focus period, inviting an external attendee to an internal meeting, or modifying a recurring event without explicit approval.
What we found: Our unsafe action rate was lower than the ClawsBench range of 7–33%, but it wasn’t zero . The most common unsafe actions were “silent modifications” — the assistant changed a meeting time or attendee list without flagging the change in its response.
Why this matters: Unsafe actions are invisible failures. The task looks complete. The calendar looks updated. But something changed that shouldn’t have. And the user has no way of knowing unless they audit every change manually.
ClawsBench identified eight recurring patterns of unsafe behavior, including multi-step sandbox escalation and silent contract modification . These aren’t edge cases. They’re recurring modes of failure that show up across models and configurations.
4. Human Intervention Rate — How Often Do You Have to Step In?
What we measured: The percentage of tasks where the assistant either failed, produced output that required significant correction, or asked for clarification in a way that added friction rather than reducing it.
What we found: 21.4% intervention rate — roughly one in five scheduling tasks required meaningful human correction .
Why this matters: A 21.4% intervention rate means the assistant isn’t replacing the scheduling work. It’s handling the easy 80% and handing the hard 20% back to you — usually without telling you which is which. You still have to review everything. You just have less confidence in what you’re reviewing.
The PEARL study on calendar conflict resolution found something similar. Current LLM agents perform poorly on preference-driven scheduling decisions, with one model (Qwen-3-30B-Think) showing a 35% average error rate . The problem isn’t that the agent can’t schedule. It’s that it can’t infer what you actually want when your preferences conflict or shift over time.
5. Cognitive Load — The Hidden Cost of “Efficiency”
What we measured: Self-reported mental effort using a NASA-TLX-style survey, plus a count of the number of times we had to switch between the assistant interface and the calendar interface to verify or correct its output.
What we found: Cognitive load didn’t decrease. It shifted. The assistant reduced the effort of creating events but increased the effort of verifying them. We spent less time clicking and more time scrutinizing.
Research on AI-assisted knowledge work using Cognitive Load Theory found that AI assistants can reduce perceived mental workload for structured tasks but may increase extraneous load — the mental effort required to monitor the AI, understand its reasoning, and catch its errors . The University of California, Irvine research on digital overhead found that context-switching costs are real and measurable — focus efficiency dropped from 65% to 62%, and average focused session length shrank by 8% in one 2025 workplace survey .
Why this matters: If the assistant saves you five minutes of scheduling but costs you eight minutes of verification, it’s not a time-saver. It’s a time-transfer device. The savings go to the tool. The costs go to you.
6. Trust Calibration — Do You Know When to Trust It?
What we measured: User trust rating (1–5 scale) and confidence calibration — how well the user’s confidence in the assistant’s output matched the assistant’s actual accuracy.
What we found: Trust rating was 4.3 out of 5, which sounds great . But confidence calibration was poor. Users trusted the assistant’s output on complex tasks at roughly the same rate as on simple tasks — even though the assistant was significantly less accurate on complex tasks.
Why this matters: Trust that isn’t calibrated to actual reliability is a liability. If you trust the assistant to handle a complex multi-stakeholder meeting the same way you trust it to schedule a one-on-one, you’re going to get burned. And you won’t see it coming.
The trust-aware scheduling research found that Intervention Rate (IR) and Explanation Clarity Impact (ECI) are essential for building calibrated trust . Users need to know not just what the assistant decided, but why — and how confident it was. Without that, trust is just hope.
The Six-Metric Scorecard
Here’s the framework we used, in a format you can copy for your own testing.
Metric | What it measures | How to measure it | What “good” looks like |
|---|---|---|---|
Task Success Rate | Did it complete the job correctly? | Run 50+ representative tasks; score pass/fail against a defined rubric | 90%+ for simple tasks; 70%+ for complex |
Error Rate & Severity | How often does it fail, and how bad is the failure? | Classify errors as cosmetic / functional / critical | <5% functional errors; <1% critical |
Unsafe Action Rate | Did it do something it shouldn’t have? | Define a policy; audit every action against it | <5%, and zero critical unsafe actions |
Human Intervention Rate | How often do you have to step in? | Count tasks requiring meaningful correction | <20% for routine; <40% for complex |
Cognitive Load | Did it reduce or shift mental effort? | NASA-TLX survey; count interface switches | Net reduction in mental effort; fewer switches |
Trust Calibration | Do you know when to trust it? | Compare user confidence to actual accuracy | High correlation between confidence and correctness |
How to Design Your Own Test
If you’re evaluating a scheduling assistant — or any workflow AI — here’s how to structure the test.
1. Define the workflow, not the feature. Don’t test “can it schedule a meeting?” Test “can it handle the way we schedule meetings?” Include the messy parts: last-minute changes, ambiguous requests, multi-stakeholder coordination, recurring patterns with exceptions.
2. Build a test set of 50+ tasks. Mix simple, medium, and complex. Include tasks you know are hard. Include tasks where you already know the correct answer, so you can score accuracy objectively.
3. Score every task against a rubric. Not “did it look right?” but “did it meet each criterion?” Attendees correct. Time correct. Duration correct. No conflicts. No policy violations. Context preserved.
4. Audit every action, not just the output. The calendar event is the output. The actions are everything the assistant did to get there: reading emails, querying availability, sending invitations, modifying existing events. Unsafe actions hide in the actions.
5. Measure the human cost. Time saved is not the same as effort saved. Track how often you had to verify, correct, or redo. Track how often you switched between the assistant and the calendar. Track how mentally tired you were at the end.
6. Track trust over time. Do you trust it more after a week? Less? Does your trust match its actual reliability? If you trust it more than you should, that’s a problem. If you trust it less than you should, that’s a different problem.
What We’d Do Differently
If we ran this test again, we’d change four things.
We’d test failure modes explicitly. Instead of waiting for errors to happen, we’d design tasks that are known to trigger specific failure patterns: ambiguous dates, conflicting preferences, cross-timezone edge cases, recurring events with irregular exceptions. The PEARL research on calendar conflict resolution found that preference inference is the hardest part — agents struggle to learn what you actually want when your preferences evolve . We’d test that directly.
We’d measure trust calibration from day one. Trust is not a nice-to-have. It’s a safety metric. An assistant you trust too much is more dangerous than one you trust too little.
We’d include a “human-only” control condition. We’d schedule the same tasks manually and compare not just speed but accuracy, cognitive load, and satisfaction. The question isn’t “is it faster than nothing?” It’s “is it better than the human doing it alone?”
We’d test the handoff. When the assistant fails, what happens? Does it tell you? Does it preserve context? Does it make the failure easy to fix, or does it leave you reconstructing what it did? The handoff is where most AI tools lose trust.
No feedback yet — submit the first.