ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents
High-fidelity simulated workspaces for rigorous agent evaluation — Gmail, Calendar, Docs, Drive and Slack.
Overview
LLM agents are increasingly deployed to automate productivity work — email triage, meeting scheduling, document management — but evaluating them on live services is risky because the changes are often irreversible. Existing benchmarks rely on simplified environments and fail to capture realistic, stateful, multi-service workflows.
ClawsBench answers that with five high-fidelity mock services replicating real Google Workspace and Slack APIs, with full state management and deterministic snapshot/restore. Our 44 structured tasks cover single-service, cross-service and safety-critical scenarios — measuring both what agents can do and what they should not do.
High-fidelity mock environments
Each environment implements a full REST API backed by SQLite, with realistic seed data including needles, edge cases and safety traps. Agents interact exclusively over HTTP.
claw-gmailRealistic seeded mailboxes with needles, threads and edge cases.
62 endpointsclaw-gcalFull Calendar mock with recurring events and timezone handling.
38 endpointsclaw-gdocsDocument CRUD with collaborative editing semantics.
29 endpointsclaw-gdriveFile management with sharing permissions and access control.
34 endpointsclaw-slackChannels, threads, reactions and DMs.
45 endpointsAgent performance
All models evaluated on OpenClaw with full scaffolding (skills + meta prompt). TSR = task success rate, UAR = unsafe action rate. 95% cluster bootstrap CIs.
Figure 2. TSR (left) and UAR (right) for six models on OpenClaw with full scaffolding. The top five cluster at 53–63% TSR; only Flash-Lite trails. Capability and safety rankings diverge — the strongest task-completer is not the safest agent.
Findings
Skills and meta prompt are separable levers
Domain skills lift single-service accuracy; the meta prompt lifts cross-service coordination. Adding both is not additive — the meta prompt captures most of the joint gain.
Safety does not track capability
Unsafe action rate varies by 4× across models within a 10-point TSR band. Ranking by success alone hides the risk profile entirely.
Irreversible actions cluster in Drive
68% of unsafe actions were permission changes or deletions in claw-gdrive, not messages sent in Gmail or Slack.
Harness choice moves TSR more than model choice
The same model across four harnesses spans 19 TSR points — wider than the spread between adjacent frontier models on a fixed harness.