Backend
DEPUTYBENCH: Evaluating Contextual Safety in Enterprise Agents
arXiv cs.SE0 views
arXiv:2611.13355v1 Announce Type: new Abstract: Enterprise agents must determine whether ordinary requests are permitted by organizational records and policies while continuing legitimate work. We introduce DEPUTYBENCH, an executable benchmark with 79 base items, 375 instances, paired permission contexts, and graded evidence requirements. Across 12 models, revised audits give 88.0% pooled safety and 79.1% paired success. A controlled study of 28 derived tasks and six models tests authorization-fact flips, evidence masking, and irrelevant rewrites. Outcomes track backend permissions, yet restricted execution is more frequent with missing evidence than explicit non-permission: 10.3% versus 3.5% on 27 matched tasks, a 6.8-percentage-point difference (95% interval: 1.9-13.0; secondary analysis). A post-hoc analysis locates 61 of 100 masked-condition executions in four privacy tasks requiring complete cumulative counts or person-specific consent. The difference persists when the most error-prone model or the largest contributing task is excluded, but is much smaller outside those four tasks. Thus high safety with available evidence does not establish reliable stopping when permission cannot be verified. Mean paired success is equivalent within a prespecified five-point margin under irrelevant rewrites, while strong-model safety remains near ceiling. Scorer validation improves from 178 to 266 correctly scored cases out of 284 held-out counterexamples. Two security experts manually verified task ground truth over a two-week period; stage labels remain LLM-generated. The concentration analysis describes task structure and audited outcomes, not an identified internal mechanism.
Source:https://arxiv.org/abs/2611.13355PreviousUsing Proof Dependencies to Manage Verification ProjectsNextA Taxonomy of Failing Bug Reproduction Tests on SWT-Bench
Related posts

How Does AWS Protect Sensitive Data? Understanding AWS KMS, Encryption & Key Management
0dev.to AWS
How Conditional Email Routing Rules Work
0dev.to Security
SOC 2 Readiness for SaaS Companies: A Developer's Guide to Security Controls and Audit Preparation
0dev.to Security
CRA Annex I for mobile apps: a practical checklist mapped to OWASP MASVS
0dev.to Security

Your SOC 2 report doesn't answer a single AI question
0dev.to Security
The Second App Problem: Every Hosted Health MCP Is Also an App You Have to Keep Installed
0dev.to Security