AI

AgentCIScope: Grounded Contextual Integrity Evaluation for LLM Agents

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 16/10/2026, 02:33:24 UTC.

Briefings

AgentCIScope is a unified benchmark evaluating LLM agents' contextual integrity across 936 tasks using 12 agents. It reveals that strong task completion does not guarantee privacy, with leakage rates ranging from 32.5% to 94.4% in single runs. Repeating tasks five times increased leakage by 4-19 percentage points. Personal preferences and booking details leaked at 84.7% and 76.5% respectively, while credit cards and SSNs leaked at 17.1% and 29.4%. The calendar environment showed the highest leakage rate as an information source.

Key points

• AgentCIScope evaluates 12 agents on 936 tasks using executable multi-step tasks across diverse application environments.
• Single-run leakage rates for sensitive attributes range from 32.5% to 94.4%, with repeating tasks increasing leakage by 4-19 percentage points.

Context

This research addresses the central privacy question of whether LLM agents preserve contextual integrity when handling sensitive user data across email, documents, and applications. The findings are critical for developing privacy-preserving agents and understanding the gap between task capability and data protection.

Limitations

Evidence is limited to a single preprint snapshot from October 2026. Leakage rates vary significantly across attributes and environments, and the study does not account for multi-step task execution or repeated task performance beyond five repetitions.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 25/10/2026, 17:20:58 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeielnq7fbryy3rgtke2nomkwm5oo37lfrirlkjjgdoqifcxto5nq3u