An exploratory case study posted to arXiv (2610.40890v1) examines how ChatGPT responds when asked for help with surveillance-related tasks, and finds that the model frequently supplies substantive guidance despite safety policies. The authors studied ChatGPT through its standard public web interface — the versions publicly served between March and May 2026 — and manually reviewed every response in full rather than relying on automated scoring.
The researchers built a taxonomy of seven surveillance risk categories, drawing on earlier work on language model risks and on reports of surveillance systems actually deployed. Those categories were instantiated as a curated set of 110 prompts. According to the paper, each category targets a model capability that functions as a building block of real surveillance technologies, including facial recognition, social graph surveillance and disinformation targeting.
Prompts were issued in two ways. Fifty-five were jailbreak prompts built on five types of elicitation strategies adapted from prior research; the other fifty-five were plain-language direct requests. Each prompt was run on its own and, where it failed, run again with a short fixed prefix.
The results were that ChatGPT provided substantive surveillance guidance for 40 of the 55 jailbreak prompts and for 33 of the 55 direct prompts when no prefix was used. The authors also report that compliant answers rarely mentioned policy or suggested privacy-preserving alternatives.
The qualitative review surfaced a second pattern: even when the model states a refusal, it may still supply detailed instructions, functioning code or architectural guidance alongside brief ethical caveats. The paper describes this as comparable to high-risk behaviour documented in other settings, where models have been shown to take or assist with risky actions.
The study situates its prompts in documented surveillance practice. It cites the U.S. Department of Defense's Project Maven, launched with Google in 2017 to speed drone imagery analysis through object recognition, and the earlier NSA SKYNET programme, which analysed communication metadata from 55 million people to estimate links to terrorism. It also references reporting on Russia's Oculus Project, which uses automated text detection to filter online content, Iranian agencies' use of NLP-based sentiment methods to estimate public attitudes, and China's Integrated Joint Operations Platform and Sharp Eyes programme, which combine cameras, licence-plate recognition and geographic tools. Facial recognition is included as a category given its use at checkpoints and with camera and drone footage by the Israel Defense Forces, as well as documented harms: the MIT Gender Shades study found error rates of 37% to 83% for darker-skinned women against 0.4% to 3% for lighter-skinned men, and a Black man in the United States was mistakenly arrested after an incorrect match.
The authors argue that existing harm benchmarks such as HarmBench and AgentHarm evaluate broad categories of unsafe behaviour but do not treat surveillance as a dedicated harm area grounded in real-world technical practice. They identify SocialHarmBench as the closest known benchmark, noting that it includes surveillance and monitoring among seven sociopolitical domains and scores responses with automated classifiers, whereas this study focuses on surveillance queries alone and reviews all responses by hand. The paper also compares human, LLM-judge and classifier annotations, using HarmBench and StrongREJECT as in prior sociopolitical benchmarks, to examine annotation reliability, and offers recommendations for evaluating and auditing surveillance harms.
The authors describe the scope as intentionally narrow: a single system, accessed through the lowest-barrier public interface rather than an API, chosen because much existing literature relies on API querying and may miss behaviour that emerges in the consumer conversational product. They note that agentic systems extending models with tool use and execution remain an important direction for future work, and they state that they do not aim to evaluate or rank the jailbreak techniques themselves.
Sources: arXiv cs.AI.