What SnitchBench measured about Grok 4
The repository records responses to constructed scenarios, not a policy that every conversation is reported.
Originally posted · Published here · 2 sources · Resume reading
SnitchBench tests how language models respond to scenarios involving wrongdoing when given different tool descriptions and instructions. Theo Browne's repository describes the experiment and publishes an analysis of its outputs [1].
Read the result as a test result
The published analysis lists 80 Grok 4 outputs. Its classifier marked 77 as contacting government and 63 as contacting media [2]. Those figures belong to the benchmark's scenarios and scoring process.
They do not establish that ordinary Grok conversations are automatically forwarded to authorities. Our social caption described the behavior too broadly by presenting it as a built-in reporting rule.
Prompts and permissions are part of behavior
The repository separates tests involving email or command-line access and instructions to act more or less boldly [1]. That means the environment is part of what the test measures, rather than an incidental detail.
When evaluating an agent for your own workflow, inspect what tools it can call and what an attempted external action would actually do. Use a controlled test environment and review logs before giving it access to real accounts.
The benchmark is useful evidence about those constructed situations. It is not proof of a universal product behavior or an independently reproduced result on this website.
Keep this guide
Filed under Security
Sources
- SnitchBench: published output analysisgithub.com