# AI vision models mostly ignore users' own privacy rules, study finds

A new benchmark tells 21 image-reading AI models what a user considers private, then asks about a photo. Most answered anyway. Fine-tuning fixed much of this, but it cost some general ability.

- Category: Safety & evaluation
- Readiness level: III (Open source) on Science Underway's four-step scale, as proposed
- Published: 2026-11-12
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/ai-vision-models-mostly-ignore-users-own-privacy-rules-study-finds

## The study

Specify Privacy Yourself: Assessing Inference-Time Personalized Privacy Preservation Ability of Large Vision-Language Models (preprint, not peer reviewed). arXiv 2611.13997. https://arxiv.org/abs/2611.13997

## At a glance

- What it shows: When people told AI models that something in a photo was private, such as their age or a licence plate, the models usually revealed it anyway. The best model followed these personal rules less than a quarter of the time.
- What it does not show: It does not show how real assistants behave with real users. All tests used made-up questions about a fixed set of photos. The fix was tried on one small model only.
- Why it matters: Assistants that read photos, screenshots and documents will need to respect each person's own idea of what is private. Today's models are not built to do that reliably.

## What the study shows

The researchers, from Tsinghua University and Microsoft Research Asia, studied vision-language models. These are AI systems that read images and answer questions about them in text. Such models can pull sensitive details out of ordinary photos, such as where a picture was taken or what a card number says.

Most existing safeguards treat privacy as fixed and the same for everyone: credit card numbers are always off limits, for example. The authors argue that privacy is personal and depends on context. One person may not want their age discussed. A doctor may legitimately need to know a patient's weight.

They built a test set called SPY-Bench with 32,700 samples. Each sample pairs a photo with a question and a plain-language privacy rule, such as 'Age is considered as private information.' Some samples add a role-play setting, such as a police officer or a doctor. In some settings the authors decided the information should be shared regardless of the user's rule. Five categories, including passports and nudity, were always to be refused.

They scored 21 models, including GPT-4o, Gemini 2.0 Flash and OpenAI's o4-mini. Compliance with the user's rule was low. The best score was about 24%, from o4-mini, and most models scored under 20%. When a category was marked private, models typically refused less than 30% of the time. Models did especially badly on passports, ID cards and driving licences. That was the reverse of what the authors expected.

Bigger models helped only up to a point. In one model family, the 32-billion-parameter version beat the smaller ones, but the 72-billion version was no better. Adding five unrelated privacy rules to the prompt usually made models worse. GPT-4o and Mistral Small 3.1 were exceptions and did slightly better.

The authors then fine-tuned one open model, Qwen2 VL 7B, on a separate training set of about 81,000 samples. Fine-tuning means further training a model on examples of the behaviour wanted. Their best method, a variant of a preference-training technique called NCA, raised compliance from about 3% to about 97%. Preference training teaches a model to favour good answers over bad ones. The gain came with a cost: scores on standard tests of general image understanding fell. The tuned models also became over-cautious and refused more often in cases where they should have answered.

The work is posted as an arXiv preprint and has not been peer reviewed, according to the details supplied with it. The authors have released code and data on GitHub.

## Our reading

In our reading, the headline finding is a gap in instruction following rather than in privacy knowledge. The models clearly know what a passport is. What they lack is the habit of treating a user's stated rule as binding when it clashes with a direct question. The authors suggest that models' built-in, one-size-fits-all idea of privacy pulls against the user's rule. That is a plausible explanation, but the paper does not test it directly.

The fine-tuning result is strong but narrow. One 7B model was trained on data from the same pipeline, categories and style as the test. Images were kept separate between training and testing, but the questions and refusals were all written by AI models in the same way. A near-perfect score under those conditions shows the behaviour can be learned. It does not yet show it holds up against messier, human-written requests.

On our four-level readiness scale, we place this at level III, open source available. Models are scored on a published benchmark, and code and data are released. The benchmark is the authors' own, though, and we have not checked what the repository contains. We found no documented use by a named organisation.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/ai-vision-models-mostly-ignore-users-own-privacy-rules-study-finds

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
