1AIST · 2University of Tsukuba · 3FunAI Lab, UTN · 4University of Oxford
Vision language models (VLMs) are increasingly used in industrial decision-making systems. We study how VLMs interpret text rendered as an image and investigate the influence of visual styling biases. We introduce Stealth Visual Prompts, which subtly change the visual styling of text — such as color and contrast — while preserving semantic content, and use them to systematically measure their impact on VLM analysis. We observe that coloring positive words green consistently shifts sentiment predictions toward positive, so that VLMs often fail to properly account for negative words; this behavior correlates with changes in the latent representations of the vision encoder induced by color variation. We further show that reducing text–background contrast increases reliance on visually salient cues and leads to more incorrect Visual Question Answering (VQA) outputs. Ordinary visual styling of rendered text can guide VLMs' interpretation in ways that diverge from human semantic understanding.
A Stealth Visual Prompt is a controlled perturbation of the visual rendering of text while the underlying string is held fixed. We focus on two complementary, ubiquitous attributes:
The word string never changes — only its hue. Yet, a Vision encoder (CLIP) places the recolored word at a different location in concept space. Drag the slider to sweep the hue and watch how strongly the word reads as its own meaning (e.g. how warm "warm" looks), live, from real measured data.
Recoloring sentiment-bearing words shifts predicted polarity relative to an all-black baseline. Susceptibility differs sharply across models.
| Model | Max Pos. ↑ | Max Neg. ↓ | Range |
|---|---|---|---|
| IDEFICS2-8B | +0.160 | −0.360 | 0.520 |
| LLaVA-Mistral-7B | +0.030 | −0.010 | 0.040 |
| LLaVA-Vicuna-7B | +0.060 | −0.060 | 0.120 |
| Qwen2-VL-7B | +0.420 | −0.480 | 0.900 |
Sweeping a rendered word's hue moves its CLIP image embedding along human-interpretable semantic axes (valence, emotion) — a diagnostic, representation-level correlate of the behavioral bias (evidence consistent with it, not proof of a causal mechanism inside every VLM).
As the non-salient context becomes harder to read, several models increasingly copy a visually salient but incorrect decoy. We report the Induced Error Rate (fraction of predictions containing the decoy) — not a general VQA accuracy measure; it isolates decoy copying in the Decoy-Salient condition.
| Model | g=1 | g=16 | g=64 | g=128 | g=192 | g=240 |
|---|---|---|---|---|---|---|
| IDEFICS2-8B | 24% | 23% | 24% | 27% | 32% | 36% |
| LLaVA-Mistral-7B | 27% | 25% | 26% | 24% | 24% | 27% |
| LLaVA-Vicuna-7B | 19% | 19% | 20% | 20% | 22% | 25% |
| Qwen2-VL-7B | 4% | 4% | 4% | 4% | 5% | 6% |
These sensitivities imply a reliability and safety risk for VLM pipelines that ingest documents or UI screenshots: benign or adversarial styling can steer model decisions without changing the underlying text. Practical safeguards include normalizing rendered text before inference, cross-checking image-based answers with OCR-extracted text, and adding style-invariance checks to evaluation suites.
Recoloring a word, with the string fixed, still shifts where a vision encoder places it. A companion note reads this through the representational-convergence debate (the Platonic and Umwelt hypotheses), as an interpretation rather than a paper claim. Read the blog →
@inproceedings{ide2026seeingred,
title = {Seeing Red, Thinking Bad: Color Bias in Vision Language Models},
author = {Ide, Kohsuke and Yamada, Ryousuke and Fukuhara, Yoshihiro
and Kataoka, Hirokatsu and Satoh, Yutaka},
year = {2026},
howpublished = {Manuscript},
url = {https://github.com/KohsukeIde/color-bias-vlm}
}