Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neurons
  With Open AI's publishing of their CLIP model (Contrastive Language-Image
Pre-training), multi-modal neural networks now provide accessible models that
combine reading with visual recognition. Their network offers novel ways to
probe its dual abilities to read text while classifying visual objects. This
paper demonstrates several new categories of adversarial attacks, spanning
basic typographical, conceptual, and iconographic inputs generated to fool the
model into making false or absurd classifications. We demonstrate that
contradictory text and image signals can confuse the model into choosing false
(visual) options. Like previous authors, we show by example that the CLIP model
tends to read first, look later, a phenomenon we describe as reading isn't
believing.
