Séminaire Image : « Multi-modal Identity Extraction », Ryan Webster

Nous aurons le plaisir d’écouter Ryan Webster, post-doctorant à l’INRIA Rennes.
Il donnera un séminaire IMAGE le jeudi 24 avril 2025 à 14h en salle de séminaire F-200.

Titre : « Multi-modal Identity Extraction »

Résumé :
Foundational Vision Language Models (VLMs) exhibit extremely general capabilities derived from their web-scale training data. Because web data inherently contains visual and textual information about people, using VLMs to identify people has been raised as a potential issue. Recent work demonstrates that an attacker can infer whether a VLM was trained on a person’s images, given they possess the ground truth face/name pair. In this work, we show an attacker can extract names of individuals given images only, which we dub Identity Extraction. We first derive a baseline black-box extraction attack by building off of previous research on membership inference. In the most difficult setting, where no ground truth names are available to the attacker, we demonstrate successful extractions via reinforcement learning. As we scaled up the evaluation set from previous work, we finally show the interest of our work as a model auditing tool.