Gregory's Server@simon I guess I don't understand how this is an attack. The malicious prompt came from the attacker, but so did everything else. So the attacker already has access to the "exfiltrated" data, right?
Or is there some missing context here?
|
4 comments
 @colby there are a ton of ways this might happen - running LLMs against received emails, running them against content scraped from a URL, running them against Google Docs that have been shared with the user etc @simon I guess it takes some understanding of what Google AI Studio actually is and how it used. In this case, attacker has advance notice/suspicion that victim will try to use a vulnerable IA to summarize a collection, among which is some text that attacker controls (or has crafted, at least). It's the victim preparing the ZIP, not the attacker. @colby yes, exactly - in this attack the attacker has to suspect that a victim will be feeding their attack into a larger prompt for some reason, which means it’s relatively unlikely to affect most people It still matters though, especially when the mitigation here (don’t render markdown images to external domains) is an obvious fix Here are the other places I’ve seen fix the same vulnerability: https://simonwillison.net/tags/markdown-exfiltration/ |
@colby the attack here happens when I’m using an LLM and I copy in some data (like a document) that, unbeknown to me, contains an attack - eg because I want a summary of it