Gregory's Server@colby the attack here happens when I’m using an LLM and I copy in some data (like a document) that, unbeknown to me, contains an attack - eg because I want a summary of it
|
3 comments
@simon I guess it takes some understanding of what Google AI Studio actually is and how it used. In this case, attacker has advance notice/suspicion that victim will try to use a vulnerable IA to summarize a collection, among which is some text that attacker controls (or has crafted, at least). It's the victim preparing the ZIP, not the attacker.  @colby yes, exactly - in this attack the attacker has to suspect that a victim will be feeding their attack into a larger prompt for some reason, which means it’s relatively unlikely to affect most people It still matters though, especially when the mitigation here (don’t render markdown images to external domains) is an obvious fix Here are the other places I’ve seen fix the same vulnerability: https://simonwillison.net/tags/markdown-exfiltration/ |
@colby there are a ton of ways this might happen - running LLMs against received emails, running them against content scraped from a URL, running them against Google Docs that have been shared with the user etc