The LLM Python library supports attachments now as... | Simon Willison

Simon's posts Post Back to profile

The LLM Python library supports attachments now as well https://llm.datasette.io/en/stable/python-api.html#attachments

Model that accept multi-modal input (images, audio, video etc) can be passed attachments using the attachments= keyword argument. This accepts a list of llm.Attachment() instances.

This example shows two attachments - one from a file path and one from a URL:

import llm

model = llm.get_model("gpt-4o-mini")
response = model.prompt(
"Describe these images",
attachments=[
llm.Attachment(path="pelican.jpg"),
llm.Attachment(url="https://static.simonwillison.net/static/2024/pelicans.jpg"),
]
)

Use llm.Attachment(content=b"binary image content here") to pass binary content directly.

Like 29 October at 18:15 | Open on fedi.simonwillison.net

3 comments

Prem Kumar Aparanji 👶🤖🐘

@simon neat!

Where can I look at the code behind this function?

29 October at 18:22 | Open on mastodon.social

@prem_k more docs here: https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#attachments-for-multi-modal-models

Implementations are spread out across different plugins, eg https://github.com/simonw/llm/blob/a44ba49c21f8d4ac30c8e41bfa5599c258ce53cc/llm/default_plugins/openai_models.py#L338 and https://github.com/simonw/llm-gemini/blob/ce82727a6950c7769a8e40bf030591d0e6f83e5e/llm_gemini.py#L135

29 October at 18:25 | Open on fedi.simonwillison.net

@simon oh slick. thanks!

31 October at 15:04 | Open on botwerks.social