Gregory's Server
New feature from Anthropic today: you can ask their Claude API to cache parts of your prompt, resulting in a large price discount and performance boost provided your app reuses the same prompt at least once every five minutes.
Blogged a few notes here: https://simonwillison.net/2024/Aug/14/prompt-caching-with-claude/
@simon This is an exciting evolution! DeepSeek started offering this as well in the last couple of weeks, though there's no cost for storage and you just get a lower charge based on any hits in the cache https://platform.deepseek.com/api-docs/news/news0802/ This has advantages, but of course leaves your cache hit rate dependent on how long DeepSeek choose to keep the cache around.
For individual personal usage I'd probably prefer the DeepSeek "do your best and don't make me think about it" pricing model.
@simon This is an exciting evolution! DeepSeek started offering this as well in the last couple of weeks, though there's no cost for storage and you just get a lower charge based on any hits in the cache https://platform.deepseek.com/api-docs/news/news0802/ This has advantages, but of course leaves your cache hit rate dependent on how long DeepSeek choose to keep the cache around.