 Video scraping: extracting JSON data from a 35 second screen capture for less than 1/10th of a cent https://simonwillison.net/2024/Oct/17/video-scraping/ I needed to extract information from a dozen emails in my inbox... so I ran a screen capture tool, clicked through each of them in turn and then got Google's Gemini 1.5 Flash multi-modal LLM to extract (correct, I checked it) JSON data from that 35 second video. Total cost for 11,018 tokens: $0.00082635 25 comments
Here's another example of multi-modal vision LLM usage: I collected the prices for the different preset models by dumping screenshots of their pricing pages directly into the Claude conversation Full transcript here: https://gist.github.com/simonw/6b684b5f7d75fb82034fc963cc487530  @simon Leaks show that the ChatGPT Mac and/or web app are going to get screen sharing soon via the Realtime API. Seems like this is the next frontier: dumping the whole personal computing experience into models.  @simon I'm a little confused by the OCR part - is that just some unrelated (but obviously useful) service tacked on the front, or is there some way LLMs are involved in the character recognition itself? (15 years ago OCR quality was related to text modelling, there was some interest in using our geotagger to do feedback for OCR of map labels, but I haven't dug into that space in a while) @simon Do you use https://openrouter.ai to connect to different models, or do you use each service with it's own api and cost traccking? @hoffmann I mostly use the service APIs directly - I have an OpenRouter account too but I like to stay deeply familiar with all of the different APIs as part of developing my https://llm.datasette.io tool  @dbreunig I'm still frustrated that Anthropic don't release their tokenizer! Gemini have an API endpoint for counting tokens but I think it needs an API key @simon Now that you mention it, I'm curious how different each platform is with tokens and how that might affect pricing (or just be a wash) @dbreunig yeah it's frustratingly difficult to compare tokenizers, which sure make price per million less directly comparable @dbreunig running a benchmark that processes a long essay and records the input token count for different models could be interesting though @simon Is it also possible to calculate how much energy these things use, and some comparisons of what that's equivalent to? I hear that AI is energy intensive but I have zero concept of what that means in reality for a single "thing" like this. @philgyford if that's possible I haven't seen anyone do it yet - the industry don't seem to want to talk specifics GPUs apparently draw a lot more power when they are actively computing than when they are idle, so there's an energy cost associated with running a prompt that wouldn't exist if the hardware was turned on but not doing anything @simon the fact that this works as well as it did kinda blows my mind — what an absolutely _wild_ pattern for data scraping.
[DATA EXPUNGED]
@th0ma5 I checked it all, didn't take long (I watched the 35s video and scanned the JSON) - it was exactly correct @superFelix5000 @th0ma5 right - the single hardest thing about learning to productively work with LLMs is figuring out how to get useful results out of inherently unreliable technology @simon @superFelix5000 Sorry my previous reply I didn't realize that you coded. The manual setup and such that you did is the hard part right? Extracting frames and OCRing them can be done with just command line tools. I think your documentation is great, but it doesn't feel like a net gain to me. |
Gregory's Server
Bonus from that post: I got fed up of calculating token prices by hand, so I had Claude Artifacts spin up this pricing calculator tool with presets for all of the major models https://tools.simonwillison.net/llm-prices