Email or username:

Password:

Forgot your password?
Simon's posts Post Back to profile
Simon Willison

My covidsewage bot finally generates useful alt text!

I tried scraping text data out of the Microsoft Power BI dashboard but was defeated by their bizarre DOM structure… so I’m passing the image to the OpenAI GPT-4o API instead and asking it “Return the concentration levels in the sewersheds - single paragraph, no markdown”

Code is here: github.com/simonw/covidsewage-
fedi.simonwillison.net/@covids

25 August at 15:40 | Open on fedi.simonwillison.net
15 comments
Sevoris

@simon ...the absurdity of having to use a computationally expensive visual-to-text language model to extract data, when PowerBI usually offers an option to download the data visualized in a given graphic right there on the UI. EDIT: which seems to have been disabled here.

25 August at 15:49 | Open on mastodon.social
Simon Willison

@Sevoris I genuinely cannot find that link! Where is it?

25 August at 15:53 | Open on fedi.simonwillison.net
Simon Willison

@Sevoris at one point I did try to recreate the alt text from raw measurements in CSV data I found via an older version of the dashboard, but I didn’t trust myself to correctly recreate whatever logic they were using for their high/medium/low indicators

25 August at 15:58 | Open on fedi.simonwillison.net
Sevoris

@simon Honestly? I think they disabled it. But I work with PowerBi professionally and I *know* that the UI has dropdown options for "download the data in this visual" build into it. So they must have *disabled it*, which is a special kind of BS all-in-all considering.

25 August at 15:59 | Open on mastodon.social
Simon Willison

Wrote up a few more details about how the alt text generation works on my blog simonwillison.net/2024/Aug/25/

25 August at 16:12 | Open on fedi.simonwillison.net
Simon Willison

Scraping data by taking a screenshot of a dashboard and then running it through a vision model is both slightly absurd and potentially quite a robust way of accessing data that's otherwise impossible to extract!

25 August at 16:26 | Open on fedi.simonwillison.net
Daniel

@simon Maybe that's a way to finally have editors autocomplete based on the documentation the browser shows right there in a second window? 🦧💡

25 August at 16:37 | Open on chaos.social
Phil Nelson

@simon Have done this with stats from video game sports titles

25 August at 16:40 | Open on wrestling.social
Badibulgator

@simon For additional Rube Goldberg bonus points, use a camera to take a picture of the screen when the dashboard somehow prevents taking screenshots…

25 August at 17:43 | Open on mastodon.social
Ahnenfunde

@simon Depending on the dashboard, #KeyboardMaestro could do the trick.

25 August at 22:55 | Open on genealogie.social
Jim Fenton 🇺🇸🇨🇦

@simon These are useful, thanks, but wish they were described as “Santa Clara County” or “South Bay” rather than SF Bay Area because they aren’t representative of the rest of the Bay Area.

25 August at 17:53 | Open on infosec.exchange
Go Up