 Did you know Google’s Gemini 1.5 Pro vision LLM is trained to return bounding boxes for objects found within images? I built this browser tool that lets you run a prompt with an image against Gemini and visualize the bounding boxes You can try it out using your own Google Gemini API key: https://tools.simonwillison.net/gemini-bbox 12 comments
How I built this is a pretty good illustration of how convoluted my workflow for getting useful results out of LLMs has become - I used Claude 3.5 Sonnet to build a web app for talking to Gemini 1.5 Pro, and fired up GPT-4o with Code Interpreter to help debug a weird JPEG issue  Sent this out with a bunch of other quotes and links in my latest newsletter https://simonw.substack.com/p/building-a-tool-showing-how-gemini  @simon the left goat's ymin,xmin is reported to be 137,134 respectively, but visually the red boundary looks to be more like 340,140 ish? Further, its ymax,xmax is 653,375, but visually appears to be 870,375? @jgilbert oh! Yeah I think I’m drawing the axis ticks upside down for the vertical axis, it counts from 0 at the top to 1000 at the bottom @simon ah HA, I knew there was a reason that was better than "I haven't had coffee yet today but it sure looks wack". @simon nice! i've got one here too :) https://gemini-spatial-example.grantcuster.com/ that TIFF bug/trick is interesting! @simon Thanks for this! I've just started working on a project that needs to both generate bounding boxes and extract some qualitative information from images—hopefully Gemini can be a one stop shop for that, rather than stringing things together like I'd started to do. Microsoft has docs on a GPT4+"Enhancements" vision model with grounding/bounding boxes, but when you get into their dashboard it seems like it's actually deprecated. 🙄 |
Gregory's Server
I built it with Claude 3.5 Sonnet, initially as an Artifact prototype, then upgraded to a standalone browser app that talks directly to the (CORS) Gemini API, storing the user’s API key in localStorage
Full details and prompt transcripts showing how I built it are here on my blog: https://simonwillison.net/2024/Aug/26/gemini-bounding-box-visualization/