@simon Nice! You should drop a tokenizer in there for people.

Like 17 October at 14:02 | Wall-to-wall | Open on note.computer

5 comments

@dbreunig I'm still frustrated that Anthropic don't release their tokenizer!

Gemini have an API endpoint for counting tokens but I think it needs an API key

17 October at 14:15 | Open on fedi.simonwillison.net

@simon Now that you mention it, I'm curious how different each platform is with tokens and how that might affect pricing (or just be a wash)

17 October at 14:16 | Open on note.computer

@dbreunig yeah it's frustratingly difficult to compare tokenizers, which sure make price per million less directly comparable

17 October at 14:27 | Open on fedi.simonwillison.net

@dbreunig running a benchmark that processes a long essay and records the input token count for different models could be interesting though

17 October at 14:28 | Open on fedi.simonwillison.net

17 October at 15:54 | Open on note.computer