Gregory's Server@dbreunig I'm still frustrated that Anthropic don't release their tokenizer!
Gemini have an API endpoint for counting tokens but I think it needs an API key
|
4 comments
 @dbreunig yeah it's frustratingly difficult to compare tokenizers, which sure make price per million less directly comparable @dbreunig running a benchmark that processes a long essay and records the input token count for different models could be interesting though |
@simon Now that you mention it, I'm curious how different each platform is with tokens and how that might affect pricing (or just be a wash)