Gregory's Server@dbreunig yeah it's frustratingly difficult to compare tokenizers, which sure make price per million less directly comparable
|
2 comments
|
|
Top-level
 @dbreunig yeah it's frustratingly difficult to compare tokenizers, which sure make price per million less directly comparable 2 comments
|
@dbreunig running a benchmark that processes a long essay and records the input token count for different models could be interesting though