Interesting notes from Paul Gauthier on how asking... | Simon Willison

Simon's posts Post Back to profile

Interesting notes from Paul Gauthier on how asking an LLM to return code wrapped in a JSON object can result in a quality reduction compared to asking for that code in a less complex format such as fenced code Markdown blocks https://aider.chat/2024/08/14/code-in-json.html

(Cross-posted from my blog: https://simonwillison.net/2024/Aug/16/llms-are-bad-at-returning-code-in-json/)

Coding skill by model and code wrapping strategy - four models, each showing their pass rate % average of five runs. Claude 3.5 Sonnet gets 60.5% with Markdown, 54.1% with JSON. DeepSeek-Coder V2 0724 gets 60.6% with Markdown, 51.1% with JSON. GPT-4o-2024-05-13 gets 60.0% with Markdown, 59.6% with JSON. GPT-4o-2024-08-06 gets 60.8% with Markdown, 57.6% with JSON, and 56.9% with JSON (strict). Markdown consistently performs better than JSON across all models.

Like 16 August at 17:22 | Open on fedi.simonwillison.net

1 comment

Martin Owens :inkscape:

Format encoding robs the network of capacity I said. Better to write encoder/decoders I said. Just because it can write #svg don't mean it should I said. 😉

Good to have data though.

This might be interesting for you @diacritica

16 August at 17:33 | Open on floss.social