@jonny @walruslifestyle
Especially given the flood of YouTube videos demonstrating ChatGPT solving coding problems in minutes
youtu.be/jwpja9fcqaM

The implied assumption is that humans with their very limited ability to memorize answers would have to understand code in order to arrive at the correct answer. We apply that same assumption to LLMs at our own peril. Surely it couldn't have simply memorized all the answers and is simply applying pattern matching to generate the answer, right?