@jonny OK but you don't know that till after you've observed it awhile and i guess what I'm wondering is where one would get the idea in the first place? from someone else? from the internet? there's a stunningly wide gap in understanding if a person thinks chatgpt is evaluating code, both of chatgpt and of coding! how does that happen to a person? I'm fully failing to understand π
@walruslifestyle
Their mental model is that "I can talk to this thing, when I give it some code it knows what that code is and can tell me about it in the same way that it seems to tell me about lots of things," and they are not so naΓ―ve in my opinion because products like copilot do advertise themselves as understanding code, so thinking the LLM is actually parsing it and reasoning about it rather than generating plausible text from some seed vector in its latent space is reasonable enough to me.