@darkling@mstdn.social @cstross@wandering.shop It's the wrong question. The correct question is, "What proportion of answers to programming questions given by programmers who understand the language and the question are wrong."

Ask almost ANY question of someone who doesn't actually understand the question or the subject, and the answer you receive is overwhelmingly likely to be wrong. (This is doubly true in America, where it seems to be considered almost a mortal sin to ever be heard to say "I don't know".)

@darkling @cstross Indeed I wonder what the StackOverflow accepted answer fail rate is. It's quite subjective. Usually the second or third answer on SO is the correct one, the first usually being the product of some diseased brain that doesn't do real work.

@darkling @cstross An important difference is that on Stack Overflow, volunteers will usually have posted corrections.

Whereas with ChatGPT, people turn up in forums to ask someone else to do the work for them of determining whether the slop from the bot is correct or not.

@darkling @cstross I'd argue that if you hire a software engineer and they're wrong 52% of the time, that wasn't a good hire.

@darkling @cstross
The critical difference, something I don't think I saw anyone mention in this thread, is that human beings understand how to say "I don't know."

An LLM is an even worse version of some asshole that weighs in on *everything* and asserts wrong answers just as confidently as right answers