@alyssa
This is something I frequently ran into as a researcher already years and years ago, way before LLMs:

Some publication or columnist confidently spouts absolute nonsense about stuff in my own field; and now I can never trust what they say about any other subject.

@alyssa Reminds me a bit of "journalism" in general. The amount of nonsense I read in credible newspapers, speaking about "tech(nology)".

Makes me wonder how much nonsense they report in areas I'm not very knowledgeable about.

@alyssa what are your thoughts on isolated decision making behind a fixed input/output layer? For example imagine if Siri etc. Used a LLM to interpret and do language parsing but NOT to build the response

@alyssa It's a little disheartening that the mass response to public LLM availability seems to be "Well now I get to think EVEN LESS!".

@alyssa LLMs works best for popular texts, with huge corpus and data. For sure GPU drivers programming is faaar from it. They might be useful, as a tool, for helping in some tedious, repetitive work, but for sure not for such niche work.

It should always be used as a tool - with big amount of scrutiny.

Huge problem is, that if they „don’t know” something - LLMs are simply confabulating 🙃 Extremely dangerous for beginners or less ML-literate people 🫤

on a slightly related note: the annoyances of hearing clearly-unqualified people tell you your job is worthless, your passion is not worth pursuing and your work is unnecessary because of LLMs and AIs in general...

you know they're wrong, but it's disheartening still.

@alyssa Could LLMs be suitable for generating proofs which are then checked by a _sound_ proof checker that is secure against malicious input? If the proof is wrong, the checker will catch it, so no harm has been done.

@alyssa do you find the code it emits too buggy to use, or the chat interface too time-consuming to use?

@alyssa @anildash “cited to GPT-4” … ooof … reminds me of spam blogs that put “Source: Reddit” on their content regurgitation mill.

@alyssa

One of the best articles I've read isn't about specific types of highly specialized work, or even loosely specialized. It's something a five-year-old can typically explain:

Tic-Tac-Toe.

And ChatGPT is excellent at coming up with a seemingly convincing explanation for its tactics. Long-winded and verbose. But it's pants at playing, and I think that perfectly illustrates the difference between the illusion of intelligence, and actual brains.

https://www.aiweirdness.com/optimum-tic-tac-toe/

@alyssa LLMs are an excellent way of getting quick answers to some problems, and if you're open, learn a thing or two that you haven't considered. It's useful. But taking any LLM output at face value without double checking means you're foolish and naive.

@alyssa I asked ChatGPT some fairly basic math questions (not computations) and it very confidently gave wrong answers. Have to be really careful with it.

@alyssa

Me: "In what year was the Battle of Hastings?"

AI: "The Battle of Hastings took place on October 14, 1066."

I'm sorry but it's your credibility on the subject that's suspect. Wild general statements like "LLMs are only for entertainment" are ridiculous. Here are some *reasonable* statements:

"Be skeptical of everything you read, and when it really matters always verify."

"Don't use a hammer to fix your plumbing. Every tool has an ideal use, choose wisely."

@alyssa I find they can be occasionally useful if you want exactly what they generate, which is a regression to the mean of subjective answers, queries like “what is a typical naming scheme for a node in a kubernetes cluster” have a very low likelihood of causing actual harm, just don’t trust them for anything more complicated than incredibly general questions that don’t have wrong answers. Which to be clear, is most things. You shouldn’t trust them for most things

@alyssa

Is this the first instance of someone being "chatsplained" to?

@alyssa I ‘almost’ told the parties recently that ‘no AI was used in the preparation of this (arbitration) Award,’ which was true, but I finally figured it would be worse to say it: most wouldn’t yet get it, and those who did would think I was making a jk, that might not land right.

@alyssa Yeah. I've used it for daft things like "write me a dire straits song about Macbeth". I thought considering they did one about Romeo and Juliette it'd be interesting to see what it'd do.
Using it for anything serious, not a chance.

@alyssa Gosh, I think what you really need right now is an overconfident straight white guy who got his online learning certificate in GPT4 from LinkedIn to wade into the topic and “educate” you. 🤣

@alyssa “In a time of deceit telling the truth is a revolutionary act.”
― George Orwell

@alyssa This 1000%. ChatGPT should never be used for serious work, especially anything like legal defense. I'm sorry you have a bunch of ChatGPT apologists in your replies, so I'd like to let you know, there are a lot of us who fully agree with you

They're also good for any time when the output is easily tested/verified. I've used GPT for little scripts in unfamiliar languages for example.

They're good for manipulating information you give them, but can't be trusted to go out and find it in the first place.

@alyssa This, for everything.

@alyssa I use LLMs for research, for economics, for summarizing, and for programming. It's all fine. LLMs are highly useful tools.

Publishing LLM-produced output without extensive checking and editing is dumb.

@alyssa Yeah I had a friend mess with OpenAI to write a fake vulnerability report. It lied to him multiple times, and as he continued correcting it, ChatGPT started making dead links to non-existent vulnerabilities so he LOL’d hard at it because it was just citing unrelated CVEs in his troll report 😂

@alyssa@social.treehouse.systems in 👏 this 👏 house 👏 we 👏 only 👏 trust 👏 LLVMs

@alyssa I have to admit I find it quite useful for generating initial PoC code.

Though usually I still have to do edits before the code even runs.

That being said, one should never use LLMs as a knowledge source. Today I had it tell me that `let` in Rust is used declare mutable variables xD

@alyssa Feel like a whole looooooooooooota mothafuckers are about to learn in real time how trust / journalistic integrity works

@alyssa LLM is mansplaining-as-a-service - confidently given completely wrong answers 🤣

@alyssa Yeah I just use it like google. For some reason people know when they click a link on google they understand the information can be unreliable. With Chat GPT they implicitly trust it.

Where do people think Chat GPT gets it's data from?

In this case it's probably just quoting something from a stack overflow question from someone that didn't know what they were doing (which is why they were there). But sounding really confident while doing it.

@alyssa I've heard some interesting arguments for using it to come up with project names or to expand on an email you're having trouble writing.

Of course the asterisks to that are you need to edit it to your own after gpt goes at it and you need to verify everything contained within is true.

@alyssa I mean I use copilot for coding and it's an LLM and in that context I find it very helpful. It's not doing the decision making though, mostly just remembering obscure syntax for me

@alyssa
Using LLMs for anything is climate science denial. https://news.climate.columbia.edu/2023/06/09/ais-growing-carbon-footprint/

@alyssa Respectfully, I don't agree. LLMs are super at helping out when you can *know* without the tiniest sliver of doubt that the results are correct, and when you treat the results like a suggestion to be vetted, corrected and massaged and not a completed final deliverable.

@alyssa There are some LLMS specifically designed for permutative writing instead of generation, such as Perplexity.ai, which do show promise for being a viable AI search engine replacement, but I feel it AI is currently misunderstood and misused by the public, especially those who confuse chatbots for base models and general AI

tldr; we need better AI education for the general public to solve for false credibility

@alyssa maybe I’m wrong here, but I think it’s not bad to use LLMs to ask them questions as long as you order them to give you their sources, too. So that you check again what they tell you.
Did that when asking for dog facts for a friend. The specific question I had wasn’t answered properly by any other search engine. So I asked ChatGPT but also asked it to provide me its sources.

@alyssa Had a similar experience a couple of times. ChatGPT straight up doesn't understand pointers - I was debugging some code and ran it through to see if it could find the error and it very confidently told me that actually I should allocate space for a double pointer to a data structure, and assign the result (a pointer) to a double pointer - in which case I'm returning a pointer to a double pointer and assigning that pointer to a double pointer and ending up with a triple pointer

@alyssa Thank you for the post. This sort of thing is going to be a huge issue going forward.

@alyssa reminds me of this: https://www.johndcook.com/blog/2021/01/18/gell-mann-amnesia/

@alyssa Yes! That!

Trusting what boils down to an "autocomplete on steroids" for answering you accurate informations is completely asinine.

At best, use it to reformulate nicely information that you know and you're feeding to it.

Or don't use it in scientific context at all.

@alyssa@social.treehouse.systems
I sort of agree.
The issue is that some people think of LLMs as a knowledge systems, which they aren't.
But I don't think this means that they're just for entertainment either. There are legitimate use cases for making sense of garbled data for example. There is also emergent behavior, like problem solving, that will be really useful in the future, I think.