Gregory's Server
@luminesce @jonny the energy consumption per demand would likely increase because there's more overall computation involved in the process. the LLM itself would be only one step in an arbitrarily complex chain of queries depending on the precise application
from what i understand, the primary energy consumption of LLMs comes from the training process, but i don't know the full details
the specific application i was seeing discussed was to create a way for doctors to use human language to query medical records and collate data to facilitate diagnosis. so while it is important to keep in mind the environmental impact of using LLMs, it's also important to weigh that against potential benefits as well.
frankly, i'd rather we weren't using LLMs at all, but i figure this is one of the less bad ways they can be used
@juliana
@luminesce
Yes totally^ it depends on the application re: energy use. Chaining a bunch of models together would probably use more energy, but google et al want to use smaller models hooked up to knowledge graphs to make runtime inference feasible as a consumer product too, so that kind of application would be designed to use less.
The medical case has its own set of fun complications 💓
https://jon-e.net/surveillance-graphs/#nih-the-biomedical-translator
@juliana
@luminesce
Yes totally^ it depends on the application re: energy use. Chaining a bunch of models together would probably use more energy, but google et al want to use smaller models hooked up to knowledge graphs to make runtime inference feasible as a consumer product too, so that kind of application would be designed to use less.