Good look at the impact of open-source and small models...

All posts Jeff's posts Post Back to profile

Good look at the impact of open-source and small models on AI:
"What was once the reserve of the resource-rich has become a playground for anyone with curiosity, coding skills, and a good laptop."
This isn’t the first time companies have ignored the power of the open-source community. https://slate.com/technology/2023/05/ai-regulation-open-source-meta.html?utm_medium=social&utm_campaign=traffic&utm_source=article&utm_content=twitter_share

Like 31 May 2023 at 11:14 | Open on mastodon.social

6 comments

Sagebiel Markus

@jeffjarvis @charles_ex

31 May 2023 at 11:22 | Open on mastodon.social

Erick RM

@jeffjarvis Well said Mr. Jarvis, specifically about having a good laptop.

31 May 2023 at 12:06 | Open on mastodon.online

DELETED

@jeffjarvis

Real open source LLMs predate LLAMA by years. The article seems to ignore this.

GPT-J 6B is a transformer model with 6 billion trainable parameters. It was released by the non-profit research group Eleuther AI in July 2020. The model is based on a 825 gigabyte open-source language modeling data set called The Pile.

To self-host GPT-J 6B, you may need a machine with at least 48 GB RAM, a minimum of 12-16 GB GPU RAM for inference, and about 90 GB GPU RAM for training.

31 May 2023 at 12:09 | Open on mastodon.social

Don Marti

@i_understand @jeffjarvis Goes back to the "we have no moat" problem. The differentiating advantages for generative AI contenders are going to be not technology or scale, but willingness and ability to violate norms or laws (use copyrighted content without a license, train on forum users without consent, omit filters for some illegal output...)

https://blog.zgp.org/generative-ai-antimoats/

31 May 2023 at 14:05 | Open on federate.social

ryb

@i_understand It doesn’t matter because language models before ChatGPT were underwhelming and not particularly useful.

1 Jun 2023 at 1:55 | Open on mastodon.sdf.org

DELETED

@ryb It wasn't until 2017 when Transformers revolutionized AI, starting with NLP.

And I'm not sure what you mean by it doesn't matter, since ChatGPT isn't based on LLAMA either. It's based on GPT-3, GPT-2 was open sourced in 2019. Again, all of these events and innovations (including the release of ChatGPT) pre-date the release of LLAMA.

1 Jun 2023 at 13:32 | Open on mastodon.social