 Holy _shit_ this paper, and the insight behind it. You know how every receiver is also a transmitter, _well_: every text predictor is also text compressor, and vice-versa. You can outperform massive neural networks running millions of parameters, with a few lines of python and a novel application of _gzip_. 57 comments
  If this is reliable, this is "take something that needed a datacenter last year and do it on a phone this year" material.  @mhoye every once in a while we find another one of these: https://en.wikipedia.org/wiki/Noether%27s_theorem @fogti @XaiaX So, I don't actually think that acres of storage space is all that meaningful if you just want to focus good-enough utility. We all kind of know you're not getting more than varying heats of spicy autocomplete out of these tools, and the core insight of statistics as a field is that you can make a very accurate approximation of the state of a large dataset out of a small subset of that data. So, maybe a wikipedia dump and a gutenberg mirror is plenty? @mhoye @XaiaX > spicy autocomplete huh, nah the topic here is text classification, which is similar to text prediction (and there is probably a way to use Huffman tables to produce suggestions, etc.), but not the same. > So, maybe a wikipedia dump and a gutenberg mirror is plenty? probably yes. (imo coolest would be producing a tool that could both classify and predict using the same infrastructure (light preprocessed large text dumps (<10GiB), massive improvement) @mhoye @XaiaX interesting thing is that it is probably *much* easier to simultaneously get the information for text completion *and* also what authors were involved in that match (instead of a large pool of authors just the relevant subset). [in the case of these compression-decompression + reference dataset models] Idle thought that LaTeX equations rendering properly would be a pretty good use case for a Mastodon client, given the overall geeky clientele on here! cc. @ivory @mhoye Wait, what? So their claim is that anything that is decent at compression, should also be decent at prediction? So therefore we've missed the boat because all of the work on compression the past few decades means we have really good predictors? Definitely going to be reading this.  @gabrielesvelto @rmflight It is, and if this paper's results hold up it we're talking about how large scale deep-learning networks are fundamentally a technical dead end, and something that takes a datacenter to do with a DNN can be done better with a clever application of gzip on a phone. @mhoye @gabrielesvelto @rmflight So what you’re saying is that I’m about to get a screaming deal on a graphics card? @nickw @gabrielesvelto @rmflight My friend, you are about to get an _astounding_ deal on a graphics card. @mhoye @gabrielesvelto @rmflight Hold on. I’m headed out to Hayes Valley with a canvas bag and a roll of twenties. @mhoye @gabrielesvelto @rmflight On first look I think what this paper suggests is 1) for some classification tasks there's nicely simple approach that works well and 2) this is a promising path towards better feature engineering for language models that will in turn result in better accuracy vs cost. @mhoye @gabrielesvelto @rmflight If this works out well we'll see better + smaller models for all tasks (not just classification) that outperform both current DNNs and the NCD technique they use at moderate cost. There's precedent of this being a successful approach for example using frequency domain data for audio models instead of raw PCM. There's also precedent for finding ways DNNs waste a lot of capacity on effectively routing data around and restructuring to fix (ResNets for example). @mhoye @gabrielesvelto @rmflight Overall though in recent history data-based approaches have tended to win so I would expect the useful bits to get incorporated into DNNs rather than DNNs being obsoleted in almost any context. My favorite essay on that topic by Rich Sutton: http://incompleteideas.net/IncIdeas/BitterLesson.html @paulmheider @rmflight And Ted Chiang has referred to ML-generated text as "a jpeg of a language", yeah. But to see that come together in fifteen lines of python that out-do these massive, crazy expensive DNN models is bonkers, jaws on the floor material.  @mhoye holy fuck this is potentially brilliant, and I am so excited to dive into reading this after work  @mhoye compression methods have been used in text classification for authorship analysis for quite a while.  @mhoye Compression is ... very interesting. Recalling here Schmidhuber's take on compression and its predictability being at the root of learning, beauty, novelty, interestingness, boredom ... https://arxiv.org/pdf/0812.4360 Previously: https://mathstodon.xyz/@albertcardona/110686536069075845  @mhoye (this is not particularly novel, I remember reading a similar paper about using gzip to de-anonymize written works back around when I was learning about singular value decomposition to support vector space document searching from Maciej Ceglowski, I think back in the early 2000s. But it definitely is novel that this old technique is now outperforming current tech darlings.)  @mhoye This is awesome, but I'm surprised it wasn't better known. I have vague memories of going to a talk by a researcher in Oxford about 25 years ago about using gzip compression for text analysis. His presentation explained about entropy and how compression is prediction, then looked at categorising text by gzipping it. Can't remember the name; some guy doing inference stuff in the psychology department. This is going to bug me now. @simoncozens From what I can tell the fact of it wasn't a big secret, but the idea that with apparently negligible effort you can outperform tools that are insanely expensive and wildly more complicated is the interesting part. @mhoye The right algorithm in the right place beats an inscrutable pile of ReLUs every damned time. @mhoye Found it. Patrick Juola was the guy. 1998, categorizing the complexity of languages by gzipping them. https://scholar.google.co.uk/scholar?q=patrick+juola+complexity+compression&hl=en&as_sdt=0&as_vis=1&oi=scholart#d=gs_qabs&t=1689286759338&u=%23p%3DbcHXg7kuXAIJ  @mhoye So this works for classification, but can it be adapted to work for transformers with attention? I still can't fully get my head around how transformers work, so I don't know if this technique translates, but if it does... 🤯 @mhoye you can outperform a non-pre-trained deep neural network. I don’t understand text classification enough to know what the difference is there. ”Non-pre-trained” sounds like a key factor in that comparison though. |
Gregory's Server
@mhoye !!!!!