Review by Ashwin #7
Description
current version: ae83a28
Introduction
- Replace Stallman's "Pretend Intelligence" quote with Emily M. Bender and Timnit Gebru et al.'s Stochastic Parrots term (https://doi.org/10.1145/3442188.3445922) which is more famous and for other reasons including: more rational and balanced observation and makes for equal representation with female researchers in the field. The phrase speaks volumes of what these models are. I haven't found a nice quote yet except this one:
... LM [Language Model] is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.
Of course this can be too much. If we simply want to lower the hype, then there is something like this too in the introduction of the paper:
As argued by Bender and Koller, it is important to understand the limitations of LMs and put their success in context. This not only helps reduce hype which can mislead the public and researchers themselves regarding the capabilities of these LMs, but might encourage new research directions that do not necessarily depend on having larger LMs
- Discussion: Why does training data matter?
- The size of typical datasets also implies that to train a model from the scratch is a heavy commitment. It is expensive, both in terms of time (person-hours) and compute. Which in turn means that very few actors do this
Note
WIP review...