#167 How Large Language Models Prove Chomsky Wrong with Steven Piantadosi

SlatorPod

Joining SlatorPod this week is Steven Piantadosi, Associate Professor of Psychology at UC Berkeley. Steven also runs the computation and language lab (colala) at UC Berkeley, which studies the basic computational processes involved in human language and cognition.
Steven talks about the emergence of large language models (LLMs) and how it has reshaped our understanding of language processing and language acquisition.
Steven breaks down his March 2023 paper, "Modern language models refute Chomsky’s approach to language”. He argues that LLMs demonstrate a wide range of powerful language abilities and disprove foundational assumptions underpinning Noam Chomsky's theories and, as a consequence, negate parts of modern Linguistics.
Steven shares how he prompted ChatGPT to generate coherent and sensible responses that go beyond its training data, showcasing its ability to produce creative outputs. While critics argue that it is merely an endless sequence of predicting the next token, Steven explains how the process allows the models to discover insights about language and potentially the world itself.
Steven acknowledges that LLMs operate differently from humans, as models excel at language generation but lack certain human modes of reasoning when it comes to complex questions or scenarios. He unpacks the BabyLM Challenge which explores whether models can be trained on human-sized amounts of data and still learn syntax or other linguistic aspects effectively.
Despite industry advancements and the trillion-dollar market opportunity, Steven agrees with Chomsky's ethical concerns, including issues such as the presence of harmful content, misinformation, and the potential impact on job displacement.
Steven remains enthusiastic about the potential of LLMs and believes the recent advancements are a step forward to achieving artificial general intelligence, but refrains from making any concrete predictions.

Chapters

1. Intro and Agenda (00:00:00)

2. UC Berkeley and The Computation and Language Lab (00:00:37)

3. Why LLMs are Philosophically Important (00:02:24)

4. How LLMs Refute Chomsky's Approach to Language (00:05:11)

5. Prompting ChatGPT Beyond its Training Data (00:11:49)

6. Critic's View on LLMs (00:14:50)

7. Reaction from the Linguistics Community (00:18:07)

8. How LLMs Differ from Humans when Generating Language (00:20:27)

9. BabyLM Challenge (00:24:50)

10. Ethical Concerns of LLMs (00:31:19)

11. Progress Towards Artificial General Intelligence (00:38:43)