Title: Less Superhuman Language Models

Abstract: AI models surpass human capabilities in many areas: as a simple example, unlike humans, models can recall long lists of digits without errors. In this talk, I'll discuss ongoing work that highlights two consequences of the growing gap between models and humans, and proposes ways to address them. The first consequence of models' superhumanness is diminished effectiveness as cognitive models: I will show this through the example of human next-word prediction in reading. The second is their limited usefulness as human simulators; I will argue that if models incorporated human resource limitations, they could be an effective way to evaluate and improve AI models' ability to interact with humans.

Tal Linzen is an Associate Professor of Linguistics and Data Science at New York University, and a Research Scientist at Google. He directs NYU's Computation and Psycholinguistics Lab, which uses behavioral experiments and computational methods to study how people learn and understand language. At both Google and NYU, he works on large language model post-training, evaluation and interpretability.