Direct and dual Information Bottleneck frameworks for Deep Learning
Description
The Information Bottleneck (IB) is an information theoretic framework for optimal representation learning. It stems from the problem of finding minimal sufficient statistics in supervised learning, but has insightful implications for Deep Learning. In particular, it's the only theory that gives concrete predictions on the different representations in each layer and their potential computational benefit. I will review the theory and its new version, the dual Information Bottleneck, related to the variational Information Bottleneck which is gaining practical popularity. In particular, I will discuss the implications of the critical points (phase transitions) of the IB and dual IB and their importance for topological transformations of the consecutive successively refinable representations.
Based on joint works with Ravid Schwartz Ziv, Noga Zaslavsky, and Zoe Piran.
Speaker Bio
Dr. Naftali Tishby is a professor of Computer Science, and the incumbent of the Ruth and Stan Flinkman Chair for Brain Research at the Edmond and Lily Safra Center for Brain Science (ELSC) at the Hebrew University of Jerusalem. He is one of the leaders of machine learning research and computational neuroscience in Israel and his numerous ex-students serve at key academic and industrial research positions all over the world. Prof. Tishby was the founding chair of the new computer-engineering program, and a director of the Leibnitz research center in computer science, at the Hebrew University. Tishby received his PhD in theoretical physics from the Hebrew University in 1985 and was a research staff member at MIT and Bell Labs from 1985 to 1991. Prof. Tishby was also a visiting professor at Princeton NECI, University of Pennsylvania, UCSB, and IBM Research. His current research is at the interface between computer science, statistical physics, and computational neuroscience. He pioneered various applications of statistical physics and information theory in computational learning theory. More recently, he has been working on the foundations of biological information processing and the connections between dynamics and information. He has introduced, with his colleagues, new theoretical frameworks for optimal adaptation and efficient information representation in biology, such as the Information Bottleneck method and the Minimum Information principle for neural coding.
Additional Info
The MIT Colloquium on the Brain and Cognition is a lecture series held weekly during the academic year and features a wide array of speakers from all areas of neuroscience and cognitive science research. The social teas that follow these colloquia bring together students, staff, and faculty to discuss the talk, as well as other research activities within Building 46, at MIT, and around the world. This event is co-sponsored by the Department of Brain and Cognitive Sciences, the McGovern Institute for Brain Research, and the Picower Institute for Learning and Memory at MIT. Colloquia are open to the community, and are held in MIT's Building 46, Room 3002 (Singleton Auditorium) at 4:00PM with a reception to follow.