CS Talk
Alex Lew
Host: Zhong Shao
Title: Scaling Probabilistic AI via Automatic Differentiation of Probabilistic Programs
Abstract:
By automating the error-prone math behind deep learning, systems such as TensorFlow and PyTorch have supercharged machine learning research, empowering hundreds of thousands of practitioners to rapidly explore the design space of neural network architectures and training algorithms. In this talk, I will show how new programming language techniques, especially generalizations of automatic differentiation, make it possible to generalize and extend such systems to support probabilistic models. Our automation is rigorously proven sound using new semantic techniques for reasoning compositionally about expressive probabilistic programs, and static types are employed to ensure important preconditions for soundness, eliminating large classes of implementation bugs. Providing a further boost, our tools can help users correctly implement fast, low-variance, unbiased estimators of gradients and probability densities that are too expensive to compute exactly, enabling orders-of-magnitude speedups in downstream optimization and inference algorithms.
To illustrate the value of these techniques, I’ll show how they have helped us experiment with new architectures that could address key challenges with today’s dominant AI models. In particular, I’ll showcase systems we’ve built for (1) auditable reasoning and learning in relational domains, enabling the detection of thousands of errors across millions of Medicare records, and (2) probabilistic inference over large language models, enabling small open models to outperform GPT-4 on several constrained generation benchmarks.
Bio:
Alex Lew is a PhD candidate at MIT, co-advised by Vikash Mansinghka and Joshua Tenenbaum. His research focuses on the intersection of programming languages and probabilistic machine learning, aiming to develop systems and theory that facilitate the invention, application, and understanding of scalable algorithms for probabilistic modeling and inference. Prior to his PhD, Alex taught computer science to high school students at the Commonwealth School in Boston, MA. Alex’s work has been recognized with a 2019 Facebook Probability and Programming Award, a 2020 NSF Graduate Research Fellowship, and 2023 ACM SIGPLAN and ACM SIGLOG Distinguished Paper awards.