Link Search Menu Expand Document

Weekly Schedule

Week-to-week schedule and papers covered are tentative, and may change by the start of the quarter

Introduction and taxonomy of distribution shifts

Sep 20
Introduction
Lecture
  1. Overview of the course
  2. Distribution shifts in the real world
  3. A taxonomy of distribution shifts and how they arise
Sep 22
Covariate and label shifts
Lecture + Discussion
  1. What is a covariate shift?
  2. Handling covariate shift under distribuitonal overlap.
  3. Shortcut Learning in Deep Neural Networks
Sep 27
Covariate and label shifts 2
Discussion
  1. Improving Predictive Inference Under Covariate Shift by Weighting the Log-Likelihood
  2. Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure

Domain adaptation theory

Sep 29
Domain adaptation
Lecture
  1. When can we provably learn under distribution shift?
  2. Can unlabeled data help?
  3. Defining generalization bounds under distribution shift.
Oct 4
Domain adaptation 2
Discussion
  1. A Theory of Learning from Different Domains
  2. Optimal Transport for Domain Adaptation

Neural and representation-based methods

Oct 6
Neural domain adaptation
Lecture
  1. Indistinguishability over representations.
  2. Adversarial approaches to neural domain adaptation.
  3. Connections to classical theory.
Oct 11
Neural domain adaptation 2
Discussion
  1. Domain Adversarial Training of Neural Networks
  2. Geometry-Consistent Generative Adversarial Networks for One-Sided Unsupervised Domain Mapping
Oct 13
Neural domain adaptation 3
Lecture
  1. Provable guarantees from representational indistinguishability
  2. Self-training based domain adaptation
  3. Self-supervision based domain adaptation
Oct 18
Learning from invariant representations 2
Discussion
  1. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts
  2. Support and Invertibility in Domain-Invariant Representations

Robustness and domain generalization

Oct 20
Empirical phenomena in robust machine learning
Lecture
  1. How do different robustness interventions fare in practice?
  2. Can (data augmentation / unlabeled data / bigger models) help?
Oct 25
Empirical phenomena in robust machine learning 2 + Project (Progress report due)
Discussion
  1. Using Pre-Training Can Improve Model Robustness and Uncertainty
  2. Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization
Oct 27
Connections to causality
Lecture
  1. Distribution shifts as arising from causal interventions.
  2. Existing connections between causality and robustness.
  3. Robustness and invariance as tools for causal inference.
Nov 1
Connections to causality 2
Discussion
  1. Conditional Variance Penalties and Domain Shift Robustness
  2. Invariant Risk Minimization
Nov 3
Minimax methods
Lecture
  1. Robustness as a minimax game between nature and the model.
  2. Tractable families of worst-case distributions and duality.
  3. Pitfalls and pessimism from worst-case bounds.
Nov 8
Minimax methods 2
Discussion
  1. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
  2. Certifiying Some Distributional Robustness with Principled Adversarial Training

Adversarial robustness

Nov 10
Adversarial examples
Lecture
  1. Defining and motivating adversarial examples.
  2. Heuristic defenses and their pitfalls
  3. Provable defenses.
Nov 15
Adversarial examples 2
Discussion
  1. Unlabeled Data Improves Adversarial Robustness
  2. Certified Adversarial Robustness via Randomized Smoothing
Nov 17
Data poisoning
Lecture
  1. Classical robust statistics
  2. High-dimensional mean estimation
  3. Convex optimization under data poisoning
Nov 29
Data Poisoning 2
Discussion
  1. Being Robust (in High Dimensions) Can Be Practical
  2. SEVER: A Robust Meta-Algorithm for Stochastic Optimization
Dec 1
Short project presentations
Project