Uncertainty Estimation in Machine Learning

Lecturer

Matias Valdenegro-Toro, m.a.valdenegro.toro@rug.nl

Marco Zullich, m.zullich@rug.nl

Content and organization

This course is tailored for PhD or MSc students with a background in Machine Learning (ML) or Statistics.
The lectures will take students from a frequentist view of ML and Neural Networks (NNs) towards a Bayesian setting using a friendly and practice-oriented approach.

The course is organized in 10 lectures (6 frontal lectures and 4 practical sessions) centered on the following topics:

1. Introduction to Uncertainty in Machine Learning
  - Motivation for Uncertainty, Types of Uncertainty, connection to Explainability.
2. Methods for Uncertainty in Machine Learning I
  - Direct methods: Gaussian NLL, Ensembles, DUQ, DDU.
  - Sampling methods: MC-Dropout, MC-DropConnect.
3. Methods for Uncertainty in Machine Learning II (with additional 2 hours lab)
  - Bayesian Neural Networks
  - Variational Inference for BNNs: Bayes by Backprop and Flipout.
  - Disentanglement of Uncertainty.
  - Practical session: Pytorch/Keras implementation of the models introduced in the previous lectures.
4. Evaluation of Uncertainty (with additional 2 hours lab)
  - Losses and Metrics.
  - Evaluation of Calibration.
  - Uncertainty Plots.
  - Practical session: implementing metrics for evaluation of uncertainty in NumPy, drawing uncertainty plots using NumPy and Matplotlib
5. Calibration (with additional 2 hours lab)
  - Concept of Calibration
  - Methods to improve Calibration
  - Practical session: measuring calibration of a (bayesian) neural network in Keras/PyTorch. Improving the calibration of a model using Keras/PyTorch.
6. Out of Distribution Detection and Research Challenges (with additional 2 hours lab)
  - Concept
  - Evaluation of OOD Detection
  - Methods for OOD Detection
  - Practical session I: OOD detection in Keras/PyTorch: evaluation and methods.
  - Practical session II: research challenges, presentation of recent publications in the field of uncertainty estimation, with comments on the practical implementation (research plan, evaluation of results, model implementations…)

Course Duration

Course Type

Short Course

Participation terms

If you are an AIDA Student* already, please a) register by sending an email with subject "[AIDA] registration to UEML course" to m.a.valdenegro.toro@rug.nl AND m.zullich@rug.nl AND b) also enroll in the same course in the AIDA system on this page, in order for this course to be included on your AIDA Course Attendance Certificate. If you are not an AIDA Student, follow only the instructions in step (a). *AIDA Students should have been registered in the AIDA system already (they are PhD students or PostDocs that belong only to the AIDA Members list).

Schedule

Thursday January 25th from 15 to 17 Monday January 29th from 15 to 17 Wednesday January 31st from 17 to 19 Monday February 5th from 15 to 17 Wednesday February 7th from 15 to 17 Friday February 9th from 15 to 17 Monday February 12th from 15 to 17 Wednesday February 14th from 15 to 17 Friday February 16th from 15 to 17 CET