Uncertainty Quantification

Data vs Model Uncertainty

Uncertainty can arise from noisy data, model architecture, incomplete data, etc. and generally can be classified as either model or data uncertainty. From them modeler’s point of view, we are most concerned with model uncertainty since we are able to improve our models through training of a DNN. We can use the Bayesian framework to formalize model uncertainty as the probabliity distribution over model parameters \(\theta\) for a given dataset, and data uncertainty as the probility distribution over our outputs \(y^*\) given a parameterized model \(f_\theta\)

[1] A Survey of Uncertainty in Deep Neural Networks. Figure 11.

\(p(y^*|x^*, D) = \int \underbrace{p(y^*|x^*, \theta)}_{\text{Data}} \underbrace{p(\theta|D)}_{\text{Model}} \, d\theta\)

Data Uncertainty

Data uncertainty can be traced to errors in data measurement, and is caused by a loss of information when translating real world observations into measurable ones. In a classification task with the MNIST dataset, data uncertainty could be caused by lower resolution versions of the images. In a regression task predicting various qualities of wine, any inaccuracies caused by the measuring equipment, weather, or human error cannot be learned by the neural network. Data uncertainty is considered irreducible due to the imperfect manner in which we collect data and conduct experiments. This introduces an unavoidable element of randomness that causes variability in experiment outcomes. This means we can only estimate data uncertainty, since reducing it is not a possibility.

[1]

Figure 1 Figure 2

[2] Generative models and Bayesian inversion using Laplace approximation Figure 2.

Model Uncertainty

Epistemic uncertainty is caused by the model, which is why its referred to as model uncertainty. Model uncertainty is typically encapsulated by errors in architecture design, training procedures, or insufficient training data . As the modeler, we have full control over both the architecture and training procudure, thus model uncertainty is theoretically reducible. Therefore it is our goal as modelers to create DNNs that can handle uncertain inputs, reduce or remove model uncertainty, and provide data uncertainty estimates for our predictions.

[1]

References

[1] Gawlikowski, J., Njieutcheu Tassi, C. R., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., Shahzad, M., Yang, W., Bamler, R., & Zhu, X. X. (2022). A Survey of Uncertainty in Deep Neural Networks. arXiv preprint arXiv:2107.03342.

[2] Marschall, M., Wübbeler, G., Schmähling, F., & Elster, C. (2023). Generative models and Bayesian inversion using Laplace approximation. Computational Statistics, Advance online publication. https://doi.org/10.1007/s00180-023-01345-5