Model weights are assumed to have a true value that is just unknown
All weights are having a single fixed value as is the norm
Often the absence of a statistical flavor to such an analysis is prone to overfitting on selected examples and in general, presents challenges to draw confident conclusions.
Softmax
Model can be uncertain in its predictions even with a high softmax output
While the continuous neural net is responsible for adjusting the ration of class probabilities, softmax squashes these ratios into a simplex
Softmax is also notorious with inflating the probability of the predicted class as a result of the exponent employed on the neural net outputs
๐ฐ Probabilistic Neural Network
Model weights are treated as random variables
Probabilistic interpretation of deep learning models
๐ฐ Bayesian Neural Network
Probabilistic interpretation of deep learning models
Robust to over-fitting & offer uncertainty estimates & easily learn from small datasets
by inferring distributions over the model's weights
Specify a distribution over functions, known as stochastic process
Model weights are treated as random variables.
Model weights are represented by probability distributions over possible values
Represent a distribution over these weights in terms of probabilities we can observe, resulting in the distribution of model parameters conditional on the data we ahve seen
We want to learn a distribution of these parameters conditional on training data
Learnt representations and computations must therefore be robust under perturbation of the weights, but the amount of perturbation each weight exhibits is also learnt in a way that coherently explains variability in the training data.
Passing the distribution thorugh a softmax better classification uncertainty far from the training data
Gaussian process: Probability distributions over functions, model uncertainty can still be obtained by placing distributions over the weights
Stochastic regularization technique(SRT)
Probabilistic classification model
Placing a distribution over model parameters, and then marginalizing these parameters to form a whole predictive distribution in a procedure known as Bayesian model averaging