Analysis | Visual | Sensitivity | Confidence Intervals
When the number of instances used for machine learning the model is limited, it can be useful to analyze the confidence intervals associated with each estimated (conditional) distribution of the network.
The parameters of the models are estimated via Maximum Likelihood, by using the frequencies observed in the data set:
- is the estimated probability
- xi is the state i of variable x
- N(.) represents the number of occurrences of the argument in the data set.
Suppose we have an estimation of 0.1 for x0. This can be obtained with 1 instance out of 10, or 1,000 instances out of 10,000. Clearly, these two scenarios are not equivalent. Our intuition is that, in the second scenario, we should be more confident in our estimate.
Frequentist Estimation of Confidence Intervals
We estimate a confidence interval from the observed data for each parameter.
For a confidence level of 95%, the intervals are computed as follows:
When there is no instance for a given state, the estimation is defined with the Rule of Three as +/- 3/N.
Note that using the Smooth Probability Estimation allows not having to use this heuristic.
The confidence level is the frequency of confidence intervals that contain the actual parameter value Window | Preferences.. It can be set via
This level of confidence is for each parameter. For probability distributions, this means that this confidence level is only correct for binary variables.
For variables with more states, the confidence level associated with the estimation of the distribution of the variable decreases exponentially with the number of states.
Monte Carlo Simulation
Once the confidence intervals estimated for every parameter in the Bayesian network, i.e. every cell of the (conditional) probability tables, a Monte Carlo simulation is carried out to generate a set of networks. Each of these networks uses parameters sampled from the confidence intervals.
These networks are thus used for measuring how these confidence intervals impact the represented joint probability distribution.