Child pages
  • Targeted Evaluation (5.3)

Contents

Context

Analysis | Network Performance | Target 

Updated Feature: Evaluate Target State

Target Performance Evaluation offers two modes:

  • Evaluate all states
  • Evaluate target state

The latter mode has been extended, so it can now also be used for evaluating Target Nodes with more than two states. 

The decision rule used for computing the precision and the confusion matrix can be based on:

  • Maximum Likelihood: the predicted state is the one with the highest probability
  • One Threshold: the predicted state is the target state if its probability is greater than or equal to the specified threshold, otherwise it is the most likely state
  • Multiple Thresholds: the predicted state is the target state if its probability is greater than or equal to the currently tested threshold, otherwise the most likely state.
Example

As an example, we will use the Perfume dataset with Purchase Intent as a Target Node with 3 states.

The following network was learned with the Augmented Markov Blanket Learning algorithm, using Conditional Probability Trees. The small bullseye symbol in the Monitor indicates that Purchase Intent=Low is the Target State.

 

 

There is a tab for each state of the Target Node, which allows you to access the respective curves and indexes.

 

The results are the same as those obtained in "Evaluate all states" mode. However, curves and local indexes are only available for the Target State, Purchase Intent=Low.

Setting the threshold to 0.2 (i.e. slightly above the prior probability P(Target State=Low)=0.18) increases the precision of the Target State Low, but decreases the precision of the state Mid while leaving the precision of the state High unchanged.

The curves and indexes are computed using the probability returned by the model. Thus, they are not affected by the decision rule used for computing the precision and confusion matrix.

Here, 50 thresholds have been tested. Selecting a threshold from the list allows you to retrieve the performance of the corresponding decision rule.

 

New Feature: Maximum Size of Evidence

This new option allows you to measure the predictive quality of a network given a constraint with regard to the number of pieces of evidence. 

This evaluation simulates the behavior of the Adaptive Questionnaire. The variables used for the prediction are dynamically selected based on their Mutual Information given the currently available information.

Example

Let's assume we wish to evaluate the precision of our model by only using 3 questions, namely the 3 most efficient questions in terms of the reduction of the uncertainty of the Target Node.

 

The total precision is almost the same as the one we obtained by observing all seven nodes, i.e. one piece of evidence for each node.

The following panel show two possible sequences for collecting/setting evidence. Whereas the first question is the same in all the sequences, the subsequent questions depend on the answer of the prior questions.

The selection of the questions to ask (i.e. the evidence to set) depends on the Binary Mutual Information, i.e. considering the Target State versus the other states.

New Feature: Charts for Comprehensive Report with Multiple Thresholds  

The Multiple Thresholds option in Evaluate Target State generates a set of decision rules based on different probability thresholds. Selecting a threshold from the drop-down list shows the performance measures for the corresponding decision rule. 

The Comprehensive Report lists the metrics corresponding to each threshold in a HTML report, thus offering you an alternative way for selecting the best decision rule.

This report now has a new Charts function that generates curves illustrating the performance on the basis of each decision rule.