So let’s begin with the first one<\/p>\n
<\/p>\n
1. Formulation of the Problem<\/h3>\nSuppose we have an input vector x together with a corresponding vector t of the target variable. This set of input vector x, and target vector t forms the training data set.<\/div>\n<\/div>\nNow, the goal is to predict t for a new value of x. For classification problem, t would represent class labels. The joint probability distribution p(x, t) provides a complete summary of the uncertainty associated with these variables. Determination of this probability from the training data set is an example of inference problem<\/div>\n <\/p>\n
2. The Cancer Diagnosis Example<\/h3>\nLet’s take the example of the medical diagnosis problem where we need to determine whether or not the patient have cancer. An x-ray image of the patient is taken, so the input vector x is a set of pixel intensities of the x-ray image.<\/div>\nThe output variable t will represent the presence or absence of cancer. Now we can form two classes C1<\/sub> and C2<\/sub>. C<sub1 represents the presence of cancer while C2 represents the absence of cancer.<\/div>\nWe need to assign the input to one of the two classes.<\/div>\n <\/p>\n
3. The Inference and Decision Problems<\/h3>\nThe general inference\u00a0 involves determining the joint distribution p(x, Ck), or p(x,t), which give the most complete probabilistic description of the problem. This is the inference step.<\/div>\nAfter this, a decision is made whether to give treatment or not. This is the decision step. This step becomes easy if the inference problem have been solved.<\/div>\n <\/p>\n
4. The Role of Probability<\/h3>\nWhen the x-ray images is obtained x for a new patient, the goal is to determine which of the two classes to assign the image.<\/div>\n<\/div>\nWe could assign it to any of the two classes, C1 or C2 (let’s use Ck, for k = {1,2}). We need to find the probabilities of the two classes given the image. This probability is given by p(Ck | x). Using Bayes theorem, these probabilities can be expressed in the form:<\/div>\n<\/div>\n<\/div>\n![](\"https:\/\/3.bp.blogspot.com\/-_xOUL80zqAU\/WmSTn9rciSI\/AAAAAAAAA4M\/IAZTHVvp-icqK7pQ8lrlxGHEtjd-y_2zACLcBGAs\/s320\/Bayes%2BTheory%2Bof%2BProbability.jpg\")
<\/a><\/div>\n\n <\/p>\n
5. Minimizing Misclassification<\/h3>\n
Sometimes in classification, we may assign an input to the wrong class. In this case, misclassification have occurred. The goal is to make as few misclassifications as possible.<\/p>\n<\/div>\n
Misclassification occurs when an input variable is assigned to the wrong class. The goal is to minimize the number of misclassifications.<\/div>\n<\/div>\nWe handle this by creating a rule that assigns each x to one of the available classes. This rule would divide the input space into regions Rx called decision regions, one region for each class.<\/div>\nPoints in Rx are assigned to Ck.<\/div>\n<\/div>\nA misclassification when a point in a region is assigned to the wrong class e.g<\/div>\n\n- x is in R1 but is assigned to class C2<\/li>\n
- x is in R2 but is assigned to class C1<\/li>\n<\/ul>\n<\/div>\nThe probability of misclassification is given by:<\/div>\n<\/div>\n
![](\"https:\/\/3.bp.blogspot.com\/-5NzGAv667wc\/WmTiFQ0szEI\/AAAAAAAAA48\/Q_KuwBnXkoAWzTKGbDQg6uu6fZ66v_ifQCLcBGAs\/s400\/Minimizing%2BMisclassification.jpg\")
<\/a><\/div>\n<\/div>\nTo minimize this error, the rule must assign x<\/b><\/span> to which class has minimum integrand:<\/div>\n<\/div>\np(<\/i>x<\/b>, Ck<\/sub>) = p(Ck<\/sub> | <\/i>x<\/b>)p(<\/i>x<\/b>)<\/i><\/span><\/span><\/div>\n<\/div>\nFrom the above formula, to minimize misclassification, x should be assigned to which class the posterior probability is largest.<\/div>\n<\/div>\n\n <\/p>\n
6. Reducing Expected Loss<\/h3>\n
The case of cancer diagnosis shows that a loss incurred when there is misclassification could be of different degrees.<\/p>\n<\/div>\n
So we introduce a loss function or cost function E(L).<\/div>\nSuppose that for a new value of x, the correct class is Ck but we assign it to Cj (where j\u00a0 k). In this case, we have incurred some loss Lkj.<\/div>\nwhere k and j are elements of the loss matrix, (as in the case of cancer diagnosis).<\/div>\nThe optimal solution is the one that minimizes the value of the loss function.<\/div>\n
<\/div>\n
Now, the goal is to predict t for a new value of x. For classification problem, t would represent class labels. The joint probability distribution p(x, t) provides a complete summary of the uncertainty associated with these variables. Determination of this probability from the training data set is an example of inference problem<\/div>\n2. The Cancer Diagnosis Example<\/h3>\n
<\/p>\n
2. The Cancer Diagnosis Example<\/h3>\nLet’s take the example of the medical diagnosis problem where we need to determine whether or not the patient have cancer. An x-ray image of the patient is taken, so the input vector x is a set of pixel intensities of the x-ray image.<\/div>\nThe output variable t will represent the presence or absence of cancer. Now we can form two classes C1<\/sub> and C2<\/sub>. C<sub1 represents the presence of cancer while C2 represents the absence of cancer.<\/div>\nWe need to assign the input to one of the two classes.<\/div>\n <\/p>\n
3. The Inference and Decision Problems<\/h3>\nThe general inference\u00a0 involves determining the joint distribution p(x, Ck), or p(x,t), which give the most complete probabilistic description of the problem. This is the inference step.<\/div>\nAfter this, a decision is made whether to give treatment or not. This is the decision step. This step becomes easy if the inference problem have been solved.<\/div>\n <\/p>\n
4. The Role of Probability<\/h3>\nWhen the x-ray images is obtained x for a new patient, the goal is to determine which of the two classes to assign the image.<\/div>\n<\/div>\nWe could assign it to any of the two classes, C1 or C2 (let’s use Ck, for k = {1,2}). We need to find the probabilities of the two classes given the image. This probability is given by p(Ck | x). Using Bayes theorem, these probabilities can be expressed in the form:<\/div>\n<\/div>\n<\/div>\n<\/a><\/div>\n\n <\/p>\n
5. Minimizing Misclassification<\/h3>\n
Sometimes in classification, we may assign an input to the wrong class. In this case, misclassification have occurred. The goal is to make as few misclassifications as possible.<\/p>\n<\/div>\n
Misclassification occurs when an input variable is assigned to the wrong class. The goal is to minimize the number of misclassifications.<\/div>\n<\/div>\nWe handle this by creating a rule that assigns each x to one of the available classes. This rule would divide the input space into regions Rx called decision regions, one region for each class.<\/div>\nPoints in Rx are assigned to Ck.<\/div>\n<\/div>\nA misclassification when a point in a region is assigned to the wrong class e.g<\/div>\n\n- x is in R1 but is assigned to class C2<\/li>\n
- x is in R2 but is assigned to class C1<\/li>\n<\/ul>\n<\/div>\nThe probability of misclassification is given by:<\/div>\n<\/div>\n<\/a><\/div>\n<\/div>\nTo minimize this error, the rule must assign x<\/b><\/span> to which class has minimum integrand:<\/div>\n<\/div>\np(<\/i>x<\/b>, Ck<\/sub>) = p(Ck<\/sub> | <\/i>x<\/b>)p(<\/i>x<\/b>)<\/i><\/span><\/span><\/div>\n<\/div>\nFrom the above formula, to minimize misclassification, x should be assigned to which class the posterior probability is largest.<\/div>\n<\/div>\n\n
<\/p>\n
6. Reducing Expected Loss<\/h3>\n
The case of cancer diagnosis shows that a loss incurred when there is misclassification could be of different degrees.<\/p>\n<\/div>\n
So we introduce a loss function or cost function E(L).<\/div>\nSuppose that for a new value of x, the correct class is Ck but we assign it to Cj (where j\u00a0 k). In this case, we have incurred some loss Lkj.<\/div>\nwhere k and j are elements of the loss matrix, (as in the case of cancer diagnosis).<\/div>\nThe optimal solution is the one that minimizes the value of the loss function.<\/div>\n
The output variable t will represent the presence or absence of cancer. Now we can form two classes C1<\/sub> and C2<\/sub>. C<sub1 represents the presence of cancer while C2 represents the absence of cancer.<\/div>\n
We need to assign the input to one of the two classes.<\/div>\n3. The Inference and Decision Problems<\/h3>\n
<\/p>\n
3. The Inference and Decision Problems<\/h3>\nThe general inference\u00a0 involves determining the joint distribution p(x, Ck), or p(x,t), which give the most complete probabilistic description of the problem. This is the inference step.<\/div>\nAfter this, a decision is made whether to give treatment or not. This is the decision step. This step becomes easy if the inference problem have been solved.<\/div>\n <\/p>\n
4. The Role of Probability<\/h3>\nWhen the x-ray images is obtained x for a new patient, the goal is to determine which of the two classes to assign the image.<\/div>\n<\/div>\nWe could assign it to any of the two classes, C1 or C2 (let’s use Ck, for k = {1,2}). We need to find the probabilities of the two classes given the image. This probability is given by p(Ck | x). Using Bayes theorem, these probabilities can be expressed in the form:<\/div>\n<\/div>\n<\/div>\n<\/a><\/div>\n\n <\/p>\n
5. Minimizing Misclassification<\/h3>\n
Sometimes in classification, we may assign an input to the wrong class. In this case, misclassification have occurred. The goal is to make as few misclassifications as possible.<\/p>\n<\/div>\n
Misclassification occurs when an input variable is assigned to the wrong class. The goal is to minimize the number of misclassifications.<\/div>\n<\/div>\nWe handle this by creating a rule that assigns each x to one of the available classes. This rule would divide the input space into regions Rx called decision regions, one region for each class.<\/div>\nPoints in Rx are assigned to Ck.<\/div>\n<\/div>\nA misclassification when a point in a region is assigned to the wrong class e.g<\/div>\n\n- x is in R1 but is assigned to class C2<\/li>\n
- x is in R2 but is assigned to class C1<\/li>\n<\/ul>\n<\/div>\nThe probability of misclassification is given by:<\/div>\n<\/div>\n<\/a><\/div>\n<\/div>\nTo minimize this error, the rule must assign x<\/b><\/span> to which class has minimum integrand:<\/div>\n<\/div>\np(<\/i>x<\/b>, Ck<\/sub>) = p(Ck<\/sub> | <\/i>x<\/b>)p(<\/i>x<\/b>)<\/i><\/span><\/span><\/div>\n<\/div>\nFrom the above formula, to minimize misclassification, x should be assigned to which class the posterior probability is largest.<\/div>\n<\/div>\n\n
<\/p>\n
6. Reducing Expected Loss<\/h3>\n
The case of cancer diagnosis shows that a loss incurred when there is misclassification could be of different degrees.<\/p>\n<\/div>\n
So we introduce a loss function or cost function E(L).<\/div>\nSuppose that for a new value of x, the correct class is Ck but we assign it to Cj (where j\u00a0 k). In this case, we have incurred some loss Lkj.<\/div>\nwhere k and j are elements of the loss matrix, (as in the case of cancer diagnosis).<\/div>\nThe optimal solution is the one that minimizes the value of the loss function.<\/div>\n
After this, a decision is made whether to give treatment or not. This is the decision step. This step becomes easy if the inference problem have been solved.<\/div>\n4. The Role of Probability<\/h3>\n
<\/p>\n
4. The Role of Probability<\/h3>\nWhen the x-ray images is obtained x for a new patient, the goal is to determine which of the two classes to assign the image.<\/div>\n<\/div>\nWe could assign it to any of the two classes, C1 or C2 (let’s use Ck, for k = {1,2}). We need to find the probabilities of the two classes given the image. This probability is given by p(Ck | x). Using Bayes theorem, these probabilities can be expressed in the form:<\/div>\n<\/div>\n<\/div>\n<\/a><\/div>\n\n <\/p>\n
5. Minimizing Misclassification<\/h3>\n
Sometimes in classification, we may assign an input to the wrong class. In this case, misclassification have occurred. The goal is to make as few misclassifications as possible.<\/p>\n<\/div>\n
Misclassification occurs when an input variable is assigned to the wrong class. The goal is to minimize the number of misclassifications.<\/div>\n<\/div>\nWe handle this by creating a rule that assigns each x to one of the available classes. This rule would divide the input space into regions Rx called decision regions, one region for each class.<\/div>\nPoints in Rx are assigned to Ck.<\/div>\n<\/div>\nA misclassification when a point in a region is assigned to the wrong class e.g<\/div>\n\n- x is in R1 but is assigned to class C2<\/li>\n
- x is in R2 but is assigned to class C1<\/li>\n<\/ul>\n<\/div>\nThe probability of misclassification is given by:<\/div>\n<\/div>\n<\/a><\/div>\n<\/div>\nTo minimize this error, the rule must assign x<\/b><\/span> to which class has minimum integrand:<\/div>\n<\/div>\np(<\/i>x<\/b>, Ck<\/sub>) = p(Ck<\/sub> | <\/i>x<\/b>)p(<\/i>x<\/b>)<\/i><\/span><\/span><\/div>\n<\/div>\nFrom the above formula, to minimize misclassification, x should be assigned to which class the posterior probability is largest.<\/div>\n<\/div>\n\n
<\/p>\n
6. Reducing Expected Loss<\/h3>\n
The case of cancer diagnosis shows that a loss incurred when there is misclassification could be of different degrees.<\/p>\n<\/div>\n
So we introduce a loss function or cost function E(L).<\/div>\nSuppose that for a new value of x, the correct class is Ck but we assign it to Cj (where j\u00a0 k). In this case, we have incurred some loss Lkj.<\/div>\nwhere k and j are elements of the loss matrix, (as in the case of cancer diagnosis).<\/div>\nThe optimal solution is the one that minimizes the value of the loss function.<\/div>\n
<\/div>\n
We could assign it to any of the two classes, C1 or C2 (let’s use Ck, for k = {1,2}). We need to find the probabilities of the two classes given the image. This probability is given by p(Ck | x). Using Bayes theorem, these probabilities can be expressed in the form:<\/div>\n
<\/div>\n
<\/div>\n<\/a><\/div>\n
<\/a><\/div>\n\n
<\/p>\n
5. Minimizing Misclassification<\/h3>\n
Sometimes in classification, we may assign an input to the wrong class. In this case, misclassification have occurred. The goal is to make as few misclassifications as possible.<\/p>\n<\/div>\n
Misclassification occurs when an input variable is assigned to the wrong class. The goal is to minimize the number of misclassifications.<\/div>\n
<\/div>\n
We handle this by creating a rule that assigns each x to one of the available classes. This rule would divide the input space into regions Rx called decision regions, one region for each class.<\/div>\n
Points in Rx are assigned to Ck.<\/div>\n
<\/div>\n
A misclassification when a point in a region is assigned to the wrong class e.g<\/div>\n
- \n
- x is in R1 but is assigned to class C2<\/li>\n
- x is in R2 but is assigned to class C1<\/li>\n<\/ul>\n<\/div>\nThe probability of misclassification is given by:<\/div>\n<\/div>\n<\/a><\/div>\n<\/div>\nTo minimize this error, the rule must assign x<\/b><\/span> to which class has minimum integrand:<\/div>\n<\/div>\np(<\/i>x<\/b>, Ck<\/sub>) = p(Ck<\/sub> | <\/i>x<\/b>)p(<\/i>x<\/b>)<\/i><\/span><\/span><\/div>\n<\/div>\nFrom the above formula, to minimize misclassification, x should be assigned to which class the posterior probability is largest.<\/div>\n<\/div>\n\n
<\/p>\n
6. Reducing Expected Loss<\/h3>\n
The case of cancer diagnosis shows that a loss incurred when there is misclassification could be of different degrees.<\/p>\n<\/div>\n
So we introduce a loss function or cost function E(L).<\/div>\nSuppose that for a new value of x, the correct class is Ck but we assign it to Cj (where j\u00a0 k). In this case, we have incurred some loss Lkj.<\/div>\nwhere k and j are elements of the loss matrix, (as in the case of cancer diagnosis).<\/div>\nThe optimal solution is the one that minimizes the value of the loss function.<\/div>\n
![](\"https:\/\/2.bp.blogspot.com\/-hw9FmMTepE8\/WmTmfV5MyAI\/AAAAAAAAA5I\/2NjTeoIizvwMa7agrxry9A41FWtGdvzLwCLcBGAs\/s320\/Loss%2BFunction.jpg\")
<\/a><\/div>\n![](\"https:\/\/2.bp.blogspot.com\/-O-Q6lToclZE\/WmTpnOIKQ5I\/AAAAAAAAA5U\/uonr-KNZ1Rs6QtObyDlKdjIbbIyrPvNYACLcBGAs\/s320\/Discriminant%2BFunction.jpg\")