{"id":1909,"date":"2019-04-20T12:00:00","date_gmt":"2019-04-20T10:00:00","guid":{"rendered":"https:\/\/kindsonthegenius.com\/blog\/machine-learning-101-minimizing-misclassification-rate-in-bayes-classifier\/"},"modified":"2026-07-05T03:23:00","modified_gmt":"2026-07-05T01:23:00","slug":"machine-learning-101-minimizing-misclassification-rate-in-bayes-classifier","status":"publish","type":"post","link":"https:\/\/kindsonthegenius.com\/blog\/machine-learning-101-minimizing-misclassification-rate-in-bayes-classifier\/","title":{"rendered":"Machine Learning 101 \u2013 Minimizing Misclassification Rate in Bayes\u2019 Classifier"},"content":{"rendered":"<p>In <a href=\"https:\/\/kindsonthegenius.com\/tempsite\/machine-learning-101-the-bayes-classfier\/\">Lecture 4<\/a>, we learnt about the Bayes&#8217; classifier. Here\u00a0 we would see how to minimize misclassfication rate in Bayes classifier. Again, we would review the cancer diagnosis example.<\/p>\n<p>&nbsp;<\/p>\n<p><strong>Review of Cancer Diagnosis Example<\/strong><\/p>\n<p>In this example, the doctors need to determine if the patient has cancer or now. To make this decision, they take an X-ray image of the patient. The output of this image is represented a a vector <strong>x<\/strong> which represents a set of pixels from the X-ray image.<\/p>\n<p>So this is a typical classification problem.\u00a0 They need to classify the patient into one of two classes:<\/p>\n<ul>\n<li>C<sub>1<\/sub> &#8211; cancer present<\/li>\n<li>C<sub>2<\/sub> &#8211; cancer absent<\/li>\n<\/ul>\n<p>If we let k = 1, 2, then we can represent the classes as C<sub>k<\/sub>.<\/p>\n<p>So, given the patient&#8217;s X-ray image x, what is the probability of Ck (that is cancer is present or absent)?<\/p>\n<p>We can represent this as:<\/p>\n<p><em>p(C<sub>k<\/sub> | <strong>x<\/strong>)<\/em><\/p>\n<p>Do you remember that this is conditional probability?<\/p>\n<p>And by Bayes&#8217; Theorem, this is gotten using the formula:<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-762 aligncenter\" src=\"https:\/\/www.kindsonthegenius.com\/wp-content\/uploads\/2020\/09\/Bayes-Theorem-for-Cancer-1-300x71.jpg\" alt=\"\" width=\"300\" height=\"71\" \/><\/p>\n<p>&nbsp;<\/p>\n<p><strong>When Misclassification occurs<\/strong><\/p>\n<p>Misclassification occurs if cancer is present but the doctors decide that it is not.<\/p>\n<p>Representing this with notations, we say that:<\/p>\n<ul>\n<li>misclassification occurs if the patient belongs to class C<sub>2<\/sub>, but the doctors assign him to C<sub>1<\/sub> (false positive)<\/li>\n<li>Or the patient belongs to C<sub>1<\/sub> but the doctors assign him to C<sub>2<\/sub> (false negative)<\/li>\n<\/ul>\n<p><strong>Misclassification Rate<\/strong><\/p>\n<p>This is number of misclassifications divided by total number evaluated<\/p>\n<p>&nbsp;<\/p>\n<h4><strong>Minimizing Misclassification Rate<\/strong><\/h4>\n<p>First, we need to partition the input space into regions. (input space is all possible values <strong>x<\/strong> can take).<\/p>\n<p>These regions are called decision regions. Let&#8217;s represent them using R<sub>k<\/sub>. This is because, the number of decision regions equals number of classes. So one region for each class.<\/p>\n<p>Now, all points in R<sub>k<\/sub> are assigned to C<sub>k<\/sub>.<\/p>\n<p>The boundary between the decision regions are known as <strong>decision surfaces<\/strong> or <strong>decision boundaries<\/strong>.<\/p>\n<p>So we need to a rule that would take each input x, and assign it to a region.<\/p>\n<p>From our example, a mistake can occur in two possible ways:<\/p>\n<ul>\n<li><strong>x<\/strong> belongs to R<sub>1<\/sub> but assigned to C<sub>2<\/sub><\/li>\n<li><strong>x<\/strong> belongs to R<sub>2<\/sub> bu assigned to C<sub>2<\/sub><\/li>\n<\/ul>\n<p>The probability of a mistake occurring is now, the sum of this two probabilities (remember the sum rule). This is given as<\/p>\n<p><em>p(mistake) = p(<strong>x<\/strong>\u00a0\u2208 R<sub>1<\/sub>, C<sub>2<\/sub>) + p(<strong>x<\/strong>\u00a0\u2208 R<sub>2<\/sub>, C<sub>1<\/sub>)<\/em><\/p>\n<p>This can be broken down by taking these probabilities over the regions R<sub>1<\/sub>, and R<sub>2<\/sub>. Therefor we have:<\/p>\n<p>&nbsp;<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-765 \" src=\"https:\/\/www.kindsonthegenius.com\/wp-content\/uploads\/2020\/09\/Probability-of-Misclassification-in-Bayes-Classifier-2.jpg\" alt=\"Probability of Misclassification in Bayes' Classifier\" width=\"522\" height=\"85\" \/><\/p>\n<p>&nbsp;<\/p>\n<p>Clearly, to\u00a0 minimize p(mistake), we need to try to assign x to whichever class has the smaller value of the integrand.<\/p>\n<p>Therefore, if <em>p(<strong>x<\/strong>, C<sub>1<\/sub>) &lt; p(<strong>x<\/strong>, C<sub>2<\/sub><\/em>) for a given <strong>x<\/strong>, then we should assign<em> x<\/em> to C<sub>1<\/sub>.<\/p>\n<p>Let&#8217;s evaluate p(mistake) further using the <a href=\"https:\/\/kindsonthegenius.com\/tempsite\/machine-learning-101-rules-of-probability-bayes-theorem\/\">product rule<\/a>.<\/p>\n<p>We know from the product rule that:<\/p>\n<p><em>p(<strong>x<\/strong>, C<sub>2<\/sub>) = p(C<sub>2<\/sub> | <strong>x<\/strong>)p(<strong>x<\/strong>)<\/em><\/p>\n<p>also<\/p>\n<p><em>p(<strong>x<\/strong>, C<sub>1<\/sub>) = p(C<sub>1<\/sub> | <strong>x<\/strong> ) p(<strong>x<\/strong>)<\/em><\/p>\n<p>If we use this in the formula for p(mistake), we would then have:<\/p>\n<p>&nbsp;<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-767 aligncenter\" src=\"https:\/\/www.kindsonthegenius.com\/wp-content\/uploads\/2020\/09\/Probability-of-Misclassification-in-Bayes-Classifier-using-Conditional-Probability.jpg\" alt=\"Probability of Misclassification in Bayes Classifier using Conditional Probability\" width=\"566\" height=\"80\" \/><\/p>\n<p>&nbsp;<\/p>\n<p>You can now see that the terms in the integral is either p(C<sub>2<\/sub> | <strong>x<\/strong>) d<strong>x<\/strong> or<\/p>\n<p>p(C<sub>1<\/sub> | <strong>x<\/strong>) d<strong>x<\/strong><\/p>\n<p>These conditional probabilities can work if we already know the posterior probabilities of C<sub>1<\/sub> and C<sub>2<\/sub> after obtaining x.<\/p>\n<p>Therefore, the probability\u00a0 of misclassification p(mistake) is minimized if each value of x is assigned to the class with the largest posterior probability.<\/p>\n<p>&nbsp;<\/p>\n<p><strong>Probability of Correct Classification<\/strong><\/p>\n<p>Similarly, we can apply the same method to obtain <em>p(correct).<\/em><\/p>\n<p>I recommend this as an exercise for you.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In Lecture 4, we learnt about the Bayes&#8217; classifier. Here\u00a0 we would see how to minimize misclassfication rate in Bayes classifier. Again, we would review &hellip; <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"pagelayer_contact_templates":[],"_pagelayer_content":"","footnotes":""},"categories":[16],"tags":[],"class_list":["post-1909","post","type-post","status-publish","format-standard","hentry","category-machine-learning"],"_links":{"self":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1909","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/comments?post=1909"}],"version-history":[{"count":1,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1909\/revisions"}],"predecessor-version":[{"id":2077,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1909\/revisions\/2077"}],"wp:attachment":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/media?parent=1909"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/categories?post=1909"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/tags?post=1909"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}