{"id":176,"date":"2018-01-21T11:54:00","date_gmt":"2018-01-21T10:54:00","guid":{"rendered":"https:\/\/kindsonthegenius.com\/blog\/2018\/01\/21\/how-to-reduce-expected-loss-in-classification-machine-learning\/"},"modified":"2020-08-22T09:22:48","modified_gmt":"2020-08-22T07:22:48","slug":"how-to-reduce-expected-loss-in-classification-machine-learning","status":"publish","type":"post","link":"https:\/\/kindsonthegenius.com\/blog\/how-to-reduce-expected-loss-in-classification-machine-learning\/","title":{"rendered":"How to Reduce Expected Loss in Classification (Machine Learning)"},"content":{"rendered":"
\n
This lessons explains in simple terms how to minimize expected loss during classification.
Remember that when an input variable is classified wrongly, a loss is incurred. However, this loss can be large or can be minimal<\/p>\n
<\/a><\/div>\n

Example of Cancer<\/b><\/span><\/span>
Scenario 1<\/b><\/span>: A patient that does not have cancer is misclassified as having cancer
Scenario 2<\/span><\/b>: A patient that has cancer is misclassified as not having cancer.<\/p>\n

For these two misclassification scenarios, we can see that the loss incurred by the second scenario can be enormous, which may actually be death of the patient.<\/p>\n

So the objective is to reduce such misclassifications that would result in much loss. That means reducing the expected loss<\/p>\n

Formalism<\/b><\/span><\/span><\/span>
We can formalize the discussion by introducing a loss function<\/i><\/span> also known as cost function.<\/i><\/span> The loss function<\/i><\/span> is a single overall measure of the  loss incurred in taking any of the available decisions.<\/p>\n

The goal is to minimize this total loss incurred.<\/p>\n

The Loss Matrix<\/b><\/span><\/span>
The Loss Matrix is a table showing the decision that was taken relative to the true class. This is shown in Figure 1:<\/p>\n\n\n\n\n
<\/a><\/td>\n<\/tr>\n
Figure 1: Loss Matrix for the Cancer Example<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

The elements of the  loss matrix represents the loss incurred by taking one of the available decision. The rows of the loss matrix represents the true class while the columns of the matrix represents the assignment of class based on our decision<\/p>\n

Assuming that for a new value of x, we assign it to class Cj<\/sub><\/i><\/span> whereas the real correct class is Ck<\/sub><\/span><\/span><\/i>. It means we have incurred a loss Lkj<\/sub><\/span><\/i>, which is the k, j<\/span><\/i> element of the loss matrix.
The average loss function is given by the equation:<\/p>\n

<\/a><\/div>\n
<\/div>\n

The best solution is one that minimizes the average loss function. For a given input vector x, our uncertainty in the correct class is expressed through the joint probability distribution p(<\/span><\/span><\/i>x<\/span><\/span><\/b>, Ck<\/sub>).<\/span><\/span><\/i><\/p>\n

As before, we can use the product rule which states that<\/p>\n

p(<\/span><\/i>x<\/span><\/b>, Ck<\/sub>) = p(Ck<\/sub> | <\/span><\/i>x<\/b><\/span>)p(x) <\/span><\/i><\/span><\/div>\n
<\/div>\n

to eliminate the common factor p(x). Therefore, the decision rule that minimizes the expected loss is one that assigns x to the class for which the quantity:<\/p>\n

<\/a><\/div>\n

is minimum.
This can be found once we know the posterior class probabilities p(Ck<\/sub> | <\/span><\/i>x<\/span><\/b>)<\/span><\/i><\/span><\/p>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

This lessons explains in simple terms how to minimize expected loss during classification.Remember that when an input variable is classified wrongly, a loss is incurred. … <\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0},"categories":[16],"tags":[],"_links":{"self":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/176"}],"collection":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/comments?post=176"}],"version-history":[{"count":1,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/176\/revisions"}],"predecessor-version":[{"id":1441,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/176\/revisions\/1441"}],"wp:attachment":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/media?parent=176"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/categories?post=176"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/tags?post=176"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}