{"id":1910,"date":"2019-04-18T12:00:00","date_gmt":"2019-04-18T10:00:00","guid":{"rendered":"https:\/\/kindsonthegenius.com\/blog\/machine-learning-101-k-nearest-neighbors-classifier\/"},"modified":"2026-07-05T03:23:02","modified_gmt":"2026-07-05T01:23:02","slug":"machine-learning-101-k-nearest-neighbors-classifier","status":"publish","type":"post","link":"https:\/\/kindsonthegenius.com\/blog\/machine-learning-101-k-nearest-neighbors-classifier\/","title":{"rendered":"Machine Learning 101 \u2013 K-Nearest Neighbors Classifier"},"content":{"rendered":"

In the last lecture, we discussed Bayes’ Classifier<\/a>. Now, we are going to discuss K-Nearest Neighbors Classifier.<\/p>\n

Remember that Bayes Classifier tries to classify X depending on the conditional probability of Y given X. However, the conditional distribution of Y over X is not known. Therefore, we can’t actually use Bayes Classifier in practical scenarios.<\/p>\n

One approach would be to estimate the conditional distribution of Y given X. Using this, we then classify any observation to the class with the highest estimated probability.<\/p>\n

This is how the K-nearest neighbors (KNN) classifier works. Let’s now examine KNN more closely.<\/p>\n

 <\/p>\n

How KNN Works<\/strong><\/h4>\n

Start by choosing initial value of and integer K. That is a certain number of data points in the training data. Then choose test observation, say x0<\/sub>.<\/p>\n

Next, the KNN identifies the first K points that are closest to x0<\/sub>. These points form a region N0<\/sub>. Then KNN estimates the conditional probability for a class j to be the fraction of point in N0<\/sub> whose response value equals j. That is points that belong to\u00a0 class j.<\/p>\n

This conditional probability is written as:<\/p>\n

 <\/p>\n

\"K-Nearest<\/p>\n

This equation reads as:<\/p>\n

The sum over N0\u00a0<\/sub>of the conditional probabilities of Y = j given x0<\/sub><\/p>\n

Finally, Bayes rule is applied to classify the test observation x0<\/sub> to the class with the largest probability.<\/p>\n

 <\/p>\n

Illustrating K-Nearest Neighbors<\/strong><\/h4>\n

Let’s illustrate KNN using an example.<\/p>\n

In Figure 1 below, we have a plot of the training data set. It’s made up of 6 blue observations and 6 orange observations. Now, we would like to classify the data point marked with a black cross (x).<\/p>\n

We would take the following steps:<\/p>\n

Step 1:<\/strong> We choose the value of K = 3<\/p>\n

Step 2:<\/strong> Identify 3 observations that are nearest to the cross. This is shown enclosed in a green circle. It has two blue points and one orange point<\/p>\n

Step 3:<\/strong> Estimate the probability for each class given the data point (marked with cross) we are trying to classify.<\/p>\n

P(blue class | observation) = 2\/3<\/em><\/p>\n

P(orange class\u00a0 | observation) = 1\/3<\/em><\/p>\n

 <\/p>\n

\"K-Nearest<\/p>\n

 <\/p>\n

Step 4:<\/strong> Draw a conclusion. Since the the blue class has the highest probability given the observation, therefore we classify the black cross as belonging to the blue class.<\/p>\n

This process is repeated until all the datapoints is classified.\u00a0 I recommend you watch the video explanation of this.<\/p>\n

However, while K-nearest neighbor does well in classification, it is possible that misclassification can could occur. In the next lesson<\/a> we’ll see how to minimize misclassification<\/p>\n","protected":false},"excerpt":{"rendered":"

In the last lecture, we discussed Bayes’ Classifier. Now, we are going to discuss K-Nearest Neighbors Classifier. Remember that Bayes Classifier tries to classify X … <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"pagelayer_contact_templates":[],"_pagelayer_content":"","footnotes":""},"categories":[16],"tags":[],"class_list":["post-1910","post","type-post","status-publish","format-standard","hentry","category-machine-learning"],"_links":{"self":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1910","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/comments?post=1910"}],"version-history":[{"count":1,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1910\/revisions"}],"predecessor-version":[{"id":2078,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1910\/revisions\/2078"}],"wp:attachment":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/media?parent=1910"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/categories?post=1910"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/tags?post=1910"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}