{"id":1868,"date":"2018-12-03T12:00:00","date_gmt":"2018-12-03T11:00:00","guid":{"rendered":"https:\/\/kindsonthegenius.com\/blog\/support-vector-machines-in-machine-learning-a-simple-tutorial\/"},"modified":"2026-07-05T03:21:20","modified_gmt":"2026-07-05T01:21:20","slug":"support-vector-machines-in-machine-learning-a-simple-tutorial","status":"publish","type":"post","link":"https:\/\/kindsonthegenius.com\/blog\/support-vector-machines-in-machine-learning-a-simple-tutorial\/","title":{"rendered":"Support Vector Machines in Machine Learning ( A Simple Tutorial)"},"content":{"rendered":"

Just as you know, I would try to explain Support Vector Machines (SVM)\u00a0 in a vary simple and clear way. I know many find it a bit tough, but I tell you, you can understand it with little effort. The lesson would be in two parts.<\/p>\n

So let’s dive in!<\/p>\n

\"\"<\/p>\n

We would cover the following topics:<\/p>\n

    \n
  1. What are Support Vector Machines?<\/a><\/li>\n
  2. Maximum Margin Classifiers<\/a><\/li>\n
  3. Concept of Hyperplane<\/a><\/li>\n
  4. Equation of a Hyperplane<\/a><\/li>\n
  5. Above and Below the Hyperplane<\/a><\/li>\n
  6. Performing Classification using a Hyperplane<\/a><\/li>\n
  7. Separating Hyperplane<\/a><\/li>\n<\/ol>\n

     <\/p>\n

     <\/p>\n

    1.\u00a0 What are Support Vector Machines<\/h4>\n

    As you already know, classification is one important aspect of Machine Learning. Also recall that Supervised Learning is divided into Classification, Regression and Density Estimation.<\/p>\n

    Support Vector Machines (SVM) fall under Classification. In fact SVM is a method used for classification and can be considered one of the most best classifiers.<\/p>\n

     <\/p>\n

     <\/p>\n

    2. Maximum-Margin Classifiers<\/h4>\n

    SVM come under this type of classifiers. They tent to perform classification by finding the largest margin between the two classes. The challenge with this type of classifiers is that it requires that data be linearly separable.\u00a0 That means that, some kind of line could be fitted to separate the two classes. This is illustrated in Figure 1.0<\/p>\n

    \"\"
    Figure 1.0: Linear Separability<\/figcaption><\/figure>\n

    One good application of SVM is in binary classification, where we need to classify data into two distinct classes.<\/p>\n

     <\/p>\n

     <\/p>\n

    3.\u00a0 The Concept of Hyperplane<\/h4>\n

    I learning Support Vector Machines, you need to clearly understand the term Hyperplane. So let me just give a formal definition: A hyperplane is a plane which is one dimension less than the ambient plane. That means that if we have data in two dimensions, then the hyperplane would be a single line as in Figure 1.0. In the Figure 1.0. the blue lines are hyperplanes.<\/p>\n

    In the same way, if we have data in 3-dimensional space, then the hyperplane would be a 2-dimensional plane. And so on.<\/p>\n

     <\/p>\n

     <\/p>\n

    4. Equation of a Hyperplane<\/h4>\n

    You already know that a line, or a plane have an equation that defines it. In the same way, a hyperplane is defined by the equation:<\/p>\n

    \u03b20<\/sub> +\u00a0\u03b21<\/sub>X1<\/sub> +\u00a0\u03b22<\/sub>X2<\/sub>\u00a0= 0<\/span><\/p>\n

    In this equation, X<\/strong> = (X<\/strong>1<\/sub>, X<\/strong>2<\/sub>) holds for all the points along this line<\/p>\n

    As you can recall, this equation is the equation of a line\u00a0 and defines a 2-dimensional hyperplane. We can extend the equation to a hyperplane of arbitrary dimension, say p.<\/p>\n

    \u03b20<\/sub> +\u00a0\u03b21<\/sub>X1<\/sub> +\u00a0\u03b22<\/sub>X2<\/sub> + . . . +\u00a0\u03b2p<\/sub>Xp<\/sub>\u00a0 = 0<\/span><\/p>\n

    This is the equations for a p-dimensional hyperplane. In the same way, the point X = (X1<\/sub>, X2<\/sub>, . . . , Xp<\/sub>) holds true for all the points on this hyperplane.<\/p>\n

     <\/p>\n

     <\/p>\n

    5. Above and Below the Hyperplane<\/h4>\n

    So far we know the equation for the line or plane which defines the hyperplane. Now consider a set of datapoints. If we are able to draw a hyperplane across these points, we can be sure of three things;<\/p>\n

      \n
    • some points may lie exactly on the line (hyperplane)<\/li>\n
    • some points lie above the hyperplane<\/li>\n
    • some points lie below the hyperplane.<\/li>\n<\/ul>\n

      This is illustrated in Figure 1.3.<\/p>\n

      \"\"
      Figure 1.3<\/figcaption><\/figure>\n

      Now have the equation for the hyperplane. We are also interested in the points that lie on both sides of the hyperplane (above and below). These point can be described with these equations:<\/p>\n

      \u03b20<\/sub> +\u00a0\u03b21<\/sub>X1<\/sub> +\u00a0\u03b22<\/sub>X2<\/sub> + . . . +\u00a0\u03b2p<\/sub>Xp<\/sub>\u00a0 < 0<\/span> for points that lie below the hyperplane<\/p>\n

      \u03b20<\/sub> +\u00a0\u03b21<\/sub>X1<\/sub> +\u00a0\u03b22<\/sub>X2<\/sub> + . . . +\u00a0\u03b2p<\/sub>Xp<\/sub>\u00a0 > 0<\/span> for points that lie above the hyperplane<\/p>\n

      This make so much sense since the function can equate to 0, greater than 0 or less than 0.<\/p>\n

      This also tells us something about the signs of the classes.<\/p>\n

        \n
      • Points below the hyperplane are associated with -ve<\/li>\n
      • Points on the hyperplane are associated with 0<\/li>\n
      • Points above the hyperplane are associated with +ve<\/li>\n<\/ul>\n

        Out comes our classifier! If the hyperplane divides the data points into two classes, then one class is -ve, the other class is -ve.<\/p>\n

         <\/p>\n

         <\/p>\n

        6. Performing Classification Using Hyperplane<\/h4>\n

        Let’s assume we have a dataset of n rows (observations) and p features. This would server as our training dataset. This dataset can be represented as a matrix of n x p dimension like this:<\/p>\n

        \"\"<\/p>\n

        which means that we can represent each of the observations using a column vector containing all the features ( this is also called a feature vector)<\/p>\n

        \"\"<\/p>\n

        Lets also define two classes y1<\/sub>, y2<\/sub>, . . . ,yn<\/sub><\/span> which could either be 1 or -1. That is<\/p>\n

        y1<\/sub>, y2<\/sub>, . . . ,yn<\/sub>\u00a0\u2208 {1, -1}<\/span><\/p>\n

        Let’s also another dataset, a test dataset, similar to the training dataset with with fewer observations and without labels or classes, only features. The goal just like in classification is to build a classifier that would correctly classify the test data using only the features.<\/p>\n

        In Support Vector Machine, the approach we would use is called the Separating Hyperplane and that is what we would discuss next<\/p>\n

         <\/p>\n

         <\/p>\n

        7. Separating Hyperplane<\/h4>\n

        The idea is to find a hyperplane that can separate the data into two classes such that one part of the data belongs to class 1 and the other part of the dataset belongs to class -1.<\/p>\n

        This hyperplane is the separating hyperplane and would satisfy the following properties<\/p>\n

         <\/p>\n

        \u03b20<\/sub> +\u00a0\u03b21<\/sub>X1<\/sub> +\u00a0\u03b22<\/sub>X2<\/sub> + . . . +\u00a0\u03b2p<\/sub>Xp<\/sub>\u00a0 < 0 if yi =1<\/span><\/p>\n

        \u03b20<\/sub> +\u00a0\u03b21<\/sub>X1<\/sub> +\u00a0\u03b22<\/sub>X2<\/sub> + . . . +\u00a0\u03b2p<\/sub>Xp<\/sub>\u00a0 > 0 if yi = -1<\/span><\/p>\n

        Additionally the separting hyperplane would also satisfy the equation:<\/p>\n

        yi<\/sub>(\u03b20<\/sub> +\u00a0\u03b21<\/sub>X1<\/sub> +\u00a0\u03b22<\/sub>X2<\/sub> + . . . +\u00a0\u03b2p<\/sub>Xp<\/sub>\u00a0 > 0 if yi<\/sub>) > 0<\/span><\/p>\n

        for all i = 1, . . . , n<\/p>\n

        I we succeed in creating this classifier using only out training dataset, then we can used it to classify the test dataset.\u00a0 The classification would be based on which side of the hyperplane a datapoint is located.<\/p>\n

        What we simply need to to is to calculate f(x*) for each observation in the test data where f(x*) is given by<\/p>\n

        f(x*) =\u00a0\u03b20<\/sub> +\u00a0\u03b21<\/sub>X1<\/sub> +\u00a0\u03b22<\/sub>X2<\/sub> + . . . +\u00a0\u03b2p<\/sub>Xp<\/sub><\/span><\/p>\n

        If f(x*) is positive, then we assign the observation to class 1 but if f(x*) is negative, we assign it to the class -1.<\/p>\n

        Additionally, we could also use the value of f(x*) to determine how far the data point is from the hyperplane. The larger the value, the farther it is from the\u00a0 hyperplane.<\/p>\n

        Now, we clearly understand the concept of separating hyperplane as well as how to use a separating hyperplane for classification. The next concept would explain is the maximum-margin-hyperplane.<\/p>\n

        We continue in the next Tutorial.<\/p>\n","protected":false},"excerpt":{"rendered":"

        Just as you know, I would try to explain Support Vector Machines (SVM)\u00a0 in a vary simple and clear way. I know many find it … <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"pagelayer_contact_templates":[],"_pagelayer_content":"","footnotes":""},"categories":[16],"tags":[],"class_list":["post-1868","post","type-post","status-publish","format-standard","hentry","category-machine-learning"],"_links":{"self":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1868","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/comments?post=1868"}],"version-history":[{"count":1,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1868\/revisions"}],"predecessor-version":[{"id":2037,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1868\/revisions\/2037"}],"wp:attachment":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/media?parent=1868"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/categories?post=1868"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/tags?post=1868"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}