{"id":191,"date":"2018-01-11T04:53:00","date_gmt":"2018-01-11T04:53:00","guid":{"rendered":"https:\/\/kindsonthegenius.com\/blog\/2018\/01\/11\/bias-variance-trade-off-in-classificationmachine-learning\/"},"modified":"2019-04-11T13:42:21","modified_gmt":"2019-04-11T11:42:21","slug":"bias-variance-trade-off-in-classificationmachine-learning","status":"publish","type":"post","link":"https:\/\/kindsonthegenius.com\/blog\/bias-variance-trade-off-in-classificationmachine-learning\/","title":{"rendered":"Bias\/Variance Trade-off in Classification(Machine Learning)"},"content":{"rendered":"

In this lesson, you will learn about Bias\/Variance Trade-off in Machine Learning.
\nThis is a concept in machine learning which refers to the problem of minimizing two error sources at the same time and this prevents the supervised learning algorithms from generalizing to accommodate inputs beyond the original training set.<\/p>\n

\u00a0<\/a><\/div>\n

We would discuss the following:<\/strong><\/p>\n

    \n
  1. What is Bias\/Variance Tradeoff (Definition)?<\/a><\/li>\n
  2. Sources of Error<\/a><\/li>\n
  3. Bias\/Variance Decomposition of Squared Error<\/a><\/li>\n
  4. Trade-off: Minimizing Error<\/a><\/li>\n
  5. Relationship to Underfitting and Overfitting<\/a><\/li>\n
  6. Illustration of Bias-Variance Trade-off<\/a><\/li>\n
  7. The Bias\/Variance Trade-off Graph<\/a><\/li>\n
  8. Summary and Final Notes <\/a><\/li>\n<\/ol>\n

     <\/p>\n

    \n

     <\/p>\n

    1. Definition of Bias-Variance Trade-off<\/strong><\/h4>\n

    First, let’s take a simple definition. Bias-Variance Trade-off refers to the property of a machine learning model such that as the bias of the model increased, the variance reduces and as the bias reduces, the variance increases.<\/p>\n<\/div>\n

    Therefore the problem is to determine the amount of bias and variance to make the model optimal.<\/div>\n
    <\/div>\n

     <\/p>\n

     <\/p>\n

    2. Sources of Error<\/strong><\/h4>\n

    We recall the problem of underfitting and overfitting when trying to fit a regression line through a set of data points.
    \nIn case of underfitting, the bias is an error from a faulty assumption in the learning algorithm. This is such that when the bias is too large, the algorithm would be able to correctly model the relationship between the features and the target outputs.<\/p>\n

    In case of overfitting, variance is an error resulting from fluctuations int he training dataset. A high value for variance would cause\u00a0 the algorithm may capture the most data points put would not be generalized enough to capture new data points. This is overfitting.<\/p>\n

    The trade-off, means that a model would be chosen carefully to both correctly capture that regularities in the training data and at the same time be generalized enough to correctly categorize new observation<\/p>\n

     <\/p>\n

     <\/p>\n

    3. Bias-Variance Decomposition of Squared Error<\/strong><\/h4>\n

    Considering the squared loss function and the conditional distribution of the training data set, we could summarize the formula for the expected loss to be:<\/p>\n

    Expected Loss = (bias)2<\/sup> + variance\u00a0+ noise<\/i><\/b><\/span><\/span><\/div>\n

    Now assuming y = f(x)<\/span><\/span><\/i>\u00a0 representing the true relationship between the variables in the training data set
    \nAlso let function f'(x)<\/span><\/i><\/span> which is an approximation of f(x)<\/i><\/span><\/span> through the learning process<\/p>\n

    Then we\u00a0 measure the mean squared error between y<\/i><\/span><\/span> and f'(x)<\/i><\/span><\/span> which is given as:
    \n(y – f'(x))2<\/sup>.\u00a0<\/span><\/i><\/span><\/span>
    \nThis error is expected to be minimal.<\/p>\n

    We than then write the original expected loss equation as:<\/p>\n

    E[(y – f'(x))2<\/sup>] = Bias[f'(x)]2<\/sup>\u00a0+ Var[f'(x)]\u00a0+ \u03c32<\/sup><\/span><\/i><\/span><\/span><\/span><\/p>\n

    where:
    \nBias[f'(x)] = <\/i>E[(f'(x) – f(x)] <\/i><\/i><\/span><\/span><\/span><\/p>\n

    and
    \nVar[f'(x)] = <\/i>E[f'(x)<\/i><\/i><\/span><\/span><\/span>2<\/sup>]<\/i><\/span><\/span><\/span> – <\/i><\/i><\/span><\/span><\/span>E[f'(x)<\/i><\/i><\/span><\/span><\/span>]2<\/sup><\/i><\/i><\/span><\/span><\/span><\/p>\n

    and
    \n \u03c32<\/sup><\/span><\/i><\/span><\/span><\/span>\u00a0 represents the noise term in the equation<\/p>\n

     <\/p>\n

     <\/p>\n

    4. The Bias\/Variance Tradeoff<\/strong><\/h4>\n

    The objective is to reduce the error E to the minimum.\u00a0 This can be done by modifying the terms of the mean square error. From the equation, we see that we could only modify the bias and the variance terms.
    \nBias arises when we generalize relationships using a function, while variance arises when there are multiple samples or input.
    \nOne way to reduce the error is to reduce the bias and the variance terms. However, we cannot reduce both terms simultaneously, since reducing one term leads to increase in the other term. This is the idea of bias variance trade\/off.<\/p>\n

     <\/p>\n

     <\/p>\n

    5. Relationship with Underfitting and Overfitting<\/strong><\/h4>\n

    A good model should do one of two things<\/p>\n