{"id":1901,"date":"2019-04-11T12:00:00","date_gmt":"2019-04-11T10:00:00","guid":{"rendered":"https:\/\/kindsonthegenius.com\/blog\/machine-learning-101-overfitting-and-underfitting\/"},"modified":"2026-07-05T03:22:43","modified_gmt":"2026-07-05T01:22:43","slug":"machine-learning-101-overfitting-and-underfitting","status":"publish","type":"post","link":"https:\/\/kindsonthegenius.com\/blog\/machine-learning-101-overfitting-and-underfitting\/","title":{"rendered":"Machine Learning 101 \u2013 Overfitting and Underfitting"},"content":{"rendered":"

Remember that in the previous lecture (Lecture 6),<\/a> we\u00a0 discuss polynomial curve fitting. We kind of saw that the relationship in a dataset can be represented using a polynomial of this form:<\/p>\n

 <\/p>\n

\"Polynomial<\/p>\n

 <\/p>\n

Just assume that this is any polynomial of order, M. We want to understand want happens when the order gradually increases from 0 to say 13.<\/p>\n

Also assume that the w terms are constant.<\/p>\n

Remember our goal is to ensure that this polynomial fits the dataset. That is, after plotting the dataset as a scatter plot, then we fit this polynomial through it. We’ll then cover the following:<\/p>\n

    \n
  1. How M Relates to E(Error)<\/a><\/li>\n
  2. If M is too Low<\/a><\/li>\n
  3. If M is too High<\/a><\/li>\n
  4. The Trade-off<\/a><\/li>\n
  5. Hands-on in Python<\/a><\/li>\n<\/ol>\n

     <\/p>\n

    1. How M relates to E<\/strong><\/h4>\n

    Also recall that we would like to minimize the error term E(x, w*). This is so that the difference between predicted values and the actual values is very small.<\/p>\n

    Now as M (the order of the polynomial) increases, the error, E decreases. This also means that the model complexity increases since we have higher order polynomial.<\/p>\n

    Similarly, when M is low, the complexity of the model reduces, meaning that the model becomes kind of simple. (I recommend you take some time to understand this relationship).<\/p>\n

    So it then appear that to reduce the error E to the barest minimum, or even zero, we simply keep increasing M. But there a problem.<\/p>\n

    Let’s now see what happens when M is too low and when M is too high.<\/p>\n

     <\/p>\n

    2. If M is very low (Underfitting)<\/strong><\/h4>\n

    As mentioned, if M is very low, then\u00a0 polynomial is will not be able to properly model the relationship between the variable.\u00a0 So the following will be true:<\/p>\n