{"id":196,"date":"2018-01-09T14:34:00","date_gmt":"2018-01-09T13:34:00","guid":{"rendered":"https:\/\/kindsonthegenius.com\/blog\/2018\/01\/09\/what-is-over-fitting-in-linear-regression-and-machine-learning\/"},"modified":"2020-08-22T11:04:48","modified_gmt":"2020-08-22T09:04:48","slug":"what-is-over-fitting-in-linear-regression-and-machine-learning","status":"publish","type":"post","link":"https:\/\/kindsonthegenius.com\/blog\/what-is-over-fitting-in-linear-regression-and-machine-learning\/","title":{"rendered":"What is Over-fitting in Linear Regression and Machine Learning"},"content":{"rendered":"
\n
In this short lesson, we will discuss the concept of over-fitting in Linear Regression. For now I would assume you have a basic knowledge of linear regression, where you have to fit a straight line through a set of data points.<\/p>\n
<\/a><\/div>\n

<\/ins> What is Over-fitting?<\/b>
Over-fitting of a model in regression is a condition where the model corresponds too closely to a particular set of data and may therefore not be able to predict new observation. When over-fitting occurs the model begins to describe random error in the data rather than relationship between two variables.<\/p>\n

Polynomial Curve Fitting<\/b>
Assuming we have an input variable x and we want to use this observation to predict a target variable t<\/i><\/span>.
Let’s also assume that we have a training dataset of N observations of the variable x which can be denoted as a vector:<\/p>\n

x = { x1<\/sub>, x2<\/sub>,…xn<\/sub><\/span><\/i>)T<\/sup><\/span><\/i><\/span><\/div>\n

The corresponding training values for t can be denotes as a vector t given by<\/p>\n

t = {t1<\/sub>, t2<\/sub>,…tn<\/sub>)T<\/sup><\/span><\/i><\/span><\/div>\n

Let’s choose values of N = 1 to 10 with intervals of 0,1
Let the relationship between the x and t for the training data set be<\/p>\n

f(x) = sin(2\u03c0x)<\/span><\/i><\/span><\/span><\/div>\n

Now the objective of polynomial fitting to discover this function given just the training observation. <\/span>
To do that, we would use a polynomial function of the form<\/span>

<\/span><\/p>\n

<\/a><\/div>\n

where M is the order of the polynomial<\/span>
xj<\/sub> <\/i><\/span><\/span>denote the power x<\/span><\/i> is raised to<\/span>

<\/span>First we need to choose the value of w to make the error E(w<\/b>)<\/span><\/i> to have a minimum value<\/span>
Next we would choose the value of the order of the polynomial M to get the best fit of our regression model.<\/span>
We already know the plot for the given function y(x) = <\/span><\/i><\/span><\/span>sin(2<\/span><\/i><\/span>\u03c0x)<\/span><\/i><\/span>. So we will first plot it, and then choose M(the order of the polynomial) for different values. For each value, we would plot it in the same plot with the know function and try to choose which of them best fits<\/span><\/span>
We would choose four values. M = 0, 1, 3, 9<\/span><\/i><\/span><\/span><\/span>
From the figure, the green plot is the given model and the red plot is our polynomial curve.<\/span><\/span><\/p>\n\n\n\n\n
M = 1 (Under-fitting)   <\/a><\/td>\nM = 3<\/a><\/td>\n<\/tr>\n
M=3<\/a><\/td>\nM = 9 (Over-fitting)<\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

From the models, we see that for M = 3 provides the best fit.
For M = 9, the model tends to capture all the data points which means the the value of the error E(w) is zero. However, this curve gives a poor representation of the original function and would not be able to capture new data points. This scenario is what is known as over-fitting<\/b><\/i><\/span>.
<\/ins>
<\/ins> <\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

In this short lesson, we will discuss the concept of over-fitting in Linear Regression. For now I would assume you have a basic knowledge of … <\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0},"categories":[15,11,16],"tags":[],"_links":{"self":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/196"}],"collection":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/comments?post=196"}],"version-history":[{"count":1,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/196\/revisions"}],"predecessor-version":[{"id":1467,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/196\/revisions\/1467"}],"wp:attachment":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/media?parent=196"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/categories?post=196"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/tags?post=196"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}