Linear Models for Regression(Linear Basis Function Models)

This article discusses the Linear Models of Regression with focus on Linear basis function model.


  • Background of Linear Regression
  • The Regression Problem
  • Linear Function Model
  • Constructing the Basis Function
  • Introducing a non-linear function
  • Final Notes

Other discussions can be found in this link: Best 23 Easy Tutorials on Machine Learning and Artificial Intelligence(Easy to Understand)

Remember that Regression is of the technique under Supervised Learning. The other is Classification. The objective of the regression model is to determine the value of one or more of a target variable t, given the value of a D-dimensional vector, x of input variables. In other words, you need to find the function that relates the input and the output. This can be done using Linear Models.
One of such modes is the polynomial curve fitting which gives a function that is a linear function of a particular parameter.
A better model is the Linear Basis Function. This is discussed next.

The Linear Basis Function
Given a set of input dataset of N samples {xn}, where n = 1, … , N, as well as the corresponding target values {tn}, the goal is to deduce the value of t for new value of x. The set of input data set together with the corresponding target values t is known as the training data set.

On way to handle this is by constructing a function y(x) that maps x to t such that:
y(x) = t
for a new input value of x.
Then we can examine this model by finding the probability that the results are correct. This means that we need to examine the probability of t given x 

Constructing the Linear Basis Function
The basic linear model for regression is a  model that involves a linear combination of the input variables:

y(w,x) = wo + w1x1 + w2x2 + … + wDxD

where x = (x1, x2, … ,xD)T

This is what is generally known as linear regression.
The key attribute of this function is that it is a linear function of the parameters w0, w1,…, wD. It is also a linear function of the input variable x. Being a linear  function of the input variable x, limits the usefulness of the function. This is because most of the observations that may be encountered does not necessarily follow a linear relationship. To solve this problem consider modifying to model to be a combination of fixed non-linear functions of the input variable.

If we assume that the non-linear function of the input variable is φ(x), then we can re-write the original function as :

y(x,w) = w0 + w1φ(x1) + w2φ(x2) + … + wDφ(xD

Summing it up, we will have:

where φ(x) are known as basis functions.

The total number of parameters in this function will be M, therefore the summation of terms is from j = 1 to M.
The parameter w0  is known as the bias parameter which allows for a fixed offset in the data. 

(Quiz: Can you remember any other topic the word bias appears? Leave it in comment on the left of the page!)

Final Notes
The topic for linear models of regression covers much more than what is presented here. But what we have discussed is the basics of the Linear Basis Function Model and I have decided to keep it simple and clear so that you can easily understand it and possibly pass an oral exam.
But if you  are a Math student and want to delve further, I would recommend reading chapter three of the book. ‘Pattern Recognition and Machine Learning‘, by Christopher Bishop.