{"id":12,"date":"2018-11-24T23:53:00","date_gmt":"2018-11-24T22:53:00","guid":{"rendered":""},"modified":"2020-08-22T08:46:56","modified_gmt":"2020-08-22T06:46:56","slug":"what-is-principal-component-analysis-pca-a-simple-tutorial","status":"publish","type":"post","link":"https:\/\/kindsonthegenius.com\/blog\/what-is-principal-component-analysis-pca-a-simple-tutorial\/","title":{"rendered":"What is Principal Component Analysis (PCA) – A Simple Tutorial"},"content":{"rendered":"

In this simple tutorial, I would explain the concept of Principal Components Analysis (PCA) in Machine Learning. I would try to be as simple and clear as possible.
\nThe we would use Python in Tutorial 2 to actually do some of the hands-on, performing principal components analysis.<\/p>\n

<\/a><\/div>\n

What is Principal Components Analysis?<\/b>
\nPrincipal Components Analysis is an unsupervised learning class of statistical techniques used to explain data in high dimension using smaller number of variables called the principal components.
\nIn PCA, we compute the principal component and used the to explain the data.
\n<\/ins>
\nHow PCA Work?<\/b>
\nAssuming we have a set X made up of n measurements each represented by a set of p<\/b><\/i> features, X1<\/sub>, X2<\/sub>, \u2026 , Xp<\/sub>. If we want to plot this data in a 2-dimensional plane, we can plot n measurements using two features at a time. If the number of features are more than 3 or four then plotting this in two dimension will be a challenge as the number of plots would be p(p-1)\/2 which would be hard to plot.
\nWe would like to visualize this data in two dimension without losing information contained in the data. This is what PCA allows us to do.<\/p>\n

 <\/p>\n

How to Computer Principal Components?<\/b>
\nGiven a dataset X of dimension n x p, how do we compute the first principal components?
\nTo do this we look for linear combination of the feature values of the form:<\/p>\n

<\/a><\/div>\n

that has the largest sample variance subject to the constraint that:<\/p>\n

<\/a><\/div>\n

This means that\u00a0 the first principal component loading vector solves the optimization problem such that we need to maximize the objective function subject to some constraint.
\nThe objective function is given by:<\/p>\n

<\/a><\/div>\n

And this is subject to the constraint:<\/p>\n

<\/a><\/div>\n

<\/ins>
\nThe objective function (function to maximize) can be rewritten as:<\/p>\n

<\/a><\/div>\n

Since this also holds:<\/p>\n

<\/a><\/div>\n

Therefore the average of z11<\/sub>,…, zn1<\/sub> will also be zero. Therefor the objective function that is being maximized is simply the sample variance of the n values of zi1<\/sub>.
\nz11<\/sub>, z2<\/sub>,…,zn1<\/sub> are referred to as the scores of the first principal component.<\/p>\n

<\/ins>
\nHow then do we maximize the given objective function?\u00a0<\/b>
\nWe do this by performing eigen decomposition of the covariance matrix. Details of how to perform eigen decomposition is explained here.<\/p>\n

Explaining the Principal Components<\/b>
\nThe loading vector\u00a0\u04241<\/sub><\/span> with elements \u042411<\/sub><\/span>, \u042421<\/sub><\/span>,…,\u0424p1<\/sub><\/span>\u00a0 defines a direction in the feature space along which there is maximum variance in the data.
\nThus, if we are to project the n data points x1<\/sub>, x2<\/sub>,…, xn<\/sub> onto this direction, then projected values are the actual principal component scores z11<\/sub>, z21<\/sub>, \u2026, zn1<\/sub>.<\/p>\n

After the first principal components, Z1<\/sub> of the features has been determined, then the second principal component is the linear combination of X1<\/sub>, ,X2<\/sub>,… Xp<\/sub> that has the highest variance out of all the linear combinations that are uncorrelated with Z1<\/sub>. The second principal component scores z12<\/sub>, z22<\/sub>,…,zn2<\/sub> take the form<\/p>\n

<\/a><\/b><\/i><\/u><\/p>\n

<\/div>\n

where \u04242<\/sub><\/span> is the second principal component loading vector, with elements \u042411<\/sub><\/span>, \u042412<\/sub><\/span>, \u2026 ,\u0424p2<\/sub><\/span> . It turns out that constraining Z2<\/sub> to be uncorrelated with Z1<\/sub> is the same as constraining the direction of\u00a0\u04242<\/span> to be orthogonal to the direction of\u00a0\u00a0\u04241<\/sub><\/span>
\nWe would now take an example to see how PCA works.<\/p>\n

 <\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

In this simple tutorial, I would explain the concept of Principal Components Analysis (PCA) in Machine Learning. I would try to be as simple and … <\/p>\n","protected":false},"author":2,"featured_media":288,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0},"categories":[11,16],"tags":[],"_links":{"self":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/12"}],"collection":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/comments?post=12"}],"version-history":[{"count":2,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/12\/revisions"}],"predecessor-version":[{"id":396,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/12\/revisions\/396"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/media\/288"}],"wp:attachment":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/media?parent=12"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/categories?post=12"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/tags?post=12"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}